Philanthropedia is a well thought out and exciting business model that the philanthropy community admires and values. We have long needed an unbiased approach that helps investors and donors identify effective charities and this site has done an excellent job so far. We are all anxious to see their recommendations in other charity categories and their future contributions to help everyone in the philanthropy community develop better ideas and metrics in defining effectiveness for charities.

The criticism from GiveWell about the site, transparency, and other identified charities went too far. Philanthropedia has to honor the privacy of their experts and for good reason they are not going to list their confidential sources as someone else could copy their idea. Its time that GiveWell take a more positive approach and focus their efforts on identifying effective charities beyond the five small ones they list on their site. What is their specific criteria for selecting their five charities and why don’t they have more recommendations?

Criticizing competing sites and other resources that are doing excellent work is not helpful to anyone in the giving and philanthropy community. I am a donor and don’t know the leaders at Philanthropedia but use the content on the site to help me identify charities that are working to make the world a better place.

I have not examined any one issue area in depth, as Givewell has, but simply looking at a random organization highlights a key question: What does Philanthropedia mean by nonprofit effectiveness?

If you do indeed “believe that there is no one metric or measure of nonprofit effectiveness,” as Saundra quotes your web site, and therefore do not tell which criteria the experts should base their assessments upon, I find it puzzling and nothing less than outrageous. Surely the extent to which organizations are helping people in the ways they intend (outcomes) are at the heart of effectiveness? Marketing, status and fundraising success are meaningless unless the people served are better off.

Helping people is why nonprofits are in business and it is why so many are looking for web sites like yours to help them find organizations that actually succeed at that difficult task. You have the opportunity to do something meaningful – as opposed to perpetuating existing views about various organizations based on their ability to market themselves.

If you define nonprofit effectiveness, I think you will find that your choice of organizations and experts will follow naturally. I randomly looked at the Tenderloin Neighborhood Development Corporation, for instance, and found no information in your section about “impact” (other than numbers about people served, which is a far cry from impact, because we know nothing about how well they were served and whether they are better off).

It strikes me that if you are unable to access data about the extent to which outcomes have been achieved, it is impossible to assess an organization on effectiveness and it should not even be mentioned (unless, of course, you make it clear to your website visitors how your definition of effectiveness is different than the one I suggest). Further, if outcomes are the focus, it follows that researchers and others who can assess organizational claims to outcomes are the most important category of experts and that they must all be provided with access to the same information. Impressions are utterly useless when it comes to the question of outcomes.

Like others who are contributing to this debate, I believe Philanthropedia is a good idea, but the questions that have been raised are so important that its current value is limited and potentially negative (in that people take the information at face value and direct their dollars to organizations that are ineffective or even harmful). I would be very interested to see a post addressing your definition of nonprofit effectiveness before you get into any other details. That definition is the foundation for everything else.

I was directed to your site by two different people with whom I had discussed the possibility of crowd sourcing charity evaluations. After I reviewed your site I came away with many of the same questions that GiveWell outlined in their post.
• Who were your experts?
• How were they chosen?
• What criteria do they use to judge charities?
• Is the criteria meaningful or is it simply things that are easy to evaluate.

Without answers to these questions it was impossible to evaluate your rating system. Thus, I was very interested to read your response to GiveWell’s post. Unfortunately, instead of helping me understand your processes better, it left me with even more questions than before.

What criteria does Philanthropedia use to choose experts?
I am still interested in more information on how experts are chosen than you’ve provided. “When we decide to expand into a new social cause, our first step is to prepare by researching and mapping out the space. In doing so, we identify – to the best of our abilities – all relevant experts in a given field. We list all foundations, nonprofits, and academic institutions and then find the experts’ contact information in these organizations. In addition, in some social causes we include journalists, policy makers, researchers, and other types of experts. The end result is a list of hundreds of experts that we then invite to participate in our surveys.”

What criteria do you use in choosing these experts? Are these just US based people or do you seek out experts from other developed, developing, and under developed countries? Is there a certain number of years of experience they must have, and at what level? Do they have to have worked for or with that particular organization or done research on it, or can they evaluate it just having heard about it through the grapevine? How large do foundations have to be, can they qualify if they are a small family foundation? Do you extend the invitation to everyone from an organization, if not how do you pick which people to invite? What percentage of experts accept your request and which percentage decline to participate?

How does Philanthropedia weigh and evaluate expert opinions?
There is bound to be disagreement, as there are in so many aspects of aid, how do you handle conflicting opinions or information?

“What is more, not all experts are “equal” in terms of their expertise or experience. And some – most notably foundation professionals and academics – bring disproportionate value to our research. However, we strongly believe in our current inclusive model, which also invites nonprofit executives and senior staff to participate for several reasons.”
Do some expert opinions carry “disproportionate value”, and if so how is that reflected in the final result?
What are your “monitoring and curating processes.”
“Of course, not every invited expert participates today. And of those who participate, we don’t necessarily include all responses from all experts because we do additional sorting of experts during the data collection process.” What additional sorting do you do? What would cause you to disregard the opinion of an expert you invited to participate?
What are your “internal checks that makes manipulating the results difficult.” How do you “aggressively punish any offenders”?

What criteria are used to evaluate the charities?
Although you refer to changing your survey and methodology, you give no indication of what you currently use. When I looked for the answer to this on your website this is what I found.

“We believe that there is no one metric or measure of nonprofit effectiveness. Therefore, we take a holistic approach to answer this question. Rather than tell the experts which criteria they should be considering when thinking about strong nonprofits, we ask our experts to identify what three criteria they use to judge nonprofit effectiveness. By gathering this information, we are able to capture the views of a variety of professionals and provide unique insight into the best way to evaluate nonprofits in a particular social sector.”

Do I understand correctly that you have no set criteria? How many and which nonprofits have you evaluated and decided against funding, and why? Do you inform the nonprofits when they have been evaluated but not chosen, and is there an appeals process or the ability to provide proof to rebut negative opinions?

While looking for the expert evaluation on Alliance for Climate Protection I found that this organization was promoted as having similar strengths as Kiva.
“Strengths
Outstanding public visibility.
‘They have great media/PR that can help to change hearts and minds on the issue. They also have the cache of Al Gore, and a new investment in a field component.’
‘They have good visibility because of linkage to Al Gore.’
‘They have a strong television/media presence.’
‘They have an excellent website and e-mail alerts on climate protection issues. They have engaging and clever internet ‘ads’ and videos.’
‘The ACP produces remarkably compelling videos. I don’t watch TV, but I know that they’re catching the attention on non-activists based on what I hear anecdotally.’
‘They are high-profile.’ ”

Just as Holden found Kiva’s marketing ability to be a questionable factor to be evaluated on, I too question whether the ability “to capture the attention and the imagination of the individual” is any indication of the quality of the nonprofit. McDonald’s owns more playgrounds in the US than any other private or public organization. While this is a great marketing tool, it doesn’t mean that their burgers are better than Wendy’s or that burgers and fries are the healthiest food to have for dinner. Is good PR one of the criteria commonly used to evaluate nonprofits?

Transparency
I am a strong advocate for transparency, both with charities and with charity rating systems. I am glad to hear that Philanthropedia is working towards more transparency. I urge you to be clear and honest with donors about your strengths, weaknesses, and areas for improvement. They have the right, and the need, to evaluate the information they are presented with before donating.

Set an example for the agencies you are evaluating. We cannot ask them to be more transparent if we ourselves are not.

Good luck with your work improving Philanthropedia’s systems. I look forward to seeing your progress and to having a clearer understanding of your evaluation system.