With this post, I want to close the loop on the recent conversation between GiveWell, Saundra, Ingvild, and us (among others) and provide a summary of the core issues and where we stand.
Issue 1: How are experts defined and who are they?
The answer to these questions is rather straightforward, as I explained here and here. In short, while there is certainly plenty of room to grow especially in terms of getting more foundation professionals to participate, we have a good grasp of how to find and recruit experts for our surveys. These experts come from the places you would expect and where, we argue, a lot of nonpublic knowledge about nonprofit impact is aggregated: among nonprofit senior staff, foundation professionals, and academics.
The related issue of transparency of our expert network is important, and we certainly plan to reveal more information in order to build credibility and trust, as well as promote open discussions about nonprofits. In their last blog post, GiveWell made an additional stronger claim that we need to associate individuals with their quotes, which I strongly disagree with. As a matter of fact, evidence and research points in the opposite direction: providing a forum for anonymous feedback creates opportunities to surface candid opinions about strengths and weaknesses (for a good example, consult literature on 360 evaluations). What is more, such a change would effectively prevent certain types of people from participating at all – most notably, foundation professionals, who need to make daily decisions about which nonprofits to support with grants. That is why we have no plans to mandate that individual experts be linked to their recommendations and we maintain committed to providing a safe forum for candid feedback about strengths and weaknesses of different nonprofits. That being said, we are planning changes to our survey and policies to increase the number of experts, who will be acknowledged as participants in our research. To us, it is a question of when rather than a question of if, so stay tuned for changes as Philanthropedia continues to mature as an organization.
Issue 2: How are organizations recommended?
In a comment in an earlier blog post, Ingvild built on GiveWell’s questions and asked how our experts recommend nonprofits (focusing on the difference between impact and effectiveness in particular). As I have mentioned before, we believe that the experts themselves have criteria for evaluating nonprofits. We collect that information in one of the first parts of our survey. Our goal is then to get recommendations from the experts about nonprofits based on these very same criteria, which we then compile and standardize to make available to the public.
Today, we ask experts about their criteria for effectiveness, which to me includes impact in addition to other elements such as marketing ability, fundraising efficiency, leadership, hiring, etc. We did that to experiment and see how far we could push our methodology and how much information we could collect. However, our number one priority remains assessing nonprofits based on impact, which is why we are planning some changes to our survey to make sure our we can collect enough specific information about the impact of nonprofits.
In summary, we have learned a lot from this debate – both in terms of how we could improve, as well as what we need to explain better. In future blog posts, I plan to address several topics including (1) our vision for the space, (2) specific changes motivated by this public discussion, (3) fundamental pros and cons of our model of using experts as a proxy (i.e. what we are and what we are not), (4) more information on our whitepaper and planned improvements to our methodology, (5) assessment of progress to date, and (6) planned social cause expansion for 2010.
Thank you all for your feedback!