Philanthropedia Blog

A Response to Good Intentions: More on Philanthropedia’s Methodology

January 11th, 2010 by Deyan Leave a reply »

This is another post in my series of articles explaining more about Philanthropedia. As always, I am extending an open invitation for others to comment and join the discussion. No point is too small or insignificant, so feel free to leave a comment! For those that prefer a private exchange, please email us at feedback@myphilanthropedia.org – as some of you already know, I answer almost all emails personally and promptly.

As a follow up to the recent exchange between GiveWell and us, Saundra from Good Intentions blogged about her questions, saying that “[u]nfortunately, instead of helping me understand your processes better, it left me with even more questions than before.” While I hope my blog post brought some clarity for readers, as a philosopher once said “the more I know, the more I realize how much more there is to know.” That is to be expected given the complexity of the subject matter and that we, ourselves are constantly coming up with new questions and searching for ways to improve our methodology. In any case, as I have said before, we are committed to transparency and plan to rework our webpage to better explain our approach.

Saundra had a number of excellent questions that I will attempt to answer below:

Choosing Experts

How do we choose experts? Our approach is rather straightforward and is based on three main categories: foundation professionals, academics, and nonprofit senior staff. When choosing experts, we currently look at a number of factors including years of experience working in the sector, job title and occupation, professional affiliations and/or academic background, with 2 years of experience and relevant career being minimum criteria. In addition, we ask experts to self-rate their expertise on a scale from 1-5 (where 5 means “most” expert), with a self-rating of “3” as a minimum requirement. Here is the scale that we used for microfinance:

On average, how would you characterize your expertise in microfinance?

Limited: “I have limited knowledge of this issue area and do not feel qualified to identify outstanding organizations.”

Basic: “I have basic knowledge of this issue area, and might be able to make a directional assessment of the organizations with which I am familiar. My professional experience and training might qualify me to identify and evaluate a few of the most outstanding organizations in the sector.”

Moderate: “I have moderate knowledge of this issue area and am confident in making a directional assessment of the organizations with which I am familiar. My professional experience and training probably qualifies me to identify and evaluate a few of the most outstanding organizations in the sector.”

Strong: “I have strong knowledge of this issue area that is both broad and deep. My professional experience and training qualifies me to identify and evaluate some, but perhaps not most or all, of the most outstanding organizations in the sector.”

Expert: “I have expert knowledge of this issue area that is both broad and deep, including experience with multiple sub-issues. My professional experience and training qualifies me to confidently identify and evaluate most or all of the most outstanding organizations in the sector.”

The vast majority of our experts are US based, reflecting the fact that this is the primary focus of our research and operations (microfinance being an exception). In addition, we ask experts to comment on the organizations that they know the best and we do not suggest or influence their responses in any way (i.e. these are open-response questions). We also always try to err on the side of inclusiveness: we do not set specific limits about a foundation’s size and we try to invite the all relevant experts within an organization. However, we do reserve the right to disqualify experts on the basis of very poor quality of responses or other related factors.

As for how many experts actually respond, here is a table that summarizes our response rates:

Social Cause Education (pilot) Climate change Bay Area Homelessness Microfinance
Invited Experts

N/A (only relied on referred contacts)

773

392

1049

Participating Experts

39

139

83

131

Conversion rate

N/A

18%

21%

12.5%

These numbers reflect a bit of the evolution of the venture. First, the main challenge in education was “can expert crowdsourcing be done at all.” As Bob Ottenhoff (GuideStar CEO and member of our Advisory Board) said, this is a challenge that the nonprofit sector has been trying to solve in the last 10 years.

Once we demonstrated some initial success with education, the next important issue was “can expert crowdsourcing be done at scale” (i.e. beyond just referrals from a few personal connections). Building on our initial momentum, we managed to overcome this challenge as well, building big and representative expert networks in these two national/international social causes. Microfinance was particularly difficult, given that it is very much an international issue. Even though we intend to focus on the US for the foreseeable future, we saw this social cause expansion as an important experiment in how far we could push our methodology and we have drawn a lot of lessons from it. And finally, we wanted to see if our approach could work on the local level, which is why we decided to expand into the Bay Area, starting with homelessness. We are quite happy with these initial results as well and plan to continue our work on the local level.

To summarize, we have put a lot of effort in the past 2 years to create a solid methodology that allows us to compile expert opinion about impactful organizations in different social causes. We now also have a really good grasp of how to go about finding and recruiting experts to participate in our research. Nevertheless, we see a lot of room for growth and continued iteration of our surveys and experimentation around the best ways to excite and educate even more experts.

Processing expert opinions

Saundra also asked how we handle disagreement, which is a great question that we constantly ask ourselves as well. The short answer is that our goal is to be able to identify all good approaches toward solving a particular problem rather than make judgment calls about what is “good” or “bad” (something that our experts are in a better position to answer). We believe that by publicly discussing the pros and cons of each solution, the sector will be able to collectively arrive at better ways to have more impact. For an example of this, you can see the organizations that we highlighted in climate change, which include policy advocacy, grassroots, and scientific-based organizations. I should repeat that in the next version of our website, we plan to better highlight the diversity of these approaches for donors and provide a forum for discussions of the pros and cons of each.

On the question of whether some experts have disproportionate value, the answer is yes, as can be seen in the graph below:

Social Cause

Education (pilot) Climate change Bay Area Homelessness Microfinance

Foundation professional avg votes

n/a

4.19 (n=21)

4.50 (n=6)

(TBD)

Academic avg votes

n/a

4.18 (n=11)

3.67 (n=6)

(TBD)

Nonprofit staff avg votes

n/a

3.55 (n=71)

2.98 (n=58)

(TBD)

Note: “avg votes” refers to number of organizations recommended by the particular survey taker group (out of 5). We have not yet computed this for microfinance.

As you can see, foundations professionals and academics do bring disproportionate value – at least in the research that we have already done. That means that they have more influence over the final results as well. We do not adjust the scores in any other way, although we are continuously reevaluating this issue and looking for ways to have better results.

In summary, we recognize and celebrate the diversity of opinions that the different types of experts bring, and are committed to highlighting our research results in an unfiltered and actionable way on our website.

Criteria for evaluation of charities

Saundra also inquired about our criteria for evaluating charities. As our current FAQs state, we do not set our own criteria, but instead ask for the criteria that experts use to evaluate nonprofits. That is a fundamental difference between our model and GiveWell’s (both of which have their pros and cons in my opinion). Currently, we do not make the compiled criteria public, but this will change in the next iteration of our website.

In terms of charities, as a result of our methodology, we generate a long list of nonprofits (for example, in climate change we had a total of 169 organizations that our experts mentioned). But out of this long list, we only highlight the few nonprofits that are consistently mentioned by our experts. And in the near future, we certainly plan to have a response box for nonprofits that want to reply to the identified areas for improvement.

Expert reviews & difference between impact and effectiveness

Saundra also pointed to the Alliance for Climate Protection expert review, asking whether having a strong marketing ability should be a reason to recommend a nonprofit. I personally believe that this is not the case – we should be recommending nonprofits based on impact first and foremost. However, our previous research has focused on effectiveness in order to study just how much good information we could extract. (We have now decided to zoom in and only focus on impact).

Effectiveness includes a number of properties: impact of their solution, efficiency of deploying resources, attracting top talent, good governance, etc. In that line of thinking, good marketing is another important characteristic of effectiveness that our experts have legitimately pointed out given the question that we asked. Of course, by zooming into any particular organization, it is easy to miss the big point: namely, there are a number of organizations in any given social cause, each with its own strengths and weaknesses. By making these pros and cons visible, and collaborating and learning together, organizations have a much better shot of coming up with better solutions that have more impact.

In summary, our intention is to compile expert opinions to highlight strengths and weaknesses of existing models to help make them better. Many of the social problems that nonprofits are trying to solve are incredibly complex and making progress takes time, effort, resources, and patience. That is why our goal is to acknowledge which organizations are doing well and then help put them on the right track towards further improvement through a combination of public scrutiny and constructive debate. I will further elaborate on this point, and how it fits into my previously discussed idea for continuous improvement, in a separate blog post.

6 comments

  1. Saudra says:

    Thanks so much for taking the time to respond to my questions. I appreciate the clear details you have provided and look forward to seeing your website updates.

    Saundra

  2. Deyan says:

    You are welcome!

  3. I am absolutely blown away at how interesting the stuff is on this website. I have written down this website and I truly plan on coming back to the site in the next few days. Keep up the fantastic work!

  4. yesss very thanks man i love this site

Leave a Reply


Philanthropedia is a registered 501(c)3 organization. All of your donations are 100% tax-deductible.