We are excited to share this whitepaper, “Consolidating Expert Opinion about High-Impact Nonprofits: Review of Philanthropedia’s Methodology.” It summarizes more than two and a half years of research and highlights the results of our climate change research in much more depth. We are copying the executive summary below. You can download the full whitepaper, as well as add your feedback at http://www.myphilanthropedia.org/whitepaper.
The Philanthropedia whitepaper, “Collecting Expert Opinion about High-Impact Nonprofits: Review of Philanthropedia’s Methodology,” serves two main purposes: (1) to make the case for using experts to identify high-impact nonprofits, and (2) to share our progress towards a specific methodology to date.
Our overall conclusions are that:
- Our methodology captures expert opinion about high-impact nonprofits in different social causes.
- Using experts to identify high-impact nonprofits offers unique advantages in terms of high quality information about nonprofits at low cost to gather that information.
Because of the potential of this approach to evaluating nonprofits, we are investing in improvements to further strengthen our research.
The Case for Using Experts to Evaluate Nonprofits
Philanthropy is primarily concerned with distributing limited monetary resources to charities doing the best work in solving social problems. Therefore, we believe there are two main problems the sector faces:
- How to identify these high-impact nonprofits
- How to how to distribute resources effectively to these nonprofits
In this whitepaper, we assess existing nonprofit evaluators and ourselves on two dimensions: quality and cost.
- Quality relates to how closely the measures used to evaluate a nonprofit are correlated with impact per dollar invested and the nonprofit’s capacity to absorb more resources. We define impact as a measure of the lasting improvements produced by that nonprofit to address the core problems in a particular social cause.
- Cost is the combination of resources needed to perform the evaluation of a nonprofit including money, people, and time. Therefore a good solution is one where there is a quick, scalable, low cost way to evaluate many nonprofits across multiple social causes.
To date, nonprofit evaluators have been able to make progress on either the cost or quality front, however, none have been able to strike a good balance between high quality and low cost. The Philanthropedia approach fills this gap by using a low-cost method of surveying experts to identify high quality information about nonprofits:
Philanthropedia offers a high quality solution because we capture the opinions of experts who are uniquely qualified to assess nonprofits. Experts are qualified because they have access to nonpublic data about charities and have advanced mental models for evaluating impact.
Philanthropedia offers a low cost solution because it takes experts only about 40 minutes to complete both online surveys, experts are not paid for their participation, and one trained employee can conduct four social cause research projects concurrently over the course of 2-3 months.
Philanthropedia’s methodology of surveying diverse and representative groups of social cause experts to identify high-impact nonprofits runs in six steps.
- Research and define the social cause or scope of the research
- Identify and recruit experts
- Run a survey asking experts to identify high-impact nonprofit
- Analyze survey data
- Run a second survey to determine agreement among experts about high-impact nonprofits, collect strengths and areas for improvement for each nonprofit, and ask experts to allocate funding across nonprofits
- Compile and analyze final results and publish a list of high-impact nonprofits for the social cause
When identifying experts, we require they have a minimum of two years of experience, relevant work experience as evidenced by current or past job titles or employers, and a minimum expert self-rating on a Philanthropedia-developed scale. Our goal is to create a representative expert network along two dimensions: profession type (foundation professionals, nonprofit senior staff, researchers, and others) and geography. And, we screen for high-quality responses.
For the top 15% of nonprofits recommended by experts, we collect the number of expert votes, the percentage of agreement across the expert network that the nonprofit is among the most effective in that sector, the percentage allocation of each nonprofit as part of an Expert Fund, and the strengths and areas for improvement.
Results and Data Analysis
To date, we have researched four social causes to test and develop our methodology: education, climate change, microfinance, and Bay Area homelessness. To demonstrate our approach, we analyze the results of our climate change research. We built a diverse and representative expert network of 139 experts with an average of 12.94 years of experience. The expert breakdown was 19% foundation professional, 12% researchers, 47% nonprofit senior staff, and 22% other (policy makers, government officials, etc). Forty-one percent of our experts came from the east coast, 9% from the midwest, 4% from the south, 6% from the northwest, 30% from the west coast, and 10% international and from other locations. Our experts recommended 15 nonprofits which represent the top 15% of the mentioned nonprofits.
In order to determine how much our final results were influenced by each expert type, we ran correlations between the combined top nonprofit list and the lists recommended by each expert type. While nonprofit senior staff had the most influence on the final list (due to their larger representation in the sample size), all three expert types (foundation professionals, nonprofit senior staff, and researchers) had a high degree of agreement about which nonprofits were most effective.
In order to determine which factors might be influencing the final ranking of nonprofits, we ran correlations with the external factors: nonprofit revenue, brand awareness (as measured by the number of Google mentions), age of organization, and size of organization (as measured by number of employees). We found that none of these external factors influenced the final nonprofit rakings in any significant way. Therefore we conclude that compiling expert opinion adds unique value when identifying high-impact nonprofits.
Areas for Improvement
The results from our climate change research and other studies are very encouraging. However, we believe there are still many areas for improvement. In particular, we intend to make changes in the way we sample experts, in the way we state the questions in our surveys, and in the way we share the results.
We intend to improve:
- the research quality in terms of expert responses and their ability to identify high-impact nonprofits.
- the clarity of the language used in our surveys in order to better communicate our goals to experts.
- our transparency by sharing more information publicly so that we can continue to build trust in the philanthropic community.
We are excited about the unique advantages that our methodology offers in terms of high quality and low cost, which is why we are investing in these further improvements. We hope this whitepaper clarifies Philanthropedia’s approach of collecting expert opinion about high-impact nonprofits.
We do this research hoping to influence donors as they search for nonprofits to support. We believe that donors can have a bigger impact in the nonprofit sector by directing more financial resources to some of the highest-impact nonprofits.
We invite feedback and discussion about this whitepaper and our work at: http://www.myphilanthropedia.org/whitepaper.