In the first part of this blog post, I gave some perspective around our progress so far and plans for the future as a first step towards responding to GiveWell’s excellent analysis on our microfinance report. In this second part, I will attempt to address the specific concerns that Holden raised. If you are short on time, you can scroll to the bottom of each paragraph, where I have summarized our plans for improvement in the future (where it made sense).
Issue 1: the definition of “expert” is unclear
I believe that Holden is right on the spot criticizing us that we have not done a good job of defining an expert publicly. The very fact that he had to search in multiple places to find something so fundamental already speaks volumes about the urgent need to bring more transparency and reorganize the way we present information. We intend to do that as soon as possible.
The more substantial issue that GiveWell is bringing up is basically how we came up with these particular experts. Let me provide some details on our process here (a better explanation will most certainly become a central part of our webpage).
When we decide to expand into a new social cause, our first step is to prepare by researching and mapping out the space. In doing so, we identify – to the best of our abilities – all relevant experts in a given field. We list all foundations, nonprofits, and academic institutions and then find the experts’ contact information in these organizations. In addition, in some social causes we include journalists, policy makers, researchers, and other types of experts. The end result is a list of hundreds of experts who we then invite to participate in our surveys.
Of course, not every invited expert participates today. And of those who participate, we don’t necessarily include all responses from all experts because we do additional sorting of experts during the data collection process. What is more, not all experts are “equal” in terms of their expertise or experience. And some – most notably foundation professionals and academics – bring disproportionate value to our research. However, we strongly believe in our current inclusive model, which also invites nonprofit executives and senior staff to participate for several reasons. First, this group is obviously much bigger than researchers or funders. Not including them would be a missed opportunity to capture knowledge from those “working in the trenches.” Second, getting nonprofit executives involved opens the door to fix the broken feedback loops by using expert reviews as a starting point. For example, many nonprofits have expressed interest in being able to respond to the reviews (a functionality that we will be developing soon) – which would get us one step closer to increased transparency. The reason why nonprofit executives form such a large part of the final expert network is that there are simply more of them, compared to academics or foundation professionals. And to address the issue of the “others” category, we plan to develop more specific categories.
So, overall, I completely agree with GiveWell that we need to disclose substantially more information to build more trust among donors. We have internally identified the same issue and are working to redesign our webpage to address the concern.
Issue 2: who are the experts?
I have already mentioned the challenge of answering the question of who our experts are because of our still incomplete research in part 1 of this blog post. Our goal is to continuously get more experts “on the record” and include their bios. Based on our experience so far, experts are excited to participate in the research and I am confident that we will end up featuring more than the 31 people we have today in microfinance.
On the more specific issue of featuring microfinance skeptics, I want to make two points. First, both microfinance skeptics and proponents have recently been announcing results from different research projects that point in different directions with respect to the effects of microfinance. Regardless of who is right (which is often somewhat subjective), one obvious limitation of any crowdsourced approach (such as ours) is that it takes time to change the perceptions and opinions of a group of people. Second, our research actually features a number of microfinance skeptics at the urging of Dean Karlan, who has kindly provided us with critical feedback, served as a sounding board, and helped us connect with more researchers in the space.
Nevertheless, GiveWell is making a terrific point here: our ability to build trust and engage donors depends on being able to attract and feature credible experts. It is our intent to make substantial progress by being selective about our experts, asking them for more participation, and continuing to build large and representative networks of experts. In addition, we intend to provide better social cause summaries, which highlight the core issues as well as most important debates.
Issue 3: the recommended charities
GiveWell makes an interesting observation that the “list reads like a who’s who in large U.S. microfinance charities.” This statement is quite surprising to me, because our own experience when testing the expert mutual fund with donors is quite different. Perhaps to insiders, the list reads like the who’s who, but to the average donor that is certainly not the case. In fact, most people’s knowledge is limited to Kiva.
In some industries, it is perfectly fine for the “average” individual to not know the “who’s who.” But in most social causes in philanthropy that is not the case, because individuals provide the vast majority of funding – and they do it on the basis of very little good information. So for our national social causes, the lists might read like the who’s who to experts, but that is certainly not the case for our target user: the individual donor. And I would personally argue that perhaps an even bigger opportunity is our work on a local level – highlighting top nonprofits in different communities, which inspires people to join forces, support good organizations, and work towards solving the important societal problems of the day.
GiveWell also criticizes the expert-recommended charities and I personally certainly respect their opinion as microfinance skeptics. However, the Philanthropedia vision of accelerating the pace of social change has a different approach. We know that no organization out there is perfect – that is why we identify both strengths and areas for improvement. Our goal is to engage experts, donors, and nonprofits to collectively find and fund the best organizations that are solving some of the world’s toughest social problems. I believe that this can only happen through an inclusive process of continuous improvement, focused on bringing more transparency and demanding more impact from these nonprofits.
Issue 4: expert quotes / reviews quality
GiveWell also uses the Kiva expert quotes to analyze the usefulness of our expert reviews. The main concern that Holden identifies is whether experts are indeed doing a good job of helping donors increase their impact. This is a concern that I personally share and that is at the core of our push to increase the quality of our research in 2010 that I wrote about in Part 1. Some of the upcoming changes that we will be making in our methodology and survey design should help us get higher quality reviews. Further, as we gain more traction and educate more experts about our mission, this problem will gradually be reduced.
However, I should note that we continuously evaluate all aspects of Philanthropedia and look for ways to improve. We have performed research on our expert reviews in Bay Area homelessness and climate change (both with foundation professionals and donors) and the feedback we got was that the comments capture the essence of what the different nonprofit’s strengths and weaknesses are. For example, one of Kiva’s big accomplishments is that it has been able to capture the attention and imagination of the individual. This is a fascinating and rather unique achievement that our experts have legitimately pointed out. So we feel that the starting point is not as low as GiveWell might portray.
In summary, I believe that our expert reviews can be further improved by better surveying and educating experts on the importance of sharing their opinion. This issue is front and center for us as we look to expand into more social causes in 2010.
Issue 5: reinforcing bad incentives
Holden also hypothesizes that our methodology encourages experts not to disclose their names, and encourages charities to do whatever to get them recommended by Philanthropedia’s set of experts. This is the main issue identified by GiveWell that I disagree with. In my opinion, Philanthropedia has the opposite effect from the one outlined in the GiveWell post. First, we are increasing the transparency of experts. Even if only 31 experts agreed to publish their bios and stand behind the results, that is exactly 31 more than before. And our plans are to further push the bar, both in terms of what experts disclose and in terms of how many experts participate.
Second, Philanthropedia is getting areas for improvement of different charities “on the record” and thus inspiring them to participate in the discussion, increase their transparency, and further improve. The idea that charities will be able to manipulate the results is certainly plausible, but there are a number of reasons why I think it is highly unlikely:
- Experts have high morals and ethics – foundation professionals, nonprofit executives, and academics operate in an environment characterized by conflicts of interest. For example, researchers might need access and data to assess nonprofits’ impact, or foundation professionals might receive multiple grant proposals from competing nonprofits, while collaborating with them on other projects. In general, in our dealings with experts, we have found them to have very high personal and professional integrity.
- The Philanthropedia methodology relies on a high number of experts, which makes gaming the system challenging.
- We currently have a number of internal checks that makes manipulating the results difficult. Moreover, we will be adding other policies such as random checks, more due diligence, better conflict of interest management, etc. to further reduce such risks.
- Of course, I fully expect that if someone is dedicated enough, they will be able to game the system. That’s why we plan on very aggressively punishing any offenders, who violate the our trust, as well as the trust of the community.
In summary, I personally disagree that we are promoting the wrong incentives. I believe that the public discussions, engagement of new donors, and reactions that nonprofits have speak to that as well.
In conclusion, let me again thank GiveWell for their time and analysis. The rest of the Philanthropedia team and I very much agree with the vast majority of the issues that were identified, as well as the suggestions for improvement. We absolutely plan on raising the bar in terms of transparency and quality in order to be able to gain the trust of both donors and experts. Stay tuned for our reorganized webpage that addresses some of the concerns that GiveWell brought up.
GiveWell, in addition to many others, have kindly said that they see a lot of potential in our model. We at Philanthropedia certainly aspire to live up to that expectation, and with more analyses and constructive feedback, we have a real shot of getting there. So thanks again for your leadership Holden and Elie and for helping us get to the next level.