Philanthropedia Blog

Responding to GiveWell’s Analysis of our Microfinance Report – Part 2/2

January 4th, 2010 by Deyan Leave a reply »

In the first part of this blog post, I gave some perspective around our progress so far and plans for the future as a first step towards responding to GiveWell’s excellent analysis on our microfinance report. In this second part, I will attempt to address the specific concerns that Holden raised. If you are short on time, you can scroll to the bottom of each paragraph, where I have summarized our plans for improvement in the future (where it made sense).

Issue 1: the definition of “expert” is unclear

I believe that Holden is right on the spot criticizing us that we have not done a good job of defining an expert publicly. The very fact that he had to search in multiple places to find something so fundamental already speaks volumes about the urgent need to bring more transparency and reorganize the way we present information. We intend to do that as soon as possible.

The more substantial issue that GiveWell is bringing up is basically how we came up with these particular experts. Let me provide some details on our process here (a better explanation will most certainly become a central part of our webpage).

When we decide to expand into a new social cause, our first step is to prepare by researching and mapping out the space. In doing so, we identify – to the best of our abilities – all relevant experts in a given field. We list all foundations, nonprofits, and academic institutions and then find the experts’ contact information in these organizations. In addition, in some social causes we include journalists, policy makers, researchers, and other types of experts. The end result is a list of hundreds of experts who we then invite to participate in our surveys.

Of course, not every invited expert participates today. And of those who participate, we don’t necessarily include all responses from all experts because we do additional sorting of experts during the data collection process. What is more, not all experts are “equal” in terms of their expertise or experience. And some – most notably foundation professionals and academics – bring disproportionate value to our research. However, we strongly believe in our current inclusive model, which also invites nonprofit executives and senior staff to participate for several reasons. First, this group is obviously much bigger than researchers or funders. Not including them would be a missed opportunity to capture knowledge from those “working in the trenches.” Second, getting nonprofit executives involved opens the door to fix the broken feedback loops by using expert reviews as a starting point. For example, many nonprofits have expressed interest in being able to respond to the reviews (a functionality that we will be developing soon) – which would get us one step closer to increased transparency. The reason why nonprofit executives form such a large part of the final expert network is that there are simply more of them, compared to academics or foundation professionals. And to address the issue of the “others” category, we plan to develop more specific categories.

So, overall, I completely agree with GiveWell that we need to disclose substantially more information to build more trust among donors. We have internally identified the same issue and are working to redesign our webpage to address the concern.

Issue 2: who are the experts?

I have already mentioned the challenge of answering the question of who our experts are because of our still incomplete research in part 1 of this blog post. Our goal is to continuously get more experts “on the record” and include their bios. Based on our experience so far, experts are excited to participate in the research and I am confident that we will end up featuring more than the 31 people we have today in microfinance.

On the more specific issue of featuring microfinance skeptics, I want to make two points. First, both microfinance skeptics and proponents have recently been announcing results from different research projects that point in different directions with respect to the effects of microfinance. Regardless of who is right (which is often somewhat subjective), one obvious limitation of any crowdsourced approach (such as ours) is that it takes time to change the perceptions and opinions of a group of people. Second, our research actually features a number of microfinance skeptics at the urging of Dean Karlan, who has kindly provided us with critical feedback, served as a sounding board, and helped us connect with more researchers in the space.

Nevertheless, GiveWell is making a terrific point here: our ability to build trust and engage donors depends on being able to attract and feature credible experts. It is our intent to make substantial progress by being selective about our experts, asking them for more participation, and continuing to build large and representative networks of experts. In addition, we intend to provide better social cause summaries, which highlight the core issues as well as most important debates.

Issue 3: the recommended charities

GiveWell makes an interesting observation that the “list reads like a who’s who in large U.S. microfinance charities.” This statement is quite surprising to me, because our own experience when testing the expert mutual fund with donors is quite different. Perhaps to insiders, the list reads like the who’s who, but to the average donor that is certainly not the case. In fact, most people’s knowledge is limited to Kiva.

In some industries, it is perfectly fine for the “average” individual to not know the “who’s who.” But in most social causes in philanthropy that is not the case, because individuals provide the vast majority of funding – and they do it on the basis of very little good information. So for our national social causes, the lists might read like the who’s who to experts, but that is certainly not the case for our target user: the individual donor. And I would personally argue that perhaps an even bigger opportunity is our work on a local level – highlighting top nonprofits in different communities, which inspires people to join forces, support good organizations, and work towards solving the important societal problems of the day.

GiveWell also criticizes the expert-recommended charities and I personally certainly respect their opinion as microfinance skeptics. However, the Philanthropedia vision of accelerating the pace of social change has a different approach. We know that no organization out there is perfect – that is why we identify both strengths and areas for improvement. Our goal is to engage experts, donors, and nonprofits to collectively find and fund the best organizations that are solving some of the world’s toughest social problems. I believe that this can only happen through an inclusive process of continuous improvement, focused on bringing more transparency and demanding more impact from these nonprofits.

Issue 4: expert quotes / reviews quality

GiveWell also uses the Kiva expert quotes to analyze the usefulness of our expert reviews. The main concern that Holden identifies is whether experts are indeed doing a good job of helping donors increase their impact. This is a concern that I personally share and that is at the core of our push to increase the quality of our research in 2010 that I wrote about in Part 1. Some of the upcoming changes that we will be making in our methodology and survey design should help us get higher quality reviews. Further, as we gain more traction and educate more experts about our mission, this problem will gradually be reduced.

However, I should note that we continuously evaluate all aspects of Philanthropedia and look for ways to improve. We have performed research on our expert reviews in Bay Area homelessness and climate change (both with foundation professionals and donors) and the feedback we got was that the comments capture the essence of what the different nonprofit’s strengths and weaknesses are. For example, one of Kiva’s big accomplishments is that it has been able to capture the attention and imagination of the individual. This is a fascinating and rather unique achievement that our experts have legitimately pointed out. So we feel that the starting point is not as low as GiveWell might portray.

In summary, I believe that our expert reviews can be further improved by better surveying and educating experts on the importance of sharing their opinion. This issue is front and center for us as we look to expand into more social causes in 2010.

Issue 5: reinforcing bad incentives

Holden also hypothesizes that our methodology encourages experts not to disclose their names, and encourages charities to do whatever to get them recommended by Philanthropedia’s set of experts. This is the main issue identified by GiveWell that I disagree with. In my opinion, Philanthropedia has the opposite effect from the one outlined in the GiveWell post. First, we are increasing the transparency of experts. Even if only 31 experts agreed to publish their bios and stand behind the results, that is exactly 31 more than before. And our plans are to further push the bar, both in terms of what experts disclose and in terms of how many experts participate.

Second, Philanthropedia is getting areas for improvement of different charities “on the record” and thus inspiring them to participate in the discussion, increase their transparency, and further improve. The idea that charities will be able to manipulate the results is certainly plausible, but there are a number of reasons why I think it is highly unlikely:

  • Experts have high morals and ethics – foundation professionals, nonprofit executives, and academics operate in an environment characterized by conflicts of interest. For example, researchers might need access and data to assess nonprofits’ impact, or foundation professionals might receive multiple grant proposals from competing nonprofits, while collaborating with them on other projects. In general, in our dealings with experts, we have found them to have very high personal and professional integrity.
  • The Philanthropedia methodology relies on a high number of experts, which makes gaming the system challenging.
  • We currently have a number of internal checks that makes manipulating the results difficult. Moreover, we will be adding other policies such as random checks, more due diligence, better conflict of interest management, etc. to further reduce such risks.
  • Of course, I fully expect that if someone is dedicated enough, they will be able to game the system. That’s why we plan on very aggressively punishing any offenders, who violate the our trust, as well as the trust of the community.

In summary, I personally disagree that we are promoting the wrong incentives. I believe that the public discussions, engagement of new donors, and reactions that nonprofits have speak to that as well.

Conclusion

In conclusion, let me again thank GiveWell for their time and analysis. The rest of the Philanthropedia team and I very much agree with the vast majority of the issues that were identified, as well as the suggestions for improvement. We absolutely plan on raising the bar in terms of transparency and quality in order to be able to gain the trust of both donors and experts. Stay tuned for our reorganized webpage that addresses some of the concerns that GiveWell brought up.

GiveWell, in addition to many others, have kindly said that they see a lot of potential in our model. We at Philanthropedia certainly aspire to live up to that expectation, and with more analyses and constructive feedback, we have a real shot of getting there. So thanks again for your leadership Holden and Elie and for helping us get to the next level.

17 comments

  1. Saundra says:

    Philanthropedia,

    I was directed to your site by two different people with whom I had discussed the possibility of crowd sourcing charity evaluations. After I reviewed your site I came away with many of the same questions that GiveWell outlined in their post.
    • Who were your experts?
    • How were they chosen?
    • What criteria do they use to judge charities?
    • Is the criteria meaningful or is it simply things that are easy to evaluate.

    Without answers to these questions it was impossible to evaluate your rating system. Thus, I was very interested to read your response to GiveWell’s post. Unfortunately, instead of helping me understand your processes better, it left me with even more questions than before.

    What criteria does Philanthropedia use to choose experts?
    I am still interested in more information on how experts are chosen than you’ve provided. “When we decide to expand into a new social cause, our first step is to prepare by researching and mapping out the space. In doing so, we identify – to the best of our abilities – all relevant experts in a given field. We list all foundations, nonprofits, and academic institutions and then find the experts’ contact information in these organizations. In addition, in some social causes we include journalists, policy makers, researchers, and other types of experts. The end result is a list of hundreds of experts that we then invite to participate in our surveys.”

    What criteria do you use in choosing these experts? Are these just US based people or do you seek out experts from other developed, developing, and under developed countries? Is there a certain number of years of experience they must have, and at what level? Do they have to have worked for or with that particular organization or done research on it, or can they evaluate it just having heard about it through the grapevine? How large do foundations have to be, can they qualify if they are a small family foundation? Do you extend the invitation to everyone from an organization, if not how do you pick which people to invite? What percentage of experts accept your request and which percentage decline to participate?

    How does Philanthropedia weigh and evaluate expert opinions?
    There is bound to be disagreement, as there are in so many aspects of aid, how do you handle conflicting opinions or information?

    “What is more, not all experts are “equal” in terms of their expertise or experience. And some – most notably foundation professionals and academics – bring disproportionate value to our research. However, we strongly believe in our current inclusive model, which also invites nonprofit executives and senior staff to participate for several reasons.”
    Do some expert opinions carry “disproportionate value”, and if so how is that reflected in the final result?
    What are your “monitoring and curating processes.”
    “Of course, not every invited expert participates today. And of those who participate, we don’t necessarily include all responses from all experts because we do additional sorting of experts during the data collection process.” What additional sorting do you do? What would cause you to disregard the opinion of an expert you invited to participate?
    What are your “internal checks that makes manipulating the results difficult.” How do you “aggressively punish any offenders”?

    What criteria are used to evaluate the charities?
    Although you refer to changing your survey and methodology, you give no indication of what you currently use. When I looked for the answer to this on your website this is what I found.

    “We believe that there is no one metric or measure of nonprofit effectiveness. Therefore, we take a holistic approach to answer this question. Rather than tell the experts which criteria they should be considering when thinking about strong nonprofits, we ask our experts to identify what three criteria they use to judge nonprofit effectiveness. By gathering this information, we are able to capture the views of a variety of professionals and provide unique insight into the best way to evaluate nonprofits in a particular social sector.”

    Do I understand correctly that you have no set criteria? How many and which nonprofits have you evaluated and decided against funding, and why? Do you inform the nonprofits when they have been evaluated but not chosen, and is there an appeals process or the ability to provide proof to rebut negative opinions?

    While looking for the expert evaluation on Alliance for Climate Protection I found that this organization was promoted as having similar strengths as Kiva.
    “Strengths
    Outstanding public visibility.
    ‘They have great media/PR that can help to change hearts and minds on the issue. They also have the cache of Al Gore, and a new investment in a field component.’
    ‘They have good visibility because of linkage to Al Gore.’
    ‘They have a strong television/media presence.’
    ‘They have an excellent website and e-mail alerts on climate protection issues. They have engaging and clever internet ‘ads’ and videos.’
    ‘The ACP produces remarkably compelling videos. I don’t watch TV, but I know that they’re catching the attention on non-activists based on what I hear anecdotally.’
    ‘They are high-profile.’ ”

    Just as Holden found Kiva’s marketing ability to be a questionable factor to be evaluated on, I too question whether the ability “to capture the attention and the imagination of the individual” is any indication of the quality of the nonprofit. McDonald’s owns more playgrounds in the US than any other private or public organization. While this is a great marketing tool, it doesn’t mean that their burgers are better than Wendy’s or that burgers and fries are the healthiest food to have for dinner. Is good PR one of the criteria commonly used to evaluate nonprofits?

    Transparency
    I am a strong advocate for transparency, both with charities and with charity rating systems. I am glad to hear that Philanthropedia is working towards more transparency. I urge you to be clear and honest with donors about your strengths, weaknesses, and areas for improvement. They have the right, and the need, to evaluate the information they are presented with before donating.

    Set an example for the agencies you are evaluating. We cannot ask them to be more transparent if we ourselves are not.

    Good luck with your work improving Philanthropedia’s systems. I look forward to seeing your progress and to having a clearer understanding of your evaluation system.

  2. Deyan says:

    These are terrific questions and an excellent topic for our next blog post! Give me a couple of days!

  3. Thanks for engaging in an open conversation about Givewell’s critique. As you know, I had many of the same questions as Givewell and Saundra when I first took a look at your site a few months ago. After reading your response, I find that I agree with Saundra that it does not entirely answer those questions.

    I have not examined any one issue area in depth, as Givewell has, but simply looking at a random organization highlights a key question: What does Philanthropedia mean by nonprofit effectiveness?

    If you do indeed “believe that there is no one metric or measure of nonprofit effectiveness,” as Saundra quotes your web site, and therefore do not tell which criteria the experts should base their assessments upon, I find it puzzling and nothing less than outrageous. Surely the extent to which organizations are helping people in the ways they intend (outcomes) are at the heart of effectiveness? Marketing, status and fundraising success are meaningless unless the people served are better off.

    Helping people is why nonprofits are in business and it is why so many are looking for web sites like yours to help them find organizations that actually succeed at that difficult task. You have the opportunity to do something meaningful – as opposed to perpetuating existing views about various organizations based on their ability to market themselves.

    If you define nonprofit effectiveness, I think you will find that your choice of organizations and experts will follow naturally. I randomly looked at the Tenderloin Neighborhood Development Corporation, for instance, and found no information in your section about “impact” (other than numbers about people served, which is a far cry from impact, because we know nothing about how well they were served and whether they are better off).

    It strikes me that if you are unable to access data about the extent to which outcomes have been achieved, it is impossible to assess an organization on effectiveness and it should not even be mentioned (unless, of course, you make it clear to your website visitors how your definition of effectiveness is different than the one I suggest). Further, if outcomes are the focus, it follows that researchers and others who can assess organizational claims to outcomes are the most important category of experts and that they must all be provided with access to the same information. Impressions are utterly useless when it comes to the question of outcomes.

    Like others who are contributing to this debate, I believe Philanthropedia is a good idea, but the questions that have been raised are so important that its current value is limited and potentially negative (in that people take the information at face value and direct their dollars to organizations that are ineffective or even harmful). I would be very interested to see a post addressing your definition of nonprofit effectiveness before you get into any other details. That definition is the foundation for everything else.

  4. W.H. Ashbury says:

    The GiveWell criticism went too far

    Philanthropedia is a well thought out and exciting business model that the philanthropy community admires and values. We have long needed an unbiased approach that helps investors and donors identify effective charities and this site has done an excellent job so far. We are all anxious to see their recommendations in other charity categories and their future contributions to help everyone in the philanthropy community develop better ideas and metrics in defining effectiveness for charities.

    The criticism from GiveWell about the site, transparency, and other identified charities went too far. Philanthropedia has to honor the privacy of their experts and for good reason they are not going to list their confidential sources as someone else could copy their idea. Its time that GiveWell take a more positive approach and focus their efforts on identifying effective charities beyond the five small ones they list on their site. What is their specific criteria for selecting their five charities and why don’t they have more recommendations?

    Criticizing competing sites and other resources that are doing excellent work is not helpful to anyone in the giving and philanthropy community. I am a donor and don’t know the leaders at Philanthropedia but use the content on the site to help me identify charities that are working to make the world a better place.

  5. Deyan says:

    I have now posted a summary of the discussion here: http://blog.myphilanthropedia.org/?p=68

  6. Ardis Brevil says:

    Ultimately, I proven the expertise I was looking out for. We have been carrying out assignments on this topic, and for 4 days I preserve acquiring web-sites which are meant to possess what I am hunting for, only to become disappointed making use of the are lacking of what I wished. I wish I could have observed your webpage sooner! I had about 25% of what I utilized to be in have to have of and your web page has that, plus the remainder of what I essential to finish my studying. We have now activated to this web site right here I like that you will observe unique article content that you will be able to hardly discover elsewhere. One particular very good thing, you quite possibly can get hold of nonetheless these types of blogs, make certain you go on! I can no longer see the preferred media. It may be there lots rubbish printed, I bear it no much much more swiftly. A really pleasant weblog and great write-up. I expend nights inside of the globe wide website learning blogs, about tons of many subjects. I must preliminary of all give kudos to whoever founded your internet sites and second of all to you for composing what i can only describe as an post. I honestly consider there’s a ability to writing content or blog site content that only several posses and frankly you may have it. The mixture of informative and exceptional information is completely exceedingly hard to find utilizing the substantial volume of web round the online entire world.Constantly maintain a quite good give excellent benefits!

  7. fireball says:

    Enterprise approach outsourcing services provide a win-win scenario for parties inside the deal. Parents organization can employ their helpful information on development actions. They are able to guide greater profits his or her functional cost is lower. Simultaneously, they could nevertheless boast of very good quality within operate because providers are well educated and trained. They are able to additionally manage to recruit more work as the expense is reduced and may therefore offer much better top quality of labor compared to they might otherwise. On the other hand, companies may trade their technical knowledge for any constant stream at work and decent spend. A sizable amount of much needed tasks are produced.

    • DylanRobbin says:

      , I’ll share what I know about the overlap:None of GiveWell’s staff or Board meberms serve on the staff or Boards Directors of GuideStar or any of the partners. I do serve on the GuideStar Exchange Advisory Board, which has no role in governance.I believe that all of the TakeAction partners (including GiveWell), as well as GuideStar, have received funding from the William and Flora Hewlett Foundation. The Hewlett Foundation has generally encouraged collaboration.We are not aware of any affiliations with GuideStar or the TakeAction partners among any of our donors aside from the Hewlett Foundation.

  8. purkett says:

    Amazing web site. A lot of handy details below. Now i’m mailing the idea to a few associates ans also spreading in scrumptious. And indeed, appreciation for your energy!

  9. aristotledj says:

    Soulful House music 24-7 on BlushFM.

  10. I visited several blogs but the audio quality foor audio songs current at this website is really fabulous.

  11. Wow! After all I got a blog from where I be capable of genuinely take hedlpful data regarding my
    study and knowledge.

  12. I absolutely love your site.. Pleasant colors & theme. Did you make this web site
    yourself? Please reply back as I’m attempting tto create my own personal blog and would
    love to find out where you got thi from oor what the
    theme is called. Thanks!

    Review my website … Plumber pompano beach fl

  13. Rochell says:

    Hello, yup this article is really fastidious and I have learned lot of things from it oon the topic of blogging.
    thanks.

    my blog :: long distgance mmoving companies (Rochell)

  14. With havin so much content do you ever runn into any issues of plagorism or copyright infringement?
    My blog has a lot of exclusive content I’ve either authored myself or outsourced but
    itt looks like a lot of it is popping it up all over the web withnout my authorization.
    Do yoou know aany methods to help protect against contesnt
    from being ripped off? I’d truly appreciate it.

    Here is my web page palm beach accounting firms

  15. Howdy I am so excited I found your blog, I really found you by mistake,
    while I was researching on Google for something else,
    Anyhow I am here now and would just like to say cheers for a tremendous post and a all round
    thrilling blog (I also love the theme/design),
    I don’t have time to look over it all at the
    moment but I have book-marked it and also added your RSS feeds, so when I
    have time I will be back to read much more, Please do keep up the fantastic work.

Leave a Reply


Philanthropedia is a registered 501(c)3 organization. All of your donations are 100% tax-deductible.