The GiveWell Blog

Why it’s important to think through all of the factors that influence a charity’s impact

Charity evaluation is rarely straightforward. Many factors, within a charity’s control or outside of it, can influence the impact a charity has.

This blog post will highlight a case that illustrates how thinking through these factors can lead to surprising information that changes our understanding of a charity’s impact.

Summary

GiveWell recommended a grant to Results for Development (R4D) in May 2016 for its recently-launched program to increase access to pneumonia treatments for children in Tanzania. We thought this program was promising enough to potentially join our short list of GiveWell top charities once we had more information on its impact.

Expanded access to treatments is a factor in reducing child mortality from pneumonia, but not the only factor. We ultimately want to know not just whether more pneumonia treatments are available in Tanzania, but whether fewer children die of pneumonia as a result of R4D’s work. We expect the program to best achieve this impact if pneumonia patients visit health clinics with treatments in stock and are diagnosed and treated correctly.

We learned as we followed R4D’s work that there was limited information available on the accuracy of clinicians’ pneumonia diagnoses. We initially guessed that clinicians were diagnosing pneumonia accurately around 80 percent of the time. R4D collected data on diagnostic accuracy and we learned that the rate of accurate pneumonia diagnosis was actually 18 percent. This caused our estimate of the program’s impact to fall, though it remains in the range that we look for in potential top charities.

This finding highlights why it’s important to think through all of the factors along the path from a charity’s activities to its ultimate impact; if we had just considered whether more treatments were available, we would have missed this part of the story. We’re excited to continue following R4D’s work because of the role it has played in collecting this information to date and our expectation that it will continue collecting information that allows us to estimate its impact on the availability of pneumonia treatments across Tanzania. We expect to consider R4D as a potential future top charity.

In this post, we discuss:

  • The background for GiveWell’s grant to R4D (More)
  • Our plans for assessing the impact of R4D’s program (More)
  • Approaches to measuring R4D’s impact (More)
  • Lessons from this work (More)

Read More

How we evaluate a study

We previously wrote about our general principles for assessing evidence, where “evidence” is construed broadly (it may include awards/recognition/reputation, testimony, and broad trends in data as well as formal studies). Here we discuss our approach to a particular kind of evidence, what we call “micro data”: formal studies of the impact of a program on…

Read More

Rethinking VillageReach’s pilot project

Background Over the past 3 years, VillageReach has received over $2 million as a direct result of our recommendation. VillageReach put these funds towards its work to scale up its health logistics program, which it implemented as a pilot project in one province in Mozambique between 2002 and 2007, to the rest of the country….

Read More

What it takes to evaluate impact

When someone asks me what makes GiveWell different from other third-party charity evaluators, I often answer by listing all the things we’ve done in order to investigate our current top-rated charity, VillageReach. We’ve done extensive background research on international aid and argued for key takeaways such as the overall promise of international aid for donors,…

Read More

In defense of the streetlight effect

In a recent guest post for Development Impact, Martin Ravallion writes: The current fashion [for evaluating aid projects] is for randomized control trials (RCTs) and social experiments more generally … The problem is that the interventions for which currently favored methods are feasible constitute a non-random subset of the things that are done by donors…

Read More

Can choosing the right charity double your impact?

Reader Evan writes: I’ve been thinking about how best to donate to Haiti, and I reviewed some of the materials on your website and found them pretty helpful and persuasive. So thank you! But then my law firm announced that it would match donations to the Red Cross or Doctors Without Borders. Given that, I…

Read More