Reservations about water quality interventions


By

When we started researching water quality interventions, we originally guessed that there was clear evidence that the programs were effective at improving people’s health based on our general intuitions about the programs and our initial read of the relevant Cochrane review’s bottom line. Once we dug into the details of the research, though, we realized that the evidence regarding the program is mixed — a major reason that we don’t currently recommend charities that work on improving water quality.

This post describes that research, and is drawn from our full report on water quality interventions, which provides more detail and citations. The goal of this blog post is to summarize that report in a more accessible form.

The key points of this post:

  • We formed an initial impression that water quality interventions (e.g., chlorine tablets or dispensers; water filters; other types of treatment that clean drinking water in the developing world) improve people’s health, based on our understanding of the dangers of contaminated water. However, digging into the evidence regarding clean water led us to conclude that the evidence base is mixed.
  • We’re not confident that water quality interventions are not effective; rather, we believe that it’s unclear whether or not they’re effective, and so we don’t include water quality among our priority programs.

The intuitive case for water quality charities

Intuitively, providing a method for cleaning water seems like it should be an effective way to improve health. There is little debate that contaminated water causes diarrhea or that the methods used to treat water reduce contamination, and so it’s reasonable to expect that interventions that reduce water contamination ought to reduce diarrhea. And because diarrhea is deadly – the Global Burden of Disease estimated that in 2010, diarrheal disease caused about 10% of child deaths in developing countries – these interventions could be potentially life-saving.

Developing world randomized controlled trials

A number of independent studies have been done on water quality programs. The results of those studies look very promising: They largely find that providing clean water reduces diarrhea rates.

Specifically, 20 randomized controlled trials conducted in developing countries with around 30,000 participants found that interventions to improve water quality were effective at preventing diarrhea, with an average 35% reduction in the odds of getting diarrhea, according to a meta-analysis (study of studies) conducted by the Cochrane Collaboration. Relative to most interventions we’ve considered, twenty randomized controlled trials is an unusually high number of studies.

We typically consider randomized controlled trials to be a particularly strong method of studying global health outcomes. These studies compare a group that receives an intervention – in this case, some method for cleaning water, like chlorine tablets – with a control group that doesn’t. The intervention and control groups are randomly chosen to try to ensure that two groups are similar in every way other than whether or not they receive the intervention. This way, any differences between the groups ought to be due to the intervention.

The Cochrane Collaboration’s findings, then, strongly suggest that the cleaner water is responsible for the fall in diarrhea rates.

Possible weakness of the randomized controlled trials

When we examined these studies more closely, we noticed a potential weakness. As the Cochrane review states: “Only four of the 22 randomised controlled studies, however, were properly blinded”. (22 because the Cochrane review also considers 2 studies conducted in the United States, which we excluded from our analysis.) In other words, the researchers conducting these studies measured diarrhea rates by asking participants to report them, and the study participants knew whether or not they had received the intervention that was being tested.

It’s possible that this knowledge influenced how participants reported diarrhea cases. For example, participants who knew that their water was being treated may have been less likely to report cases of diarrhea, perhaps because they recalled using the treatment and misremembered how many cases they had, or because they anticipated what the surveyor wanted to hear. If that were the case, the study would find an inaccurately low number of diarrhea cases among people that received the clean water intervention, which would result in the study finding an inflated effect of the intervention.

To deal with the issue of participants knowing whether or not they received the intervention (and the possibility that this influences reporting), studies can use “blinding” to disguise who has received the intervention by distributing a placebo (for example, a fake, non-functional water filter) to the control group. When this method is used, participants don’t know whether they have the fake filter or the real one, and so any bias in how they report their cases of diarrhea ought to affect both groups equally, so we can still measure the difference between the two groups without this source of bias affecting the outcome.

Blinded randomized controlled trials should let us know whether the tested intervention is actually improving people’s health, or whether other sources of harmful pathogens are negating any positive effect of the clean water programs. If the latter were true, we may not expect supporting the intervention to have a beneficial impact on the population.

Blinded studies

We identified five blinded studies of water quality interventions that have been conducted in developing countries (which includes some published after the Cochrane review). These studies measured household interventions to clean water; four studied chlorination and one examined the impact of water filters. They did not find a statistically significant effect of the interventions on diarrhea. (Pooling the effects of the studies, we calculated a 5% reduction in the percent of days with diarrhea for children under 5, with a 95% confidence interval ranging from a 6% increase to a 14% decrease, and no reduction for all ages with a 95% confidence interval ranging from a 12% increase to an 11% decrease.)

This suggests that biased reporting from study participants who knew whether or not they were receiving an intervention may explain why the non-blinded studies in the Cochrane review found a statistically significant effect – that is, it suggests that people in the non-blinded studies may have underreported diarrhea rates when they knew that they were receiving a treatment, but when people were unsure whether or not they were receiving a treatment, there was no reporting bias and so no statistically significant effect was found.

Possible weaknesses of the blinded studies

However, these blinded studies have their own weaknesses. In one, the fake filter used for the control group was actually somewhat effective at cleaning the water, meaning that the control group (which wasn’t supposed to be receiving any clean-water benefits) received water that was at least partially cleaned. It’s not surprising, then, that the study didn’t find a noticeable difference between the diarrhea rates of the two groups.

Two of the other blinded studies were conducted in groups that already had low levels of diarrheal disease, which may have made it harder to detect a difference between the control and intervention groups. The other two had small sample sizes, of 112 (the smallest included in the Cochrane review) and 287 people. (Additional problems with the blinded studies are discussed here.)

These problems suggest that the blinded studies are not conclusive in determining that there’s no statistically significant effect of providing chlorine and water filters to reduce diarrhea rates – each of those problems might explain why the blinded studies found no statistically significant effect, even if the intervention itself is actually beneficial.

(There are a handful of other, different studies that we do not find compelling – you can read more about those here.)

How could these interventions fail to reduce diarrhea rates?

Considering the scientific consensus that contaminated water can cause diarrhea, one could reasonably ask whether it’s even plausible that water quality interventions have small or negligible effects (as opposed to large effects). Is it? There is a potential explanation: Other pathways for pathogens, such as from fecal matter on unwashed hands or from food via flies or other people’s hands, especially in an environment with poor sanitation, may cause a significant number of diarrhea cases, so it’s possible that the gains from simply providing clean water could be small if people are regularly infected via other sources. In fact, it’s possible that water quality interventions might have little effect on diarrhea unless all sources of pathogens are eliminated. The possibility that providing clean water will have little effect in an environment with other sources of pathogens is especially concerning because the blinded studies found no statistically significant effect of the intervention.

Bottom line

We concluded in our intervention report (emphasis included for this post),

The strength of the evidence relies on how one chooses to weigh a few, blinded studies with some methodological weaknesses that report no effect against a large number of unblinded studies, some of which may also have some methodological weaknesses, that, on average, report large effects… Overall, we are ambivalent about the effect of water quality interventions on diarrhea. We find plausible theories grounded in the available evidence for both believing that water quality interventions reduce diarrhea and for the more pessimistic conclusion that these interventions do not have an effect.

Based on the ambiguity of the results, and the fact that it’s plausible the interventions on their own do not have an effect, we currently do not consider water quality interventions to be evidence-backed and do not include them among our priority programs. This doesn’t mean we’re confident that the interventions don’t work, but it does mean that we’re not confident that they do.

Note that Dr. Alix Zwane, formerly the Executive Director of Evidence Action (which runs one of our top charities, the Deworm the World Initiative, in addition to a charity which conducts water quality programs, Dispensers for Safe Water) and now CEO of the Global Innovation Fund, disagrees with our conclusion and has a more positive view of the benefits of water quality intervention. Her comments on our report, along with comments by Thomas Clasen, one of the authors of the Cochrane report, can be found here.

Comparison to other programs we recommend

While the evidence for water quality interventions is mixed, it is stronger than for most other interventions that we’ve come across. Depending on what you believe about the weaknesses of the evidence base, it’s arguably as good as the evidence for deworming, which two of our top charities focus on. Deworming also has a complicated evidence base which we have written about extensively, most recently here.

Our initial estimates, however, found that water quality interventions wouldn’t be as cost-effective as deworming programs, and so recently we have not prioritized further research into water quality charities.

That said, because this evidence base is stronger than most, it’s possible that a strong water quality charity could qualify as one of our standout charities, even though we would expect to still recommend our current top charities more highly. A strong water quality charity could even one day become one of our top charity recommendations, especially if our current top charities run out of room for more funding.

GiveWell research plans for 2016


By

Over the past couple of years, we’ve put a lot of effort into hiring and training staff and we now have significantly more capacity to do research than we have in past years. Some of our increased capacity will support the Open Philanthropy Project, which we hope will be a separate organization by the end of 2016; its plans for the year will be discussed on the new Open Philanthropy Project blog. We also expect to have more capacity for GiveWell’s work of finding outstanding evidence-based charities.

At the same time, we have come to believe that the kind of work we’ve recently been doing to find top charities – deeply investigating the most promising-seeming charities we know of, based largely on which interventions they carry out – has limited promise. In past years – and at the beginning of this year – we hoped that these investigations would lead relatively quickly to new top charities. Now, we believe that we’ve already (previously) identified most of the strongest charities by our criteria, and there aren’t many strong candidates left (though there are a few that we continue to investigate, and we remain willing and eager to investigate further promising groups if we come across them). With that in mind, we have begun seeing more potential in other research priorities, such as supporting the development of new organizations and new evidence bases.

A future post will elaborate on why we’ve formed this view. This post focuses on laying out our plans for GiveWell’s research work in 2016, taking this view into account.

In brief, in 2016:

  • We plan to focus much of our capacity on a small number of initiatives that are unlikely to result in new top charities in 2016, but which we hope will lead to new top charities that are competitive with our current top charities in 2017 or 2018.
  • We plan to intensify our work following our current top charities and are tentatively planning to make site visits to distributions funded by the Against Malaria Foundation and work supported by Evidence Action’s Deworm the World Initiative.
  • We are also planning a substantial project focused on the question of whether or not we should recommend that Good Ventures give significantly more than it has in the past to support insecticide-treated nets, arguably the most promising area we know of for substantial additional funding.
  • We also hope to take on additional work (described in detail below) but plan to prioritize this work below the items listed above.
  • We plan to put more staff time into donor outreach than we have in the past and discuss our priorities for that work below.

This plan represents a significant shift from previous years, when our primary goal was improving the list of top charities we published at the end of each year. We plan to write more about the reasoning behind this shift in a future post.

What we’ve done so far this year

In January and February 2016 we:

  • Put significantly more effort into getting input on our plans from non-senior staff than we have in past years. To start, we asked staff and a small number of GiveWell followers to make probabilistic predictions about which charities would become top charities if we investigated them this year. The results of this exercise led to this initial list of possible priorities (listed in no particular order). This list represents the organizations that we would guess are most likely to become top charities at the end of 2016. When we later revised our plan, we held a series of staff meetings to discuss details of the plan and what the new plan might be missing.
  • Had exploratory conversations with several charities on that list and others in the field of global health and development. The goal of conversations with charities was to explain our application process and ask them basic questions about their programs, monitoring and evaluation, and need for additional funding. The goal of conversations with others working in global health and development was to generate a list of additional organizations to contact.

Ultimately, this work made us more pessimistic that prioritizing work on all the organizations listed above would lead to new top charities by the end of 2016, as discussed in the introduction of this post, and we refined our plans for the year as a result. A future post will elaborate on this development.

Top priorities for research

Our top priorities are:

  • Supporting the development of potential future GiveWell top charities: making grants to organizations that could become top charity contenders in the future or supporting research that could lead to more organizations that are a strong fit with our criteria. This work is unlikely to result in new top charities in 2016, but we hope it will lead to new top charities that are competitive with our current top charities in 2017 or 2018. This work might include:
    • Providing early stage funding to organizations that aim to scale up programs with strong evidence of effectiveness and cost-effectiveness. (For example, New Incentives or Evidence Action’s No Lean Season program.)
    • Funding research on programs that are candidates to become priority programs. (For example, this grant to support research on an incentives for immunization program.)
    • Funding organizations that run priority programs to increase or improve their monitoring, or funding a third party to do this monitoring.
  • Considering additional funding for insecticide-treated nets: A significant funding gap exists for insecticide-treated nets, and this gap appears to be as cost-effective an opportunity as any other we have found. This project involves determining whether there are high quality opportunities to provide significantly more funding for insecticide-treated nets than we have in the past. It will involve conversations with the major bednet funders (e.g., Global Fund to fight AIDS, Tuberculosis, and Malaria and the President’s Malaria Initiative) and others familiar with how to identify funding gaps for bednets and what the options are for monitoring distributions. We have also been discussing with the Against Malaria Foundation (AMF) what it would take to quickly scale up AMF’s work. The goal of this work is to identify additional funding opportunities for funding insecticide-treated nets in 2016.
  • Intervention prioritization: quick investigations on a large number of interventions with the goal of finding more priority programs. We have looked at many interventions historically, but regularly learn of programs that we do not know very much about.
  • Current top charities: continuing to follow our current top charities and trying to answer our highest priority unanswered questions about these groups. More on this below.
  • New evidence on deworming and bednets. The next round of follow up on a key deworming study is expected to be available later this year and could make a big difference to our view of deworming. We’re also looking more into the degree to which insecticide resistance may be reducing the impact of bednets.

Other research we will undertake if we have the time to do so

  • Micronutrient fortification charities. Last year, we tried but were unable to find compelling evidence that the Iodine Global Network (IGN) or the Global Alliance for Improved Nutrition (GAIN) had successfully contributed to the impact of salt iodization programs (write-ups forthcoming). We also began investigating Project Healthy Children. We may continue some of these investigations this year and have also reached out to the Micronutrient Initiative and the Food Fortification Initiative.
  • Neglected tropical disease (NTD) charities. We began investigating Sightsavers and END Fund’s work on deworming last year and may continue with those organizations this year and expand the investigations to cover multiple NTDs. We have reached out to the Carter Center and Helen Keller International (HKI) about their NTD programs. HKI declined to participate at this time.
  • Surgery charities. We have had several conversations with organizations that work on cataract surgery and we may reach out to organizations that work on obstetric fistula surgery. Our initial impression from these conversations is that it will be very challenging to understand the impact that these charities’ programs have. We may also consider other surgical interventions (such as trachoma).
  • Other organizations. If organizations apply for a recommendation and seem sufficiently promising, we will aim to review them.
  • Publishing research we largely completed in 2015: updates on standout charities (GAIN, IGN, and Living Goods), interim reviews of charities we began investigating in 2015 (Sightsavers, END Fund, and Project Healthy Children), and intervention reports (folic acid fortification, surgery for cataracts, trachoma and fistula, measles immunization campaigns, mass drug administration for lymphatic filariasis, and “Targeting the Ultra Poor”).

Research we considered but do not expect to undertake

The following investigations are ones that we considered doing this year but don’t currently expect to get to. This could change if some of the higher priority work turns out to be less promising than expected.

  • Mega-charities. We could try to work with one or more large organizations with very diverse programs to figure out how to scale-up work on one of our priority programs.
  • Charities that work on programs that are probably more cost-effective than cash transfers but not by a large enough margin that it seems worth highly prioritizing work on them.
    • Voluntary medical male circumcision. We are interested in talking to PSI, the only major organization we know of working on this program, but do not plan to prioritize this program beyond that.
    • “Targeting the ultra poor” or “graduation” programs.
    • Lymphatic filariasis.
    • Incentives for immunization. We previously funded research on this program and have been working with IDinsight on a cost-effectiveness analysis.
  • Immunization programs. We have put in a fair amount of work into looking for room for more funding for scaling up immunization programs and have largely failed to find opportunities (2012 write-up; recent example).

More detail on potential further research on current recommended charities

One of our top priorities for 2016 is continuing to follow our current top charities and trying to answer our highest priority unanswered questions about these groups. We moved over $100 million to these groups in 2015 and whether we recommend a similar (or greater) level of support in 2016, and how we recommend allocating funds among them, depends on answering: (1) what is our best estimate of the organization’s impact and cost-effectiveness? and (2) how much room for more funding do they have?

Top charities

In past years, we’ve updated our top charity reviews once a year, in November. This year, we plan to refresh these reviews twice, in June and November. As we have at the end of the year, we expect to reconsider what recommendation we make to donors about how to allocate donations amongst our top charities in June.

Summary of our research plans for each of our top charities (note that the strategy documents were written in February):

  • Against Malaria Foundation (AMF). We plan to follow AMF’s progress closely in 2016. Key questions include (a) how quickly is AMF committing funding to new distributions, and (b) can we get a more detailed understanding of how data is collected in pre- and post-distribution surveys. More details here.
  • Schistosomiasis Control Initiative (SCI). The amount of time we spend on SCI this year depends on whether we see a significant improvement in the quality of SCI’s financial information (how it has spent funds, how much funding it holds, and projected expenses). If it does not improve, we will likely deprioritize much additional work on SCI. If it does, we would be interested in exploring the research questions detailed here.
  • Deworm the World Initiative. Of our top charities, we feel that there is the largest gap between what we could know and what we do know for Deworm the World. In particular, we’ve focused on Deworm the World’s work in India, because in the past most unrestricted funds were used in India. Going forward, unrestricted funds will largely be used in new programs. We aim to follow Deworm the World’s progress in new countries closely and to better understand its past work by learning more about its program in Kenya. Details here.
  • GiveDirectly. Our main goals from following GiveDirectly are to see if the quality of monitoring remains high, it is able to enroll new recipients quickly, and we can learn more about the impact of its work with partners to make cash a baseline against which other development programs are judged. Details here.

Standout charities

  • Development Media International. We’re not planning to consider DMI as a possible top charity in 2016. The results from a randomized controlled trial (RCT) of its program that DMI shared last year were not in line with what we would have wanted to see for DMI to become a top charity. More recently, DMI shared some additional results from the RCT (which are not yet public). We believe that taken together these results provide conflicting evidence for DMI’s impact. DMI stands out for its commitment to transparency and rigorous evaluation and we will consider working with DMI to continue to build the evidence base around behavior change through mass media. We see this as a long-term project that is unlikely to result in DMI’s being a top charity in 2016.
  • Iodine Global Network. We are planning to follow up with IGN about a few case studies that IGN thought might provide additional evidence of its impact.
  • The Global Alliance for Improved Nutrition (GAIN) – Universal Salt Iodization (USI) program. We’re not planning to consider GAIN’s USI program as a possible top charity in 2016. We have not been able to establish clear evidence of GAIN successfully contributing to the impact of iodization programs, and think it is unlikely that more work on this will be useful.
  • Living Goods. It’s fairly unlikely that we will consider Living Goods as a possible top charity in 2016. We would revisit this if we were to see significant improvements in the rigor of Living Goods’ monitoring or if we significantly changed our cost-effectiveness estimate for its work.

Plans for donor outreach

We have not historically prioritized outreach at GiveWell, instead choosing to devote staff capacity primarily to our research work. Now, with the addition of new research staff as well as the continued growth of GiveWell’s donor base, we feel it is appropriate to dedicate more capacity to outreach for GiveWell in service of our mission to make our research available to help individuals decide where to give.

In 2016, we plan to have 1.5 staff members devoted to outreach related to GiveWell and the Open Philanthropy Project. Due to this being early on in our outreach work, we’re tentatively planning to reassess our priorities every month for the first half of the year, and then every quarter. As of the publication of this blog post, we expect the following to be top priorities for GiveWell outreach in 2016:

  • Donor calls and meetings. We expect that connecting with individuals who have donated to GiveWell will be an important part of our outreach going forward, although as we’re relatively new to prioritizing this, we plan to survey donors about whether this is something that they find useful. We’re hoping to learn more about the donors who use our work and any questions or feedback they have, as well as to offer an opportunity for donors to stay up to date on GiveWell’s work. More here.
  • Launching a redesigned website. The redesign will largely improve the look and feel of the site with some minor improvements in navigation and content organization.
  • Improving GiveWell’s written communications. This includes:
    • Revisiting and refreshing content on our website (e.g., a recent update to our criteria page) to ensure it’s up-to-date and clearly presented, particularly for individuals who aren’t familiar with our research.
    • Publishing content to our blog, in the hopes of highlighting research and providing additional insight into our values, process, and findings. We will need to put more effort into writing blog posts in order to maintain our previous pace of about one blog post per week, since many types of blog posts that previously appeared here will now be appearing on the Open Philanthropy Blog.

GiveWell’s progress in 2015


By

This post reviews and evaluates last year’s progress on our traditional work of finding and recommending evidence-based, thoroughly vetted charities that serve the global poor. It has two parts. First, we look back at the plans we laid out in early 2015 and compare our progress against them, providing details on some of the most significant accomplishments and shortcomings of the year. Then, we reflect on the overall impact of our traditional work and critically evaluate some of our major strategic decisions. In our next post, we will cover our plans for GiveWell’s work in 2016.

Summary
In brief, when evaluating ourselves against the goals we laid out in early 2015, we feel that we broadly achieved our primary goals for the year while we generally fell short on several of our secondary goals.

The overall impact of GiveWell’s recommendations continued to increase substantially in 2015, as we tracked more than $100 million that was donated to our recommended charities as a direct result of our research.

This self-evaluation post focuses primarily on how we have grown as an organization and the recent strategic decisions we have made since these are most relevant to thinking about the impact of recent GiveWell-focused work.

Our progress in 2015 relative to our plans
In our “2015 plan” blog post, we wrote:

This year, our primary goals are to:

  • Build management and research capacity for GiveWell’s traditional work while further reducing senior staff time (note 1) spent on this work, primarily by reallocating Elie Hassenfeld’s management responsibilities related to GiveWell’s traditional work.
  • Maintain our core research product by completing updates on all eight 2014 recommended charities and determining which of them should be recommended as top charities for the 2015 giving season.

Our secondary goals for 2015 are to:

  • Continue to seek outstanding giving opportunities by reviewing 2-4 new charities and publishing 2-4 new intervention reports.
  • Improve the cost-effectiveness analyses and room for more funding analyses in charity reviews.
  • Finish and launch a redesigned GiveWell website.
  • Make further progress on experimental work to “seed” potential recommended charities.We expect our total output on “top charities” work to be roughly comparable to last year’s, despite a growing staff, because (a) a major focus of the coming year is training, and we expect to trade some short-term efficiency for long-run output; (b) we may be reallocating some capacity from our “top charities” work to the Open Philanthropy Project this year.

We feel that we broadly achieved our primary goals for the year, while we fell short on several of our secondary goals.

Goals that we feel we accomplished include:

  • Building management and research capacity for GiveWell’s traditional work while further reducing the time that Elie spent on this work. Specifically:
    • All GiveWell staff track how they spend time at work. In 2014, Elie spent 36% of his time on GiveWell’s traditional work, and in 2015, Elie spent 26% of his time on this work.
    • In 2014, Elie was primarily responsible for managing all research staff. In 2015, Natalie Crispin and Josh Rosenberg took on research management responsibilities, and at the end of the year, managed 10 research staff between them. Timothy Telleen-Lawton and Eliza Scheffler also took on management responsibilities, primarily for operations-focused staff.
    • Most of the other staff working on GiveWell’s traditional work in 2015 were relatively new. Because they were new, most of the time they spent was focused on training. We discuss staff time allocations in greater detail below.
  • Maintaining our core research product. We completed and published updates on all four top charities from 2014. We also conducted deeper investigations of 3 out of 4 of our standout charities (Global Alliance for Improved Nutrition (GAIN), Iodine Global Network (IGN), and Development Media International (DMI)), as planned, in an effort to determine whether they should be recommended as top charities for the 2015 giving season. We ultimately did not recommend any of these charities as top charities. We did not publish our updates on GAIN and IGN, and those are still forthcoming.
  • Improving our room for more funding analyses in charity reviews, which we wrote about extensively in our November 2015 blog post announcing our updated recommendations.
  • Continuing to make limited but steady progress on experimental work to “seed” potential recommended charities via grants to Evidence Action’s No Lean Season Program, New Incentives, and two randomized controlled trials focused on incentives for immunization.

Areas in which we fell short include:

  • We did not publish updates (or even reach tentative conclusions internally) about any new potential recommended charities, and we published only 2 new intervention reports. We engaged with Project Healthy Children, UNICEF, The END Fund, Episcopal Relief & Development’s NetsforLife® Program, and Sightsavers. (We published a page on Children Without Worms, but it was extremely short.) We published 2 new intervention reports — on vitamin A supplementation and tetanus immunization campaigns — but both of these largely relied on work we had completed in 2014.
  • As discussed in our November 2015 blog post announcing our updated recommendations, we felt more confident in our cost-effectiveness analyses (CEAs) at the end of 2015 than we had in previous years, and we made some of the improvements we planned. Nevertheless, we don’t believe that we made as much progress as we had hoped for when we set the goal to improve our CEAs in early 2015. In particular, key inputs and judgment calls in our CEA were decided on by different staff members at different points in time, which made it more difficult than ideal for individual staff members to understand all details of the CEA. We hope to resolve this issue by making at least one staff member responsible for fully understanding and explaining all aspects of our core CEAs in the future.
  • We did not finish and launch a redesigned GiveWell website. The GiveWell website is large and it took longer than the firm we worked with expected to complete a version that was working properly. The firm we contracted with also built the Open Philanthropy Project website, and in September 2015, we explicitly prioritized completing the Open Philanthropy Project (“Open Phil”) site before the GiveWell site.

Overall, we succeeded in passing significant responsibilities from Elie to others, but we saw somewhat less overall research output than we had hoped for. Part of this was because, in the process of transferring responsibilities, we made some mistaken decisions – i.e., in some cases staff put substantial work into assignments that we ultimately determined were not a fit for them.

Other self-evaluation questions
What was GiveWell’s overall impact in 2015?

The overall impact of GiveWell’s recommendations continued to increase substantially in 2015. We tracked more than $100 million that was donated to our recommended charities as a direct result of our research. Excluding Good Ventures’ giving, we moved more than $30 million to our recommended charities. (More details on our 2015 money moved will be in our forthcoming 2015 metrics blog post.) This was a major increase relative to 2014, when we tracked about $27.8 million in money moved to our recommended charities, with about $13 million coming from non-Good Ventures donors.

Does our impact justify our staff size, in absolute terms?

As mentioned above, in 2015 we moved more than $100 million to our recommended charities. Over the same period, we spent approximately $3.8 million on our operations, of which $1.3 million was spent on GiveWell’s traditional work and $2.5 million on the Open Philanthropy Project.

We previously wrote that we believe that expenses that are 15% of money moved are well within the range of normal, so we feel comfortable with the relative size of our operating expenses at this point.

Is recent growth in staff justified? Has growth in output matched growth in staff?

Staff size has grown significantly over the past few years: from 11 people at the end of 2013 to 32 at the end of 2015. We think it is reasonable to question whether this growth has been justified and productive, in light of the fact that:

  • We haven’t significantly increased the number of new major reports (charities or interventions reviewed in depth). We estimate that we published 4 major new reports in 2013, 5 in 2014, and 6 in 2015 (details on our rough estimate in this Google sheet including year-to-date information for 2016).
  • Most of our 2015 impact (in terms of money moved) likely could have been achieved with reduced research work in 2015. For example, if we had only stayed up-to-date on our past top charities, we would have been in a position to have the same top charities list as what we ultimately published at the end of 2015. The bulk of additional research work that we did in 2015 did not seem to have significant direct impact, largely because the new giving opportunities that we investigated did not lead to new top charities, though it would have been difficult to confidently predict that this would happen in advance (we reflect on whether we chose the best research priorities in 2015 below).

On the other hand:

  • Since 2013, we have greatly expanded the Open Philanthropy Project, which is currently formally housed at GiveWell though we hope to separate the organizations in 2016. Much of our increase in staff size has been for the Open Philanthropy Project.
  • Many of our most senior staff have switched over to working primarily on the Open Philanthropy Project, and senior staff are particularly challenging to replace. We don’t think we could have maintained the same output if we had simply hired one new person for each senior staff member who switched over.
  • Capacity building – hiring, training and evaluating new staff – is itself a time-consuming and long-term project. In addition, increased staff size creates the need for more work in and of itself. For example: (1) in 2015, we outgrew our office and had to find new office space. Finding new office space was only necessary because of our staff size. (2) Because our staff is larger and we have a larger office, we now need an office manager. (Early in our history, we didn’t have an office, or we worked in a shared workspace where someone else played this role.)
  • As our money moved grows, it becomes more worthwhile to be more thorough (as we believe we have been), and it becomes more worthwhile to try to find new top charities even if doing so is a long-term and uncertain proposition. While the work we did in 2015 did not lead to any immediate new top charities, we believe that some of it may have significant returns in the future (e.g., the “GiveWell experimental” work discussed above and the preliminary work that we did on new charities and interventions that we have not finished evaluating).

We previously discussed our reasoning about why we believe it is worthwhile to continue to expand GiveWell’s research capacity despite limited recent returns in last year’s self-evaluation post. This year, we decided to take a more in-depth look at specifically where our increased capacity has gone, while adjusting for staff seniority. Specifically, we roughly estimated the number of full-time, co-founder-equivalent people working on each area in each year. (Co-founder time was given a score of 1 but time spent by newer staff was scaled down significantly; for example, we multiplied time spent by entry-level staff in their first year at GiveWell by 0.1. These figures are far from precise, and are intended only to give a rough guess at our changes in total capacity over time.)

We’ve divided our work into several categories:

  • GiveWell’s traditional research: time spent on evaluating new charities or interventions and updating our work on existing top charities or interventions.
  • Open Phil: time spent on work related to the Open Philanthropy Project.
  • Other output: work that directly furthers GiveWell’s and the Open Philanthropy Project’s missions but isn’t research related. Among other things, this includes work related to donation processing (e.g., opening the mail, entering donations into our database, sending thank you receipts), donor communication (answering donor questions, staying in touch with larger donors), and general outreach (giving talks, speaking to the media, and improving our web content). All of these categories of work have increased significantly as our public profile has grown.
  • Overhead: work necessary due to the increased size of our staff, such as changing offices and having an office manager (as mentioned above).
  • Capacity building: time spent by relatively senior staff to increase our future capacity. This includes time spent (a) recruiting (e.g., reviewing resumes, interviewing and evaluating candidates) and (b) training and evaluating new staff with the goal of increasing future capacity, not generating additional short-term research output. For example, in 2015, Elie estimates that he spent approximately 60% of his time building capacity.
  • Training: time spent by newer staff being trained. These staff members are primarily working on projects where the goal is training and evaluation, not short-term research output.

The table shows that our overall capacity has increased significantly since 2012 (from approximately 3 co-founder equivalents then to 9 today). Most of that increase has gone to Open Phil (3.4 co-founder equivalents), capacity building (.8), and overhead (.9). The number of co-founder equivalents focused on GiveWell’s traditional work has stayed low (and relatively constant) since 2012.

Year GiveWell trad-itional research Other output Capacity building Train-ing Open Phil Over-head Total co-founder equiv. Total staff at year-end
2012 0.9 0.4 0.7 0 0.7 0.2 2.9 5.5
2013 0.8 0.6 0.5 0.3 1.4 0.3 3.9 11
2014 0.9 0.4 0.8 0.6 2.6 0.4 5.7 18
2015 0.9 0.7 1.5 1.1 4.1 1.1 9 32

Overall, we find this picture reasonable. While our work on GiveWell’s traditional research has stayed relatively constant in the short term, we think of it as a major benefit of increased staff that we’ve been able to do this while building the Open Philanthropy Project. And we also believe we’ve been laying the groundwork for future capacity increases, since much of our increased staff time has gone into capacity building itself.

Over time, as new staff train and are able to produce more, we hope that this picture begins to shift and that we can devote more capacity to GiveWell’s traditional research, other output, and Open Phil. Our guess is that we will start to see some of this shift in 2016.

Did we make mistakes in allocating staff capacity in 2015?

We see mistakes in our allocation of staff capacity in 2015. Specifically, we didn’t move as quickly as we should have to move staff from one type of work to another. We had several cases where we continued to try to train staff to work on research projects even though we should have already recognized that they were unlikely to be a good long-term fit for thIs type of research, and would be a better fit for another role. It is generally difficult to predict fit, so we classify this as a relatively easy-to-make mistake, though a costly one, since capacity building is a large time investment for relatively senior staff.

Did we make the right choices about how to spend the time that we had for research (i.e., did we choose the best research priorities)?

In addition to staying up to date on our top charities, we prioritized researching the below opportunities with the goal of maximizing our chances of identifying new top charities.

  • Deworming organizations. At the end of 2014, we felt that deworming was one of the most promising interventions we were aware of, but the giving opportunities we had found to scale up deworming had limited room for more funding. In 2015, we prioritized finding new deworming organizations and began evaluations of the END Fund, Sightsavers, and Children Without Worms.
  • Development Media International (DMI). At the end of 2014, we believed there was a significant probability that we would name DMI a top charity at the end of 2015. We prioritized work on DMI with this in mind, but the results from DMI’s randomized controlled trial were not in line with what they would have needed to be in order for DMI to become a top charity.
  • Micronutrient fortification organizations. We prioritized this area because micronutrient fortification programs reach many people at relatively low cost and they are often backed by strong evidence of effectiveness. We prioritized evaluating salt iodization programs run by IGN and GAIN. We also began an evaluation of Project Healthy Children’s programs and invited the Micronutrient Initiative (MI) to apply for a recommendation. MI declined to participate.
  • Bednet organizations. We invited Nothing but Nets to apply for a recommendation (it declined) and we began an evaluation of Episcopal Relief & Development’s NetsforLife® Program.

We believe these priorities were reasonable, given the amount of time we had available and the information we had at the time. However, as these investigations – and further investigations in 2016 – have had fairly disappointing results (in terms of not resulting in new top charities), we are rethinking our approach significantly going forward. More on this in a future post.

Have we failed to publish materials that we should?

We have struggled to publish information about many research projects that we have essentially completed. For example, we have formed views on several interventions for which we have not yet published intervention reports. One major reason for this is that we often try to resolve almost all of our major questions about an intervention before publishing a report on that intervention. A possible solution would be to publish more materials that explicitly note that we are sharing preliminary views that could be substantially affected by additional research. We plan to experiment with publishing more research about preliminary views we’ve reached in the future.

However, a continuing challenge in publishing our research work is that we have limited management capacity to review and sign off on public write-ups. We hope that building management and research capacity will steadily reduce this bottleneck over time.

New posts on the Open Philanthropy Blog: our early farm animal welfare grants, philosophy of hits-based giving


By

We are now posting content relevant to the Open Philanthropy Project to the Open Philanthropy Blog, rather than the GiveWell Blog. For a period of time, we will be posting notices here when new content appears, in order to ease the transition. We encourage those interested in the Open Philanthropy Project to follow it via RSS, Facebook or Twitter or subscribe to the Open Philanthropy Blog by email.

The newest two posts on the Open Philanthropy Blog are:

Seeking Feedback on Alternative, More Persuasion-Oriented Potential Format for Evidence Reviews


By

This was an April Fool’s joke.

To date, GiveWell writeups have focused on being detailed, nuanced, thoroughly cited, and clear. Something we’ve consciously not prioritized is making them “accessible”: easy and fun to read, emotionally compelling, memorable, and strongly persuasive to casual readers. Writing accessibly isn’t a natural strength of ours, and it can be difficult to reconcile with our more core goals of transparency, nuance, etc. As a result, until recently, we haven’t had the capacity to put in the substantial amount of work it would take to improve the accessibility of our writing.

We’ve recently increased our capacity for outreach, however, and over the last several months we’ve been working on a pilot project: trying an alternative way of writing our evidence reviews that aims for dramatically better accessibility, while hopefully preserving nuance. We’ve partnered with several communications firms (whose names we’ve withheld for now pending feedback on our pilot project, subject to the terms of our agreement) that we consider to be on the cutting edge of knowledge about effective communication and persuasion. These firms collaborated on an in-depth study of what we could learn from the most effective communicators from the worlds of business, entertainment, and politics. We were impressed with their methodology for identifying effective persuaders, and we believe our test writeup effectively integrates practices that have shown to be extremely impressive in mass persuasion.

We would greatly appreciate readers’ feedback on our test writeup, and the extent to which it combines emotional resonance and persuasiveness with maintaining our commitment to accuracy, detail and nuance. Please leave feedback as comments on this post.

View the test writeup

March 2016 Open Thread


By

Following up on our open thread in December, we wanted to have another one.

Our goal is to give blog readers an opportunity to publicly raise comments or questions about GiveWell or related topics (in the comments section below). As always, you’re also welcome to email us at info@givewell.org or to request a call with GiveWell staff if there’s feedback or questions you’d prefer to discuss privately. We’ll try to respond promptly to questions or comments.

If you have questions related to the Open Philanthropy Project, you can post those in the Open Philanthropy Project’s open thread.