Recent Incubation Grants

Charity evaluation is rarely straightforward. Many factors, within a charity’s control or outside of it, can influence the impact a charity has.

This blog post will highlight a case that illustrates how thinking through these factors can lead to surprising information that changes our understanding of a charity’s impact.

Summary

GiveWell recommended a grant to Results for Development (R4D) in May 2016 for its recently-launched program to increase access to pneumonia treatments for children in Tanzania. We thought this program was promising enough to potentially join our short list of GiveWell top charities once we had more information on its impact.

Expanded access to treatments is a factor in reducing child mortality from pneumonia, but not the only factor. We ultimately want to know not just whether more pneumonia treatments are available in Tanzania, but whether fewer children die of pneumonia as a result of R4D’s work. We expect the program to best achieve this impact if pneumonia patients visit health clinics with treatments in stock and are diagnosed and treated correctly.

We learned as we followed R4D’s work that there was limited information available on the accuracy of clinicians’ pneumonia diagnoses. We initially guessed that clinicians were diagnosing pneumonia accurately around 80 percent of the time. R4D collected data on diagnostic accuracy and we learned that the rate of accurate pneumonia diagnosis was actually 18 percent. This caused our estimate of the program’s impact to fall, though it remains in the range that we look for in potential top charities.

This finding highlights why it’s important to think through all of the factors along the path from a charity’s activities to its ultimate impact; if we had just considered whether more treatments were available, we would have missed this part of the story. We’re excited to continue following R4D’s work because of the role it has played in collecting this information to date and our expectation that it will continue collecting information that allows us to estimate its impact on the availability of pneumonia treatments across Tanzania. We expect to consider R4D as a potential future top charity.

In this post, we discuss:

• The background for GiveWell’s grant to R4D (More)
• Our plans for assessing the impact of R4D’s program (More)
• Approaches to measuring R4D’s impact (More)
• Lessons from this work (More)

Grant background

Pneumonia is a leading cause of children’s death worldwide.[1] R4D approached us in 2015 and told us that Tanzania did not have sufficient funding to maintain an adequate supply of pneumonia treatments in the country’s public sector health system.[2] R4D was interested in providing market-shaping technical assistance and catalytic, time-limited funding for pneumonia drug supplies, with the goal of improving the availability of drugs in order to avert more deaths.[3]

We recommended a GiveWell Incubation Grant in May 2016 of $6.4 million to support the first phase of R4D’s scale-up of pneumonia treatments in Tanzania. We thought R4D might meet our top charity criteria once we had more information with which to assess its impact.[4]

How will we know if R4D is reducing deaths from pneumonia?

Funding the purchase of additional pneumonia treatments would seem a simple solution to the inadequate supply of the drugs. But to truly assess the impact of the program on reducing child mortality from pneumonia, we wanted to understand:[5]

1. Would R4D increase the availability of pneumonia treatments?
2. Would clinicians diagnose pneumonia accurately? (We initially estimated 80 percent accuracy in diagnoses in the public and private sectors.[6])
3. Would clinicians prescribe pneumonia treatments to people who needed them?

The second and third questions relate to factors outside of the scope of R4D’s program, which aimed to increase the availability of treatments. However, they play an important role in R4D’s success in reducing deaths from pneumonia.

We were surprised by how difficult it was to answer the second and third questions. There did not appear to be existing data from Tanzania on pneumonia diagnosis and treatment and it was challenging to design effective ways to measure them.

Gathering information

A common story we hear is that many charities do not conduct surveys to verify whether they’re reaching program participants and having the hoped-for impact because:

• donors don’t want to pay for monitoring; or
• charities don’t want to implement monitoring: it’s time-consuming, expensive, and not clearly in demand from donors.

Neither was true in this case. We were interested in funding measurement of the rates of accurate diagnosis and treatment. R4D was interested not only in implementing the measurement, but in taking the lead on developing creative ways to tackle questions about the program’s impact. The latter is rare in our experience. When we have asked charities how they monitor their work, we have often been told that the charity simply knows its program works.

Initial plans

R4D initially planned to use health clinic records to see whether pneumonia treatments were increasing due to its program and whether those treatments were correctly prescribed.[7] However, R4D found in an initial investigation that these records were incomplete and thus did not indicate whether the intended impact was occurring.[8]

R4D considered and decided against a number of other means of assessing whether children who had pneumonia received treatment, such as video-recording clinicians (which was rejected due to anticipated challenges in obtaining consent for patients), surveying patients outside of health clinics (which was rejected due to its cost and anticipated challenges with patient recall), and conducting a high-quality study focused on child mortality (which was rejected due to the high cost of running a sufficiently large study).[9]

Eventual solution

R4D next partnered with IDinsight, another GiveWell Incubation Grant recipient, to develop a new approach to gathering this information.[10] Working with IDinsight, the government of Tanzania, and the Tanzanian national medical school, R4D used lung ultrasounds, which directly tested whether patients with respiratory symptoms had pneumonia, to measure the accuracy rate for clinicians’ pneumonia diagnoses—a neat solution.[11]

The lung ultrasound information yielded surprising results. The rates of accurate pneumonia diagnosis were quite low. Only 18 percent of children with pneumonia confirmed by lung ultrasound were correctly diagnosed.[12]

Getting the full picture

Even that, however, didn’t tell the full story. If we had just looked at diagnostic rates and assumed that incorrect diagnosis leads to incorrect prescription of treatment, then we would have missed another important element of the story: many children who were not diagnosed with pneumonia were still prescribed the right drug to treat pneumonia. When they had the pneumonia treatment in stock, clinicians prescribed it in 46 percent of cases in which they had incorrectly diagnosed a child as having something other than pneumonia. We are unsure why.[13]

Our estimate of the cost-effectiveness of R4D’s pneumonia program fell by 27 percent when we updated it to reflect this new information.[14]

A broader question

The importance of looking for factors that influence impact across a charity’s causal chain, whether under the charity’s control or not, is not unique to pneumonia, nor Tanzania, nor R4D. For example, when we try to understand whether GiveWell top charity Against Malaria Foundation‘s work to prevent malaria by supplying insecticide-treated nets results in fewer people dying of malaria, we think through all the parts of the process that could fail. We aim to do this for our other top charities, as well.

Our estimate of R4D’s pneumonia program’s cost-effectiveness remains in the range that we look for in potential top charities and we’re excited to continue following its work.[15] But without the new information on diagnostic accuracy, we, R4D, and the government of Tanzania might have gotten an incorrect picture of its impact.

We made another grant to R4D in January 2019 to support the second phase of the pneumonia treatment program. We forecast a 40 percent chance that R4D (as a whole) or one of its specific programs (like pneumonia treatment) is a top charity by December 2023.[16] As we move forward, we plan to continue to ask ourselves all of the ways this grant might have more or less impact, as we did before, and as we do in all cases.

Sources

Sources for this post may be found here.

The post Why it’s important to think through all of the factors that influence a charity’s impact appeared first on The GiveWell Blog.

At the end of 2017, we named Evidence Action’s No Lean Season one of GiveWell’s nine top charities. Now, GiveWell and Evidence Action agree that No Lean Season should not be a GiveWell top charity this year, and Evidence Action is not seeking additional funding to support its work at this time.

This post will discuss this decision in detail. In brief, we updated our assessment of No Lean Season, a program that provides loans to support seasonal migration, based on preliminary results Evidence Action began discussing with us in July from a study of the 2017 implementation of the program (hereinafter referred to as “2017 RCT”). These results suggested the program, as implemented in 2017, did not successfully induce migration. Taking this new information into account alongside previous studies of the program, we and Evidence Action do not believe No Lean Season meets our top charity criteria at this time.

Evidence Action’s post on this decision is here.

GiveWell’s mission is to identify and recommend charities that can most effectively use additional donations. While it may be disappointing for a top charity to be removed from our list of recommendations, we believe that adding and removing top charities from our list is an important part of our process. If our top charities list never changed, we would guess we were (a) acting too conservatively (i.e. not being open enough to adding new top charities), or (b) not being critical enough of groups once they’ve been added to our list (i.e. not being open enough to removing existing top charities).

We believe this decision speaks positively of Evidence Action and demonstrates our mutual commitment to updating our views based on new evidence. GiveWell has interacted with hundreds of organizations in our history, and very few have subjected their programs to a rigorous study in the way that Evidence Action did last year and, at smaller scale, in 2014. We’re excited to work with a group like Evidence Action that is committed to rigorous study and openness about results.

Summary

In this post, we will discuss:

• The history of GiveWell and No Lean Season. (More)
• How the 2017 RCT updated our views of No Lean Season. (More)
  • What did the 2017 RCT find? (More)
  • How did we interpret the RCT results? (More)
  • What does the future of No Lean Season look like? (More)
• Conclusion

GiveWell and No Lean Season

No Lean Season provides support for low-income agricultural workers in rural Bangladesh during the time of seasonal income and food insecurity (“lean season”). The program provides small, interest-free loans to support workers’ temporary migration to seek employment. No Lean Season is implemented by RDRS Bangladesh; Evidence Action provides strategic direction, conducts program monitoring, and provides technical assistance, among other functions. Evidence Action developed No Lean Season as part of its Beta portfolio, which is focused on prototyping and scaling cost-effective programs.

GiveWell began engaging with No Lean Season as a potential top charity in 2013, when we began to explore making an Incubation Grant to support its scale-up. We saw No Lean Season as a promising program that lacked the track record to be considered for a top charity recommendation at that time. We describe our initial interest in the program in a February 2017 blog post:

We approached Evidence Action in late 2013 to express our interest in supporting the creation of new GiveWell top charities.

In March 2014, Good Ventures made a $250,000 grant to Evidence Action to support the investigation and scale-up of promising programs.

Since then, Good Ventures has made three additional grants totaling approximately $2.7 million to support the program’s scale-up.

No Lean Season continued to test and scale their program with this and other support. We decided to recommend No Lean Season as a top charity in late 2017. We based our recommendation on three randomized controlled trials (RCTs) of the program. (We generally consider RCTs to be one of the strongest types of evidence available; you can read more about why we rely on RCTs here.)

Two of the RCTs (conducted in 2008 and 2014) indicated increased migration, income, and consumption for program participants. In the third RCT, which was conducted in 2013 and has not been published, the program is considered to have failed to induce migration, potentially due to political violence that year. We discuss the RCT evidence in greater depth in our intervention report on conditional subsidies for seasonal labor migration in northern Bangladesh.

Weighing the evidence, the cost of the program, and the potential impacts, we decided No Lean Season met our criteria to be named a top charity in November 2017. We summarized our reasoning in our blog post announcing our 2017 list of top charities, and noted the risks of this recommendation:

Several randomized controlled trials (RCTs) of subsidies to increase migration provide moderately strong evidence that such an intervention increases household income and consumption during the lean season. An additional RCT is ongoing. We estimate that No Lean Season is roughly five times as cost-effective as cash transfers (see our cost-effectiveness analysis).

Evidence Action has shared some details of its plans for monitoring No Lean Season in the future, but, as many of these plans have not been fully implemented, we have seen limited results. Therefore, there is some uncertainty as to whether No Lean Season will produce the data required to give us confidence that loans are appropriately targeted and reach their intended recipients in full; that recipients are not pressured into accepting loans; and that participants successfully migrate, find work, and are not exposed to major physical and other risks while migrating.

As indicated above, No Lean Season conducted an additional RCT to evaluate its program during the 2017 lean season (approximately September to December), the preliminary results of which indicate the program failed to induce migration. With the evidence from the 2017 RCT, the case for the program’s impact and cost-effectiveness looks weaker.

Our updated perspective on No Lean Season

The 2017 RCT was a key factor in the decision to remove No Lean Season from our top charities list. Below, we discuss:

• What the 2017 RCT found.
• Our interpretation of the RCT results.
• No Lean Season’s future.

What did the 2017 RCT find?

The 2017 RCT was a collaboration between Evidence Action, Innovations for Poverty Action, and researchers from Yale University, the London School of Economics, and the University of California, Davis. In a preliminary analysis shared with GiveWell in September 2018, the researchers did not find evidence for a negative or positive impact on migration, and found no statistically significant impact on income and consumption.[1]

However, the implementation of the program during the 2017[2] lean season and the evaluation of it differed from previous iterations. No Lean Season operated at a larger scale in the fall of 2017 than it had previously, offering loans to 158,155 households, compared with 16,268 households in 2016. Relative to earlier versions of the program, the program in 2017 involved (a) higher-intensity delivery of the intervention (offering loans to most eligible individuals) and (b) broader eligibility requirements (the eligibility rate in 2017 was 77 percent, compared with 49 percent in 2016).[3]

At this point, neither GiveWell, nor No Lean Season, nor the researchers feel we have a conclusive understanding of why the program failed to induce migration. However, No Lean Season and the researchers are exploring various hypotheses about what may explain the failure to induce migration, and they note that some suggestive evidence supports some hypotheses more than others. The researchers have posited several possibilities:

1. The way the program was targeted in 2017 was suboptimal. The Migration Organizers, who survey households for eligibility and offer and disburse loans (more detail here under “Migration Organizers”), may have focused their efforts on the individuals that were seen as most likely to migrate, rather than those who needed a loan to afford migration. The use of loan targets during implementation may have inadvertently incentivized this behavior.[4] If, for example, loan officers mostly made loans to people who would have migrated regardless of receiving a loan, this could have led to the lack of impact on migration found in the study.
2. The 2017 lean season was particularly bad for the program. The researchers note that severe flooding and associated implementation delays in some regions may have caused problems in 2017. The researchers plan to look more closely at the regions that experienced flooding, though they note that they don’t have the data necessary to make experimental comparisons.[5] In addition, a 2013 trial may have failed due to issues that were specific to the year of that trial, such as increased labor strikes.
3. There exists another (currently unknown) reason why this program won’t work at scale. Conditions in Bangladesh may have changed, negative spillovers (harmful impacts for individuals who did not receive loans) may cancel out gains, or pilot villages may have been strategically picked in earlier trials.[6]

The researchers are considering all of these possibilities. After considering various possible theories as well as some non-experimental data (including administrative data and data from a special-purpose survey of Migration Organizers who worked on the program in 2017), they feel that the ‘mistargeting’ theory is the most likely explanation and the explanation most consistent with the analysis.[7]

In scenario (1), No Lean Season may be able to identify and fix the problem. In scenario (2), GiveWell will need to update our estimate of the impact of the program to take into account the fact that periodic program failures due to external factors are more likely than we previously thought. In scenario (3), the program is unlikely to be effective in the future.

How did we interpret the RCT results?

We don’t know the extent to which each of the above explanations contributed to the study not finding an effect on migration.

We used the results of the 2017 RCT to update our cost-effectiveness estimate for the program. Cost-effectiveness estimates form arguably the most important single input into our decisions about whether or not to recommend charities (more on how GiveWell uses cost-effectiveness analyses here). When we calculate a program’s cost-effectiveness, we take many different factors into account, such as the administrative and program costs and the expected impact. We also make a number of educated guesses, such as the likelihood that a program’s impact in a new country will be similar to that in a country where it has previously worked. Below, we describe the mechanism by which the 2017 RCT result was incorporated into our model and how it changed our conclusion.

Prior to this year, we formed our view of No Lean Season based on the three small-scale RCTs mentioned above (conducted in 2008, 2013, and 2014). Each of these RCTs looked at a slightly different version of the program. We believed that the ‘high-intensity’ arm of the 2014 RCT was the version most likely to resemble the program at scale. We thus used the migration rate measured in this arm of the RCT as our starting point for calculating the program’s impact.

The high-intensity arm of the 2014 RCT also had the highest measured migration rate of the three RCTs we assessed, and so we wanted to give some consideration to the less-positive results found in the other two assessments. We applied a small, downward adjustment to the rate of induced migration observed in the 2014 high-intensity arm in our cost-effectiveness model; this was an educated guess, based on the information we had. Our best guess was that the program would lead, in expectation, to 80% of the induced migration seen in the 2014 high-intensity arm.[8]

Now, the preliminary 2017 RCT results show no significant impact on migration rates or incomes. Because this trial was large and very recent, we updated our expectations of the impact of the program substantially, and in a negative direction. Our best guess now is that the program will lead, in expectation, to 40% of the induced migration seen in the 2014 high-intensity arm. Holding other inputs constant, this adjustment reduces our estimate of No Lean Season’s cost-effectiveness by a factor of two.

This reduced cost-effectiveness, along with our updated qualitative picture of No Lean Season’s evidence of effectiveness, led to the decision to remove No Lean Season from our top charities list.

What does the future of No Lean Season look like?

Although they are not raising more funding at this time, No Lean Season has over two years’ worth of remaining funding. We understand that the organization has made changes to the program design in 2018 based on emerging interpretations of the 2017 results, and has collected additional data to evaluate some of the hypotheses which may explain those results (including, for example, a survey of Migration Organizers who worked on the 2017 program). They plan to subject the 2018 implementation round to an additional ‘RCT-at-scale,’ with a particular focus on reassessing the program’s effects on migration, income and consumption, as well as potential effects at migration destinations. They will continue to explore what may have caused the issue in the 2017 program at scale, and to see whether they can find a solution. If they do that, we’ll want to reassess the evidence and the costs to determine whether No Lean Season meets our bar for top charity status. Evidence Action believes we should have the necessary information to reassess starting in mid-2019, based on the results of the RCT conducted during the 2018 lean season and other analyses they perform.

Conclusion

This is the second time since 2011 that we have removed a top charity from our list (prior to 2011, our top charities list was fairly different from today; we made a big-picture shift in our priorities that year that led us to our more recent lists). The previous removal occurred in 2013, when we took the Against Malaria Foundation (AMF) off of our list because we didn’t believe it could absorb additional funding effectively in the near term. AMF was reinstated as a top charity in 2014.

The decision to remove a top charity is never easy. But continuously evaluating GiveWell’s recommended charities is an important part of our work, and we take it seriously. It’s easy to talk about a commitment to evidence when the results are positive. It’s hard to maintain that commitment when the results are not. We’re excited to work with a group like Evidence Action that is committed to rigorous program evaluation and open discussion of the results of those evaluations. Its openness about these results has increased our confidence in Evidence Action as an organization. We look forward to seeing the results from the 2018 RCT in 2019.

Notes

[1] “At this early stage in analysis, we find no evidence that the program had an impact (positive or negative) on migration, caloric intake, food expenditure, or income.” Evidence Action, unpublished summary document, Page 1.

[2] The 2017 RCT studied a period from the fall of 2017 through early 2018.

[3] “This study has two main goals:

1. “A replication of previous findings showing positive impact of incentivized migration on seasonal migration, caloric intake, food and non-food expenditure, income, and food security. Our aim is to estimate impact of a scaled version of the No Lean Season program: intensifying program implementation within branches and expanding the provision of loans to all eligible households.”

Unpublished summary document, Page 1.

[4] “The second set of explanations focus on unintentional implementation changes caused by the change ineligibility, the vastly expanded scope of the program, or other factors. In the most recent round, it is possible that Migration Organizers (MOs) focused their efforts on those households who were most likely to migrate even without a loan to the exclusion of the target population households who need a loan to afford migration. Such behavior may have even been encouraged by the use of targets set by the NGO to manage implementation at such a large scale. We have implemented a qualitative survey to understand the incentives and actions of MOs last year, and are revising our instructions to avoid any possibility of this issue this year.” Evidence Action, unpublished summary document (with minor revision from Evidence Action), Page 11.

[5] “Most notably, the program was affected by severe flooding in many regions, and implementation was subsequently delayed as well. We are still evaluating whether these regions are the ones with the most diminished effects, although we lack the data in control areas to conduct an experimental comparison.” Evidence Action, unpublished summary document, Page 11-12.

[6] “It is possible that what we observe this year may be the true effect of the No Lean Season program when implemented at scale. This may be because conditions in rural Bangladesh have changed since the initial years of success, spillovers at scale cancel out any gains observed in small-scale pilots, or pilot villages were selected because they were most likely to be receptive to the program.” Evidence Action, unpublished summary document, Page 11.

[7] Evidence Action, “Interpretation of 2017 Results” deck and narrative (unpublished)

[8] “This adjustment is used to account for external validity concerns not accounted for elsewhere in the CEA.

“The default adjustment value of 80% is our best guess about the appropriate value, but it is not based on a formal calculation.

“The program at scale takes place in the same region with the same implementers (RDRS and Evidence Action) as the source of our key evidence for the intervention (the 2014 RCT). The program at scale differs in some aspects of implementation, particularly the inclusiveness of the eligibility criteria and the proportion of eligible households offered an incentive. In the 2014 RCT, the subsidy was a cash transfer rather than an interest-free loan, however the 2008 RCT found a similar effect regardless of whether the subsidy was a cash transfer or an interest-free loan.

“There is some evidence (from a 2013 RCT) suggesting that the program may be ineffective when the perceived risk of migrating increases for reasons such as labor strikes and violence. The researchers estimated that these are 1-in-10 year events.

“Additional discussion related to this parameter can be found at https://www.givewell.org/charities/no-lean-season#programdifferentfromRCTs.” 2018 GiveWell Cost-Effectiveness Model — Version 10, “Migration subsidies” tab, note on cell A19.

The post Update on No Lean Season’s top charity status appeared first on The GiveWell Blog.

In July 2018, we recommended a $5.1 million grant to Evidence Action Beta to create a program dedicated to developing potential GiveWell top charities by prototyping, testing, and scaling programs which have the potential to be highly impactful and cost-effective.

This grant was made as part of GiveWell’s Incubation Grants program, which aims to support potential future GiveWell top charities and to help grow the pipeline of organizations we can consider for a recommendation. Funding for Incubation Grants comes from Good Ventures, a large foundation with which we work closely.

Summary

This post will discuss the following:

• Why Evidence Action Beta is promising. (More)
• Risks we see with this Incubation Grant. (More)
• Our plans for following Evidence Action Beta’s work going forward. (More)

Incubation Grant to Evidence Action Beta

We summarized our case for making this grant in a recently-published write-up:

A key part of GiveWell’s research process is trying to identify evidence-backed, cost-effective programs. GiveWell sometimes finds programs that seem potentially highly impactful based on academic research, but for which there is no obvious organizational partner that could scale up and test them. This grant will fund Evidence Action Beta to create … [an] incubator … focused on interventions that GiveWell and Evidence Action believe are promising but that lack existing organizations to scale them.

We have found that which program a charity works on is generally the most important factor in determining its overall cost-effectiveness. Through partnering with Evidence Action Beta to test programs that we think have the potential to be very cost-effective, … our hope is that programs tested and scaled up through this partnership may eventually become GiveWell top charities.

We believe this incubator has the potential to fill a major gap in the nonprofit world by providing a well-defined path for testing and potentially scaling … promising idea[s] for helping the global poor.

For full details on the grant activities and budget, see this page.

We believe that Evidence Action Beta is well-positioned to run this incubator because of its track record of scaling up cost-effective programs with high-quality monitoring. Evidence Action Beta’s parent organization, Evidence Action, leads two of our top charities (Deworm the World Initiative and No Lean Season) and one standout charity (Dispensers for Safe Water).

Modeling cost-effectiveness

In addition to the theoretical case for the grant outlined above, we also made explicit predictions and modeled the potential cost-effectiveness of this grant, so we could better consider it relative to other options. In this section, we provide more details on our process for estimating the grant’s cost-effectiveness.

The main path to impact we see with this grant is by creating new top charities which could use GiveWell-directed funds more cost-effectively than alternatives could.

This could occur:

1. if Evidence Action Beta incubates charities which are more cost-effective than our current top charities, or
2. if Evidence Action Beta incubates charities which are similarly cost-effective to our current top charities—in a scenario in which we have mostly filled our current top charities’ funding gaps. Right now, we believe our top charities can absorb significantly more funding than we expect to direct to them; this diminishes our view of the value of finding additional, similarly cost-effective opportunities. If our current top charities’ funding gaps were close to filled, we would place higher value on identifying additional room for more funding at a similarly cost-effective level.

This grant could also have an impact if it causes other, non-GiveWell funders to allocate resources to charities incubated by this grant. This incubator may create programs that GiveWell doesn’t direct funding to but others do. If these new opportunities are more cost-effective than what these funders would have otherwise supported, then this grant will have had a positive impact by causing funds to be spent more cost-effectively, even if GiveWell never recommends funding to the new programs directly.

We register forecasts for all Incubation Grants we make. We register these not because we are confident in them but because they help us clarify and communicate our expectation for the outcomes of the grant. Here, we forecast a 55% chance that Evidence Action Beta’s incubator leads to a new top charity by December 2023 that is 1-2x as cost-effective as the giving opportunity to which we would have otherwise directed those funds and a 30% chance that the grant does not lead to any new top charities by that time. (For more forecasts we made surrounding this grant, see here.)

We incorporated our forecasts as well as the potential impacts outlined above in our cost-effectiveness estimate for the grant: note that the potential upside coming from other funders is a particularly rough estimate which could change substantially with additional research.

Our best guess is that this grant is approximately ~9x as cost-effective as cash transfers, but we have spent limited time on this estimate and are highly uncertain about it. For context, we estimate that the average cost-effectiveness of our current top charities is between ~3x and ~12x as cost-effective as cash transfers.

Risks to the success of the grant

We do see risks to the success of this grant:

• Few programs may be more cost-effective than our current top charities, or our top charities may remain underfunded for a long time. If Evidence Action Beta fails to identify more cost-effective giving opportunities than GiveWell’s 2017 top charities, or if it only identifies similarly cost-effective giving opportunities while our current top charities remain underfunded, barring any major upside effects, this grant will have failed to make an impact.
• We expect this partnership with Evidence Action Beta to require a fair amount of senior staff capacity. If other means of identifying cost-effective giving opportunities, such as our work to evaluate policy opportunities, end up seeming more promising, this capacity may have been misused.

Going forward

This grant initiates a partnership with Evidence Action Beta toward which we might contribute substantial additional GiveWell Incubation Grant funding in the future. We plan to spend a fair amount of staff time on this ongoing partnership and follow this work closely.

We look forward to sharing updates and the results.

The post A grant to Evidence Action Beta to prototype, test, and scale promising programs appeared first on The GiveWell Blog.

We’ve added the Georgetown University Initiative on Innovation, Development, and Evaluation gui2de‘s Zusha! Road Safety Campaign (from here on, “Zusha!”) as a standout charity; see our full review here. Standout charities do not meet all of our criteria to be a GiveWell top charity, but we believe they stand out from the vast majority of organizations we have considered. See more information about our standout charities here.

Zusha! is a campaign intended to reduce road accidents. Zusha! supports distribution of stickers to public service vehicles encouraging passengers to speak up and urge drivers to drive more safely. We provided a GiveWell Incubation Grant to Zusha! in January 2017 and discussed it in a February 2017 blog post.

For more information, see our full review. Interested donors can give to Zusha! by clicking “Donate” on that page.

The post Announcing Zusha! as a standout charity appeared first on The GiveWell Blog.

In August 2017, GiveWell recommended a grant of $1.3 million to the Centre for Pesticide Suicide Prevention (CPSP). This grant was made as part of GiveWell’s Incubation Grants program to seed the development of potential future GiveWell top charities and to grow the pipeline of organizations we can consider for a recommendation. CPSP implements a different type of program from work GiveWell has funded in the past. Namely, CPSP identifies the pesticides which are most commonly used in suicides and advocates for governments to ban the most lethal pesticides.

Because CPSP’s goal is to encourage governments to enact bans, its work falls into the broader category of policy advocacy, an area we are newly focused on. We plan to investigate or are in the process of investigating several other policy causes, including tobacco control, lead paint regulation, and measures to improve road traffic safety.

Summary

This post will discuss:

• GiveWell’s interest in researching policy advocacy interventions as possible priority programs. (More)
• Why CPSP is promising as a policy advocacy organization and Incubation Grant recipient. (More)
• Our plans for following CPSP’s work going forward. (More)

Policy advocacy work

One of the key criteria we use to evaluate potential top charities is their cost-effectiveness—how much good each dollar donated to that charity can accomplish. In recent years, we’ve identified several charities that we estimate to be around 4 to 10 times as cost-effective as GiveDirectly, which we use as a benchmark for cost-effectiveness. Our top charities are extremely cost-effective, but we wonder whether we might be able to find opportunities that are significantly more cost-effective than the charities we currently recommend.

Our current top charities largely focus on direct implementation of health and poverty alleviation interventions. One of our best guesses for where we might find significantly more cost-effective charities is in the area of policy advocacy, or programs that aim to influence government policy. Our intuition is that spending a relatively small amount of money on advocacy could lead to policy changes resulting in long-run benefits for many people, and thus could be among the most cost-effective ways to help people. As a result, researching policy advocacy interventions is one of our biggest priorities for the year ahead.

Policy advocacy work may have the following advantages:

• Leverage: A relatively small amount of spending on advocacy may influence larger amounts of government funding;
• Sustainability: A policy may be in place for years after its adoption; and
• Feasibility: Some effective interventions can only be effectively implemented by governments, such as increasing taxes on tobacco to reduce consumption.

Policy advocacy also poses serious challenges for GiveWell when we consider it as a potential priority area:

• Evidence of effectiveness will likely be lower quality than what we’ve seen from our top charities, e.g. it may involve analyzing trends over time (where confounding factors may complicate analysis) rather than randomized controlled trials or quasi-experimental evidence;
• Causal attribution will be challenging in that multiple players are likely to be involved in any policy change and policymakers are likely to be influenced by a variety of factors;
• There may be a substantial chance of failure to pass the desired legislation; and
• Regulation may have undesirable secondary effects.

Overall, evaluating policy advocacy requires a different approach to assessing evidence and probability of success than our top charities work has in the past.

Incubation Grant to the Centre for Pesticide Suicide Prevention

CPSP began work in 2016 and aims to reduce deaths due to deliberate ingestion of lethal pesticides. With this Incubation Grant, which is intended to cover two years of expenses, CPSP expects to collect data on which pesticides are most often used in suicide attempts and which are most lethal, and then to use this data to advocate to the governments of India and Nepal to implement bans of certain lethal pesticides.

Research suggests that worldwide, approximately 14% to 20% of suicides involved the deliberate ingestion of pesticides. This method of suicide may be particularly common in agricultural populations. The case we see for this grant relies largely on data from Sri Lanka, where bans on the pesticides that were most lethal and most commonly used in suicide coincided with a substantial decrease in the overall suicide rate; we find the case that the decline in suicides was primarily caused by the pesticide bans reasonably compelling. CPSP’s director, Michael Eddleston, was involved in advocating for some of those bans. Read more here.

GiveWell learned of CPSP’s work through James Snowden, who joined GiveWell as a Research Consultant in early 2017. We decided to recommend support to CPSP based on the evidence that pesticide regulation may reduce overall suicide rates, our impression that an advocacy organization could effect changes in regulations, our view that Michael Eddleston and Leah Utyasheva (the co-founders) are well-positioned to do this type of work, and our expectation that we would be able to evaluate CPSP’s impact on pesticide regulation in Nepal and India over the next few years. We thus think CPSP is a plausible future GiveWell top charity and a good fit for an Incubation Grant.

While deciding whether to make this grant, GiveWell staff discussed how to think about the impact of preventing a suicide. Thinking about this question depends on limited empirical information, and staff did not come to an internal consensus. Our best guess at this point is that CPSP generally prevents suicide by people who are making impulsive decisions.

We see several risks to the success of this grant:

• Banning lethal pesticides may be ineffective as a means of preventing suicide, in India and Nepal or more broadly. The case for this area of policy advocacy relies largely on the observational studies from Sri Lanka mentioned above, supported by Sri Lankan medical records suggesting the decline is partially explained by a shift to less lethal pesticides in suicide attempts.
• CPSP may not be able to translate its research into policy change. This risk of failure to achieve legislative change characterizes policy advocacy work in general, to some extent, and requires us to make a type of prediction that is not needed when evaluating a charity directly implementing a program.
• Banning pesticides could lead to offsetting effects in agricultural production. The limited evidence we have seen on this question suggests that past pesticide bans have not led to notable decreases in agricultural production, but we still believe this is a risk.
• CPSP is a new organization, so it does not have a track record of successfully conducting this type of research and achieving policy change.

To quantify the risks above, GiveWell Executive Director Elie Hassenfeld and James Snowden each recorded predictions about the outcomes of this grant at the time the grant was made. Briefly (more predictions here), Elie and James predict with 33% and 55% probability, respectively, that Nepal will pass legislation banning at least one of the three pesticides most commonly used in suicide by July 1, 2020, and with 15% and 35% probability, respectively, that at least one state in India will do so.

Going forward

We plan to continue having regular conversations with CPSP, and a more substantial check-in one year after the grant was made. At that point, we intend to assess whether CPSP has been meeting the milestones it expected to meet and decide whether to provide a third year of funding. If this grant is successful, we hope we may be able to evaluate CPSP as a potential top charity.

The post Considering policy advocacy organizations: Why GiveWell made a grant to the Centre for Pesticide Suicide Prevention appeared first on The GiveWell Blog.

We’ve long been interested in fistula surgery as a potential GiveWell priority program. However, as with other surgery charities, we have struggled to identify an organization that meets GiveWell’s criteria. Now, we’re working with a group called IDinsight and are excited that we may be able to consider a fistula surgery organization as a potential GiveWell top charity.

Our longstanding interest in interventions to treat fistula can be attributed in part to the popular narrative presented about fistula. The condition, which is often associated with social ostracization, appears to cause a significant amount of suffering, and seems to be treatable. We’re not sure how representative the popular narrative is, but as donors, it has contributed to our continued interest in better understanding this intervention, along with the feeling that surgery charities in general may offer low-cost, life-changing impacts.

Summary

This post will discuss:

• Fistula management, including surgery, as an intervention.
• Our open questions and uncertainty around fistula management programs, particularly their costs.
• Our plans to partner with IDinsight to help answer some of our questions about fistula management.

Surgery charities and GiveWell

We recently published a blog post describing our work to better understand charities that implement cataract surgery programs and to assess whether they might be a fit for a GiveWell top-charity recommendation. As we discussed in that post, surgical interventions in general seem to intuitively appeal to donors due to their potential to offer inexpensive, large impacts on quality of life. However, our uncertainty about surgery charities’ room for more funding and monitoring information has generally led GiveWell to deprioritize research on charities implementing these programs in the past.

Now, as part of GiveWell’s Incubation Grants program to grow the pipeline of potential future top charities and improve our understanding of our current top charities, we’re researching organizations that work on cataract surgery and fistula surgery as potential future top charities.

This post focuses on the latter. Although both interventions are surgical, fistula surgery is distinct in a number of ways from cataract surgery and other interventions GiveWell recommends. Fistula surgery may be a major, invasive procedure. In addition, the largest negative effects of fistula may be psychological, economic, and social, rather than physical.

Fistula management as an intervention

An obstetric fistula, or gynecologic fistula, is an abnormal opening between the vagina and the bladder or rectum. Obstetric fistula is often caused by prolonged obstructed labor, where pressure from the fetus on the mother’s pelvic bone cuts off blood flow to soft tissues, which then die, leading to a hole through which urine or feces may leak through the vagina. Fistula can have physical, economic, and psychological consequences, including social isolation.

Fistula may be treated with surgery to close or partially close the opening. (In some cases, small fistulas may not require surgical treatment; in other cases, the damage may be too extensive for surgical repair.) Job counseling and life skills training aimed at social reintegration may also be part of fistula treatment. “Fistula management” describes all of these interventions, including surgery.

Our very rough cost-effectiveness estimate for fistula surgery is $1,400 per successful surgery performed. The severity of suffering, combined with the cost per surgery, may mean that fistula surgery is in the same range of cost-effectiveness as GiveWell’s current priority programs.

Our open questions

We recently published an intervention report on the evidence for surgery to repair obstetric fistula. We have a number of remaining questions that we’d like to answer before making a recommendation of an organization implementing a fistula management program. Key aspects of this intervention that we’d like to better understand include:

Outreach and cost-effectiveness

It may be challenging to identify and diagnose potential fistula cases. Fistula most often occurs in women who are located in very geographically remote areas or who are too poor to access health systems for delivery care in the event of prolonged obstructed labor. Women located in very remote areas may be hard to reach in general; women with obstetric fistula may be very hard to reach in particular, because they may be more likely to be socially disconnected or unreachable through regular community health systems.

In addition, fistula may not be well known, post-birth complications may be stigmatized, or the symptoms may not be recognized. It may be shameful for women to discuss fistula symptoms; for that reason, we’re unsure whether information about incontinence would be provided to a researcher or surveyor. Even in cases where symptoms are shared, they can be caused by other urological issues, complicating diagnosis of fistula.

We’d like to better understand what the most cost-effective methods of outreach to women with fistula are. We understand that outreach activities have included media outreach; training health workers, nurses, midwives, and doctors in fistula identification to strengthen referrals to treatment; and having women who have had treatment for fistula help identify and encourage women with fistula in their communities to seek medical care. We’re not sure what approach to outreach is most cost-effective.

How cost-effective are fistula management programs?

We’re unsure how the above outreach contributes to the overall cost per patient of fistula treatment. In addition to outreach activities, our impression is that fistula surgery programs often involve training surgeons and the purchase of equipment. We remain very unsure about the costs and benefits of these activities.

Fistula management programs may also involve a number of non-surgical interventions for patients. Some fistula centers offer rehabilitation activities for patients, such as counseling or job training. We’re not sure how common these types of rehabilitation are or what other types of support might entail, or how effective they are at improving social or economic outcomes; this could have a large impact on our overall estimate of fistula management charities’ cost-effectiveness.

What is a ‘successful’ surgery, and what proportion of surgeries are successful?

Obstetric fistulas vary in their size and suitability for surgical repair. For women who do receive surgical treatment, a “successful surgery” may not lead to continence. We’re unsure what proportion of operations successfully close fistulas due to a lack of available data in this space.

Among women whose surgeries do result in continence, we’re unsure of the extent to which that leads to positive social, economic, and psychological outcomes in turn. A major open question is whether operations that succeed in closing a fistula will address any possible social impacts of fistula.

In addition, the prolonged obstructed labor that caused the fistula could also lead to other health complications, such as infertility, scarring, and neurological damage. Repair of the fistula would generally not address these other issues, which could also have social and economic as well as physical consequences.

We are thus very interested in monitoring a) the success of surgery in repairing fistulas and b) the social and economic outcomes for women who receive fistula surgery with or without non-surgical interventions such as counseling. We have not looked closely at the studies that have been done on post-surgical outcomes, although our impression is that most studies in this space have been small. As the economic and social consequences of fistula may be among the greatest burdens of those who have fistula, we are particularly interested in understanding the impact surgery has on those outcomes.

Our plans with IDinsight

GiveWell partnered with IDinsight, an international NGO that focuses on providing clients with information to increase their social impact, as part of our work on GiveWell Incubation Grants. The “GiveWell embedded team” at IDinsight is working with us on a project to better understand the potential of fistula management organizations to become GiveWell top charities.

IDinsight plans to begin this work by focusing on the cost-effectiveness of fistula management, including surgery, as an intervention. It plans to work closely with Fistula Foundation, an organization that funds fistula treatment in Asia and Africa, to understand its costs per surgery for one country to help inform GiveWell’s assessment of the cost-effectiveness of fistula management programs in general. (Even though information from a single country will have limitations in its applicability elsewhere, we think starting at this scale is the right first step.)

If this intervention appears cost-effective—competitive with our current priority programs—following this initial phase of work, then IDinsight will likely work with GiveWell to identify one or more fistula management charities with which to develop and implement a high-quality monitoring system. With additional, high-quality monitoring information, we may better be able to understand whether a fistula management charity should be included in GiveWell’s list of top charities.

We expect this work to take several years, due to the amount of time required to set up and gather data from a new monitoring system, and do not expect this to impact GiveWell’s charity recommendations in 2017.

The post Deciding whether to recommend fistula management charities appeared first on The GiveWell Blog.

We’re often asked why GiveWell doesn’t recommend any organizations that focus on providing surgeries. This post will describe:

• Work we did previously to try to find surgery charities to recommend. In brief, our inability to identify organizations with room for more funding and high-quality monitoring data prevented us from recommending surgery charities in general.
• Our current (rough, preliminary) view that cataract surgery’s cost-effectiveness may be competitive with that of our priority programs, and some of the major open questions we have about our estimate.
• Organizations implementing cataract surgery programs that we’ve spoken with. They run a variety of programs, and our impression is that they do not yet have the type of high-quality monitoring information we’re interested in.
• Our plans to move forward with IDinsight to improve our understanding of cataract surgery as an intervention.

Background

Our impression is that surgical interventions are intuitively attractive to many donors because they seem to offer concrete, low-cost, and life-changing impacts. For example, some organizations performing cataract surgeries claim that each surgery costs approximately $25 (e.g. 1, 2). Fistula surgery organizations, which repair obstetric fistulas, cite costs around $450-$600 per surgery (e.g. 1, 2). These figures would likely be competitive with our cost-effectiveness estimates for our current recommended charities, a critical factor in GiveWell deciding to recommend them. This suggests that we should look into recommending organizations working on these interventions.

We briefly looked at developing-world corrective surgery as a potential priority program in 2010. However, we identified two major challenges to finding top charities working in this space:

1. Room for more funding. We were unsure whether directing additional funding to a charity would cause more surgeries to happen, or if something other than funding—such as the availability of surgeons—was the bottleneck to further surgeries being performed.
2. High-quality monitoring. Many charities don’t conduct the type of high-quality monitoring that we’d like to see, including surgery charities. We’re particularly interested in closely monitoring surgical outcomes due to our view that surgical interventions are complex relative to, for example, the distribution of a mass commodity like a deworming pill or cash. Performing surgery requires skill and we think it’s likely that the quality of surgeries varies. In addition, some surgeries may require longer-term follow-up care, and we’re unsure what the impact is for patients who do not receive this care. Monitoring information thus feels particularly important in our analysis of whether to recommend charities working on surgery.

We deprioritized additional work on corrective surgeries at the time that report was published, although we maintained our interest in potentially recommending charities working in this area. In 2016, we completed an evidence review of cataract surgery and classified it as one of our “priority programs.”

Cataract surgery as an intervention

We believe there is evidence that cataract surgeries substantially improve vision. Very roughly, we estimate that the cost-effectiveness of cataract surgery is ~$1,000 per severe visual impairment reversed.[1]

However, we have not completed an in-depth cost-effectiveness analysis of cataract surgery. We remain highly uncertain about the full costs involved because our current cost estimates are based on literature on the costs of performing surgeries. In our experience, charities’ own budgets (rather than academic literature) have given us the best information about how much an intervention costs.

In addition, we also currently have limited information about the preoperative visual acuity of cataract surgery patients.[2] And we do not have a good understanding of the progression of cataract to blindness—we have looked for this information but have not found it—and so we do not currently incorporate an estimate of the benefits of preventing future blindness in our cost-effectiveness estimate.

Why we don’t focus on trachoma

Another sight-related intervention is surgery to treat trachoma, a bacterial infection commonly transmitted by flies that can result in low vision and eventually blindness from scarring due to eyelashes rubbing on the cornea.
Our impression is that the evidence base around trachoma progression is weaker than cataract surgery, such that we have open questions around the likelihood of the infection progressing from its earlier stages (trachomatous trichiasis, where the eyelashes are rubbing against the cornea) to blindness, and the average age of onset of each. In addition, we have concerns about trachoma recurrence; because trachoma can recur, surgery to repair vision loss from trachoma may be less cost-effective than cataract surgery in the long run. Cataracts do not recur because the surgery replaces the natural lens, although individuals who have had cataract surgery may still experience vision loss due to other causes.

Organizations implementing cataract surgery programs

The organizations we spoke with as part of our investigation into cataract surgery run a variety of programs. Many of them were not directly implementing additional surgeries, but rather were conducting activities such as supporting trainings for surgeons, providing general support to hospitals for eye care interventions, or encouraging more people to access available health services. We have not yet seen from organizations compelling monitoring and evaluation to demonstrate their impact.

Our plans with IDinsight

We’re working closely with IDinsight as part of GiveWell’s Incubation Grants program to grow the pipeline of potential future top charities. IDinsight conducts impact evaluations with the goal of informing decisionmakers, such as governments or NGOs. (More on why we’re partnering with IDinsight in this post.)

We partnered with IDinsight to find a cataract surgery organization it can work with on monitoring and evaluation, as that remains one of the biggest obstacles we’re aware of to GiveWell recommending cataract surgery organizations. (As indicated above, we have spoken with a number of cataract surgery organizations but do not believe any have sufficient monitoring and evaluation information available to inform a GiveWell recommendation.) We expect IDinsight to consider a number of organizations by holding initial scoping calls, and to ultimately focus on working with a single organization that appears most likely to become a GiveWell top charity, although that organization may still not become a top charity. We expect IDinsight to focus on:

• Key monitoring questions, such as measuring pre-operative visual acuity as well as post-operative visual acuity.
• Conducting an evaluation of programs’ causal impact. Does the organization cause more surgeries to happen?

We’re also planning to ask other cataract surgery organizations for more detailed information about the costs of their programs. We hope that between IDinsight’s work and receiving additional cost information, we will better be able to assess whether cataract surgery should continue to be a GiveWell priority program.

Notes

[1] This estimate is on the higher end of the range we calculated, because it assumes additional costs due to demand generation activities, or identifying patients who would not otherwise have known about surgery. We use this figure because we expect that GiveWell is more likely to recommend an organization that can demonstrate, through its demand generation activities, that it is causing additional surgeries to happen. The $1,000 figure also reflects our sense that cost-effectiveness in general tends to worsen (become more expensive) as we spend more time building our model of any intervention. Finally, it is a round figure that communicates our uncertainty about this estimate overall.

[2] Visual acuity is reported as the ratio of the distance at which someone can distinguish a fixed detail relative to a person with “normal” vision. A ratio of 6/6 refers to “normal” vision; a ratio of 6/60 means that someone with impaired vision sees at 6 meters what someone with “normal” vision sees at 60 meters. The World Health Organization (WHO) defines binocular blindness as visual acuity worse than 3/60 in both eyes.

Visual acuity thresholds for surgical eligibility vary. Our understanding is that some portion of cataract surgery is done on individuals whose visual acuity is worse than 6/6 vision, but better than 3/60, and that this proportion likely varies by program and context. For the purposes of our cost-effectiveness estimate, we’ve assumed treatment of patients whose visual acuity is 6/60 or worse.

The post Update on our views on cataract surgery appeared first on The GiveWell Blog.

This post will highlight GiveWell’s work with IDinsight, part of our Incubation Grants program to help grow the pipeline of potential future top charities and improve the quality of GiveWell’s recommendations. We previously highlighted the work of No Lean Season and Zusha!, Incubation Grant recipients and potential 2017 GiveWell top charities. Unlike these organizations, we don’t expect IDinsight to itself become a top charity. Instead, we hope it will help GiveWell support the development of more top charities and increase our understanding of the organizations we recommend.

IDinsight is an international NGO that aims to help its clients develop and use rigorous evidence to improve social impact. GiveWell is partnering with IDinsight to support organizations’ development of monitoring and evaluation information of the type we’re interested in. This is the first partnership of this kind for GiveWell.

Summary

Working with IDinsight is a major part of our GiveWell Incubation Grants program. We hope that IDinsight can (a) conduct randomized controlled trials (RCTs) of interventions that seem promising but have little evidence to back them up, (b) work with promising organizations or existing top charities to assess and improve their monitoring systems, and (c) conduct additional research work that could inform our recommendations (e.g., additional site visits or surveying beneficiaries about their preferences).

We don’t expect IDinsight’s work to influence our top-charity recommendations in 2017. We may have information from this partnership to inform our 2018 recommendations, and we expect it to influence our 2019 recommendations.

Background

We believe IDinsight fills a unique role in the development sphere. Unlike academics, who may be incentivized to focus on advancing the academic literature, IDinsight focuses on providing decision-relevant data and assessments; it serves nonprofits and policymakers who have to decide whether to implement intervention A or B, by, e.g., producing quick, low-cost, randomized controlled trial evidence. In addition to these types of “decision-focused evaluations,” IDinsight sets up “embedded learning partnerships” within governments and NGOs to answer priority policy questions using a broad suite of data and evidence tools. We think this kind of work is highly valuable and relevant to GiveWell’s mission of finding and identifying cost-effective, evidence-backed programs and charities.

We came across IDinsight early on in our GiveWell Incubation Grants program (then known as “GiveWell’s experimental work”) when we searched for organizations that could help us “bridge the gap between research and implementation”. GiveWell made its first Incubation Grant to IDinsight in September 2014. Since then, we have provided additional support to IDinsight in June 2016 to expand their general operations (in the hopes this would lead to more evidence and organizations of the type GiveWell is interested in recommending) and in October 2016. The latter grant is to support the creation of an IDinsight “embedded team” at GiveWell, and is the focus of this post. We recently recommended an additional grant to support the scale-up of the embedded team’s work.

Funding for IDinsight’s work with GiveWell comes from Good Ventures, a large foundation with which GiveWell works closely, and which has supported our Incubation Grants program.

Goals of our partnership with IDinsight

We hope IDinsight can help fill a gap in GiveWell’s evaluation process. One of our core criteria for recommending charities is high-quality monitoring and evaluation information to demonstrate a charity is having impact. Our impression is that many organizations—likely the majority of charitable organizations—do not have this type of information; in some cases, we think it’s likely that even excellent organizations may not be collecting this type of information due to the high cost, both monetary and in staff-hours, it requires.

We hope IDinsight will help fill this gap by working with promising potential top charities to develop these kinds of monitoring systems and/or to complete an impact evaluation of their work. We’re also interested in IDinsight working to strengthen monitoring systems for some of GiveWell’s current top charities, to help us improve our understanding of their impact.

Initial plans for the IDinsight “embedded GiveWell team”

We’re still early in exploring all of the possible ways in which GiveWell and IDinsight may work together. Two projects are most likely in the near term:

1. Helping GiveWell identify and develop a potential top charity working on cataract surgery. To do this, we think IDinsight will most likely build and implement a monitoring and evaluation system for a promising cataract charity. In August 2016, we published an intervention report announcing our view that cataract surgery was a GiveWell “priority program.” In other words, we believe that the strength of the evidence for and potential cost-effectiveness of the program is competitive with the other interventions we recommend, like distributing nets to prevent malaria and providing direct cash transfers to very poor households, and that we’d be interested in considering charities implementing cataract surgery programs for a top-charity recommendation.

   However, in our early conversations with organizations working on cataract surgery programs, our impression was that they didn’t yet have the type of monitoring and evaluation information on surgical outcomes and their impact on the number of surgeries carried out that we’d like to see as part of our charity review process. That’s why we’re working with IDinsight to support the identification or development of a GiveWell top charity working in this space.

   IDinsight is also undertaking a similar project focused on organizations providing surgeries to correct obstetric fistulas.

2. Conducting an impact evaluation of New Incentives’ immunization incentives program. We may ask IDinsight to work with New Incentives, a GiveWell Incubation Grant recipient and potential future top charity, to run a randomized controlled trial of its pilot program to provide conditional cash transfers to incentivize routine immunizations of infants.

   In February 2017, IDinsight traveled to Nigeria to visit New Incentives and observe their immunization incentives program as part of this project. IDinsight’s notes from the trip are available here.

Other potential projects with IDinsight include, but are not limited, to:

• Improving the Against Malaria Foundation (AMF)’s monitoring. AMF is currently our top recommendation to donors, although we in 2016 we wrote about weaknesses in its monitoring. IDinsight may work with AMF to improve its monitoring standards.
• Conducting an RCT on GiveWell Incubation Grant recipient Charity Science: Health‘s work. Charity Science: Health is a young organization that provides SMS reminders for vaccinations in India. We do not yet have a view on this intervention and are interested in potentially working with IDinsight to better understand Charity Science: Health’s work in this space and its impact on immunization rates.
• Informing subjective value judgments in GiveWell’s cost-effectiveness model. Cost-effectiveness plays a major role in GiveWell’s top-charity recommendations, and we rely on a number of highly uncertain and subjective tradeoffs between increasing income and various health outcomes in order to compare charities that work on different programs. Better understanding the preferences of individuals impacted by the work of the charities we recommend would improve our cost-effectiveness outputs. IDinsight may design and pilot a survey of beneficiaries on how they compare certain health and non-health interventions.

We’re looking forward to seeing what we learn from this partnership.

The post Why GiveWell is partnering with IDinsight appeared first on The GiveWell Blog.

This post will discuss Zusha!, a 2017 GiveWell top charity contender and GiveWell Incubation Grant recipient. We previously highlighted No Lean Season as a potential 2017 top charity originating from our Incubation Grants work.

GiveWell first learned about Zusha! in 2013 following our publication of a shallow investigation into road safety. This month, Good Ventures made a GiveWell Incubation Grant of $900,000 to support the Georgetown University Initiative on Innovation, Development and Evaluation (gui2de) for work on Zusha!. Also this month, two GiveWell staff members visited Zusha! in Nairobi to learn more about its work. We plan to share additional details from their site visit in the future; this post is meant to provide a higher-level overview of Zusha! as a potential GiveWell recommendation.

Road safety campaign

Car accidents are a major cause of preventable death and disability around the globe, killing approximately 1.25 million people each year and injuring an additional 20 to 50 million. On the current trajectory, the World Health Organization (WHO) projects that road traffic crashes will be the 7th leading cause of death globally in 2030. The problem is particularly pronounced in low- and middle-income countries, which account for 90% of all traffic deaths, despite having ~50% of the world’s vehicles, according to the WHO.

Zusha! is a road safety campaign that targets unsafe drivers of public service vehicles. The campaign distributes stickers for buses with messages encouraging passengers to speak up and urge drivers to drive more safely—”Zusha” means “protest” in Swahili. Drivers are incentivized to keep the stickers in their vehicles via enrollment in a weekly lottery with cash prizes. The goal is to reduce traffic deaths and injuries. gui2de has primarily worked in Kenya.

Zusha! is part of gui2de. Professors James Habyarimana and William Jack have conducted two randomized controlled trials (RCTs) of the program in Kenya: first, a small pilot study of ~2,400 vehicles followed by a larger study of ~12,500 vehicles. The researchers found large, statistically significant effects of the program in reducing the number of accidents for vehicles in the treatment group.[1] With support from a Development Innovation Ventures (DIV) grant from USAID, Zusha! scaled up in Kenya following the second RCT. As of March 2016, Professors Habyarimana and Jack estimated that the campaign was reaching 25,000 minibuses and larger buses, out of roughly 40,000 in the country.

gui2de is running three additional RCTs on this program in Rwanda, Tanzania, and Uganda.

A note on terminology in this blog post

In this blog post, we generally refer to the program we’re interested in as Zusha! to distinguish it from gui2de‘s other programs (not related to road safety). However, Zusha! is only the name of the road safety campaign in Kenya; the road safety campaigns in Rwanda, Tanzania, and Uganda have other names. Although the Kenya campaign is the one we’re most knowledgeable about, it’s possible that a GiveWell top charity recommendation would include gui2de‘s road safety work in other countries. We’ve used the term Zusha! in this post for simplicity.

Potential future top charity

We’re interested in Zusha! as a potential future top charity due to the potential strength of the evidence base and cost-effectiveness.

We believe the evidence for Zusha! is compelling. The pilot study finds that driving accidents decreased by a half to two-thirds and the larger Kenya RCT finds that driving accidents decreased by between one-quarter and one-third. These effects seem surprisingly large to us, and we are interested to see whether the intervention will find similar effects in future RCTs. In our most up-to-date cost-effectiveness calculation, we estimate a cost of ~$13,000 per road accident death averted (including injuries and incorporating discounts to account for whether the studies would be likely to replicate and questions around external validity).

GiveWell’s current estimate is that the cost-effectiveness of Zusha! is comparable to the Against Malaria Foundation, one of our top charities, and about 3-4x as cost-effective as direct cash transfers, a baseline we use for comparing interventions, although this may change as we incorporate additional inputs. We incorporated age weights into this estimate that reflect the older average age of passengers on vehicles, relative to average age of people whose deaths are averted by AMF-distributed nets, and approximate GiveWell median staff values for averting an adult death.

GiveWell Incubation Grant

Good Ventures’ recent grant to gui2de is intended to:

• Allow gui2de to continue operating at scale in Kenya and collect higher-quality monitoring data of that work. Strong monitoring data, such as information to demonstrate the stickers are being distributed to the intended vehicles or that the stickers remain in use over time, is a necessary component for a top charity recommendation and is one of our biggest open questions about Zusha!.
• Increase the sample size of the RCT in Uganda by ~50%, improving the study’s power and making it more likely the results will inform our views.
• Potentially improve the quality of data collection for the RCT in Tanzania.
• Provide funding to enable gui2de to continue its ongoing work through the end of 2017, when GiveWell might potentially name Zusha! a top charity, in which case we would expect to direct it substantial funding. Enabling gui2de to continue operating in Rwanda, Tanzania, and Uganda for six additional months would also allow faster scale-up if the RCT results are positive.

A write-up on the February 2017 grant is forthcoming. It will be published here.

Our open questions

We have several open questions about Zusha!‘s work that will be key in helping us decide whether to recommend Zusha! as a top charity:

• We expect that results from the three pending RCTs in Rwanda, Tanzania, and Uganda will substantially affect our view of the likely impact of the program, although we don’t expect to have full results from all three RCTs by the end of 2017.
• Zusha! researchers found a nearly statistically significant impact for the placebo intervention (stickers that had messages like “Travel well”) in the second Kenya study. This finding casts uncertainty on the mechanism by which the intervention works and whether the intervention is having an impact. Additional RCTs may help fill out our understanding.
• Our cost-effectiveness analysis suggests that Zusha! is competitive with—but not far better than—our current top charities, at ~3-4x as cost-effective as cash transfers. GiveWell’s cost-effectiveness analyses tend to become worse (less cost-effective) as we add new inputs and adjustments. Our estimate of Zusha!‘s cost-effectiveness already became significantly worse when a GiveWell Research Analyst, Leon Zhang, identified a mathematical error in one of the studies published on Zusha!‘s program. It’s possible we will conclude that Zusha! is not as cost-effective as our other top charities after spending additional time on this.
• Provision of high-quality monitoring information to demonstrate that the stickers are being used in buses over time. We understand from our recent site visit that Zusha! tentatively plans to do three types of monitoring in Kenya going forward: At National Transport and Safety Authority (NTSA) inspection centers, bus parks where passengers are picked up, and via the lottery. We have questions about the implementation of these processes, but our impression is that Zusha! is working to significantly improve its monitoring, and we expect to have more information by the end of the year.
• Zusha! is a behavioral intervention. Over time, people may get used to seeing the stickers, causing the effect to diminish. We currently have limited information on the extent to which this has occurred or may occur in the future. We hope additional information about long-term impacts of the program will enable us to assess this question over time.

Path to GiveWell top charity

We publish our updated top charities list in November. By then, we expect to have new monitoring information from Kenya as well as preliminary RCT results from Tanzania. We also expect to have partial results from the Uganda RCT. (We do not expect to have results from the Rwanda RCT.) We guess that the information we will have by late 2017 should be sufficient to assess Zusha! for a potential GiveWell recommendation.

Notes

[1]

• Results from the pilot study (published 2010): “Our results indicate that insurance claims fell by a half to two-thirds, from an annual rate of about 10 percent without the intervention, and that claims involving injury or death fell by 60%.” Habyarimana and Jack 2010, p. 1
• Results from the larger study (published 2015): “Overall, the stickers reduce insurance claims of matatus assigned to treatment groups by between one-quarter and one-third on an intent-to-treat (ITT) basis. Among the roughly 8,000 vehicles in the treatment groups, the reduction was 25%, and we estimate that about 140 accidents were avoided per year, and about 55 lives were saved annually.” Habyarimana and Jack 2015, p. 1

The post Why we’re considering <em>Zusha!</em> as a potential 2017 top charity appeared first on The GiveWell Blog.

In recent years, we’ve added a new source for potential GiveWell top charity recommendations: GiveWell Incubation Grants. This post will highlight a GiveWell Incubation Grant recipient, No Lean Season, that we see as a top charity contender for 2017.

GiveWell has traditionally identified our top charities through our standard process, during which we examine a charity’s track record and funding needs. Our goal with GiveWell Incubation Grants, outlined in an earlier blog post, is to grow the pipeline of potential future top charities, in part by supporting organizations at an earlier stage than we would traditionally consider them for a top charity recommendation. We generally expect there to be a lag of a few years between receiving an Incubation Grant and being considered for a top charity recommendation.

Good Ventures, a large foundation with which we work closely, funds GiveWell Incubation Grants. Good Ventures made its first Incubation Grant to No Lean Season in 2014 and we now believe it is a top charity contender when we update our recommendations at the end of 2017. We’re planning to highlight another 2017 contender, Zusha!, in a future post.

Seasonal income support program

No Lean Season offers subsidies to low-income agricultural workers in Bangladesh to incentivize them to temporarily migrate from rural areas to urban areas, where they may earn higher wages seasonally. These subsidies (which may be made as grants or loans) are around $8-19 USD and cover travel costs and a couple days of food. Follow-up studies found that individuals who once received an incentive to migrate chose to do so again—without a subsidy—at a higher rate than would otherwise be expected, suggesting they found migrating to be useful.

The below infographic from No Lean Season shows a high-level overview of the intervention (click for detail):

Evidence Action, the parent organization of GiveWell top charity Deworm the World Initiative, started No Lean Season as part of Evidence Action Beta, its program to test interventions that could be significantly scaled up.

How we decided to support No Lean Season

We approached Evidence Action in late 2013 to express our interest in supporting the creation of new GiveWell top charities.

In March 2014, Good Ventures made a $250,000 grant to Evidence Action to support the investigation and scale-up of promising programs. Since then, Good Ventures has made three additional grants totaling approximately $2.7 million to support the program’s scale-up; the write-up for the most recent grant, made in December 2016, is forthcoming.

No Lean Season as a GiveWell top charity contender

We assess potential GiveWell top charities along four criteria: evidence of effectiveness, cost-effectiveness, transparency, and room for more funding. No Lean Season appears as a plausible contender when reviewed along these dimensions. We plan to spend significantly more time reviewing No Lean Season this year as we move forward in our top charity review process and update our views on its work.

Evidence of effectiveness. A number of randomized controlled trials (RCTs) have studied the effects of seasonal income support in northern Bangladesh, where Evidence Action is scaling up the program. Trials conducted in 2008 and 2014 found significant effects on household expenditures and income, respectively, during the relevant season. In addition, later follow-ups of households that were incentivized to migrate found that they did so again at higher rates, even in the absence of a continued incentive; Mushfiq Mobarak, a Yale economics professor and a lead researcher on No Lean Season, estimates that the effects persist for three additional years. However, the 2008 and 2014 studies did not measure directly comparable or combinable outcomes, so we can’t compare them or combine the results. We take these RCTs as strong evidence that No Lean Season’s seasonal migration subsidies lead to improved economic outcomes in northern Bangladesh.

Analysis is not yet complete for a separate RCT, conducted in 2013, a year when labor unrest was unusually high (see Figure 1). Given the possibility of mitigating circumstances, we’re unsure how informative the 2013 RCT results will be to predicting future success of the program.

Potential risks of the program could include negative impacts at the destination labor market (e.g. on job availability or food prices) or vulnerability of migrants or family members left at home after migration. As of January 2016, our impression was that Evidence Action planned to monitor possible negative effects on the destination labor market; Evidence Action said it had not found indication in surveys that migrants or their families were less secure (see p. 4).

Evidence Action is planning to run an RCT at scale during the 2017 lean season. Due to our current best estimate of the program’s cost-effectiveness and expectation that No Lean Season will collect and share high-quality monitoring data from its 2016 work (discussed below), we think the evidence base may be sufficient for the organization to qualify as a top charity at the end of 2017, before results from this RCT are available.

Cost-effectiveness. We currently estimate that No Lean Season is between 5-14 times as cost-effective as direct cash transfers, a baseline we use for comparison among global health and development interventions. At scale, we estimate that individuals will experience a consumption benefit of $15 for every $3 No Lean Season spends. These benefits and costs are averages over the population that is eligible for and offered the program; we believe that the benefits are actually larger for households that send a migrant and smaller for households that don’t. We’re very uncertain about the baseline per capita lean season consumption in this population, but for comparison purposes, we estimate it at roughly $116 for the entire 5-month lean season. We also expect, based on previous studies, remigration without further incentive for about two future years, as well as at least one additional migration during the lesser lean season.

Our estimate of No Lean Season’s cost-effectiveness is in the range of our current top charities. We believe that the four deworming charities we recommend are ~4-10x as cost-effective as cash transfers, and the two charities we recommend for their work to prevent malaria, the Against Malaria Foundation and Malaria Consortium, are ~4x as cost-effective as cash transfers. However, our cost-effectiveness estimates typically become worse—the cost-effectiveness decreases—as we spend more time on our analysis and incorporate additional inputs and discounts. We expect this is likely to occur with our current estimate of No Lean Season’s cost-effectiveness, as well.

Monitoring. We have not yet reviewed No Lean Season’s monitoring. However, we are quite familiar with its parent organization, Evidence Action, as a result of our recommendation of the Deworm the World Initiative as a top charity since 2013, and our previous reviews of its work, and feel confident Evidence Action will share its monitoring of No Lean Season based on its track record. We expect this monitoring to be of a high quality.

Organizational strength and transparency. We have a positive view of Evidence Action as an organization, based on our significant experience communicating with its staff. Our impression is that Karen Levy, Evidence Action’s Director of Global Innovation and Beta, played a key role in scaling the Deworm the World Initiative, suggesting the organizational capacity exists to similarly scale a program like No Lean Season. We also believe that Evidence Action, based on its track record and our experience, will operate No Lean Season transparently.

Room for more funding. Evidence Action currently estimates that No Lean Season could productively use approximately $16 million over five years in Bangladesh and Indonesia, and additional funding to expand to India and Ghana; this estimate may be adjusted in the future.

We remain unsure whether this program will be successful in locations beyond Bangladesh. If it isn’t a good fit in other locations, No Lean Season’s overall room for more funding could be quite limited.

Progress to date and future plans

As of early 2016, Evidence Action planned to scale up its program in Bangladesh to offer a total of 16,000 subsidies and reach 9,000 households with its implementing partner, RDRS Bangladesh, in 2016, the first of a four-year scale-up. By 2019, No Lean Season provisionally plans to offer ~295,000 subsidies and reach ~165,000 households in Bangladesh (see p. 5 here; No Lean Season staff plan to update these figures going forward).

Evidence Action is also exploring the possibility of working in locations beyond Bangladesh. It visited Zambia and Malawi in 2014 to assess whether the program might help alleviate seasonal hunger in those locations. Our understanding is that Evidence Action is not planning to expand in Malawi based on its findings. We believe Evidence Action is also not planning to scale up in Zambia.

Mobarak, the No Lean Season researcher, has conducted two research studies in Indonesia, which he says suggest the country may have similar underlying conditions to Bangladesh. As of August 2016, Evidence Action was also considering expanding into India, particularly states close to Bangladesh, although it had not yet done any research there. Evidence Action also considered Ghana as a potential future location, although we are not aware of concrete plans to begin implementation there.[1]

Path to GiveWell top charity

By November 2017, we expect to see results from No Lean Season’s first year of a four-year scaling effort (the September-December 2016 seasonal effort to reach 9,000 households described above). This, combined with the fact that there are multiple rounds of randomized controlled trials in the past and a large forthcoming RCT at scale, maybe sufficient for No Lean Season to qualify as a 2017 top charity.

Notes

[1]
“Indonesia

Mushfiq Mubarak has completed two research studies related to No Lean Season in Indonesia:
1. An exploratory study, similar to those done in Zambia and Malawi.
2. A small-scale pilot in West Timor conducted by the Southeast Asian office of the Abdul Latif Jameel Poverty Action Lab (J-PAL). This study was similar to a previous trial in Bangladesh, but was not a randomized evaluation.

The studies’ initial results are promising. Indonesia appears to have similar underlying conditions to Bangladesh, including: lean season migration; availability of jobs in urban areas; and migration from rural to urban areas and between islands.

India

No Lean Season is also considering expanding into India but has not yet conducted research there. Its first activity in India would likely be a small-scale project, such as a pilot in one village. The underlying conditions in Indian states bordering the Rangpur region of Bangladesh are similar to those that exist in Rangpur.

Ghana

No Lean Season is considering expanding into Ghana but does not yet have concrete plans to initiate a project there. In general, it does not intend to expand into new countries until it has sufficient capacity to do so.”

From a conversation with Dr. Karen Levy and Guillaume Kroll, August 2, 2016, p 1.

The post Why we’re considering No Lean Season as a potential 2017 top charity appeared first on The GiveWell Blog.

GiveWell Incubation Grants have become an increasingly substantial part of our work, and our impression is that not everyone who follows GiveWell is familiar with this program. This blog post is intended to (a) briefly explain and outline our main goals and expectations for this work, and (b) share some updates on promising organizations that have been supported by Incubation Grants.

The goal of GiveWell Incubation Grants (previously known as GiveWell’s experimental work) is to support the development of future top charities and improve our understanding of our current top charities. We plan to do this in a few ways (not an exclusive list):

1. Increasing the body of evidence around potential top charities and priority programs;
2. Providing early-stage support for new organizations;
3. Supporting improved monitoring and evaluation for potential or current top charities.

Good Ventures, a foundation with which we work closely, has funded the grants made as part of this work, which are listed here.

Promising investigations

Due to the nature of this support—early-stage funding, intended to allow an organization to develop a stronger track record or to collect more evidence on a promising program—we don’t expect Incubation Grants to produce new top charities over very short time horizons. We expect there will be, in many cases, a period of multiple years between a grant and an organization or intervention being considered a potential top charity or priority program.

This post highlights grants that we don’t expect to lead to top charities before 2018. It should provide a reasonable overview of the type of grants we’re excited to recommend as part of this work. Future posts will highlight the organizations we’re closely tracking as potential 2017 top charities (No Lean Season and Zusha!).[1]

This post will discuss Incubation Grants to:

• IDinsight
• New Incentives
• Results for Development (R4D)
• Charity Science: Health
• Mindset engagement for cash transfers
• Incentives for immunization studies

IDinsight
IDinsight supports and conducts rigorous evaluations of development interventions with an explicit focus on providing useful data to inform funders and policymakers. Good Ventures made a $1.985 million grant to IDinsight for general support in June 2016 as part of GiveWell Incubation Grants.

In conversations with our network, we’ve often heard that IDinsight fills a unique gap in the development sector. There are other organizations that conduct research and advocate for evidence-based decision-making, but our impression is that IDinsight is currently the one most focused on research whose primary goal is to help decision-makers with specific decisions (in contrast to e.g. academic merit). We have seen some indications of other organizations moving in a similar direction, however. We hope that this grant allows IDinsight to grow its staff and take on more projects. IDinsight’s work has the potential to inform GiveWell’s list of top charities by increasing the body of evidence around potential priority programs and improving available monitoring and evaluation information around specific organizations.

Recently, Good Ventures made an additional grant to IDinsight to support an “embedded IDinsight team” for GiveWell top charities, i.e., a small group of IDinsight staff explicitly focused on supporting the creation of high-quality monitoring and evidence for current and future GiveWell top charities. For example, IDinsight may work with New Incentives to run an impact study, and possibly a randomized controlled trial (RCT), on its pilot program to incentivize immunization. Another possible project for the embedded team is conducting monitoring and evaluation of cataract surgery programs, which could improve our understanding of the efficacy of the program and whether we should recommend charities that work on it. Additional possible projects for the IDinsight embedded team are discussed here.

We don’t expect a new GiveWell top charity to originate from this work in 2017, but hope that it will inform our future recommendations.

New Incentives
We made three Incubation Grants to New Incentives for its conditional cash transfer program aimed at preventing mother-to-child transmission (PMTCT) of HIV and encouraging pregnant women to deliver in health facilities (e.g., rather than at home). We decided not to recommend New Incentives’ PMTCT and facility delivery program as a 2016 top charity due to insufficient evidence supporting the program, although we were impressed by the organization’s staff. We wrote about this decision at length in this blog post.

With our encouragement, New Incentives shifted its focus to a new program, conditional cash transfers to incentivize immunizations in Nigeria. We’re planning to follow its work on this program as a potential future top charity, although we do not consider it likely to become a GiveWell-recommended charity in 2017.

Results for Development (R4D)
Pneumonia is one of the leading killers of children worldwide, and our impression is that there is no dedicated funding stream for its treatment (as there is for other major diseases like AIDS, tuberculosis, and malaria). R4D is implementing a program to increase use of amoxicillin, the World Health Organization-recommended first-line treatment, to treat childhood pneumonia in Tanzania. In May 2016, Good Ventures provided $6.4 million to support this program as part of GiveWell Incubation Grants.

We have a positive view of R4D as an organization: its staff, evidence-driven approach, and transparency. We also believe that the use of amoxicillin to treat childhood pneumonia could be competitive with our current priority programs. Our key question around this program as a possible GiveWell top charity is monitoring and evaluation. We’re unsure whether R4D’s monitoring will lead us to feel confident that children sick with pneumonia actually receive treatment. This is due to the complex nature of the intervention, which may make it more challenging to collect high-quality monitoring data comparable with that of our current top charities.

We currently expect that R4D will have the data available to potentially qualify as a top charity in 2018 or 2019 and we hope to evaluate it then.

Charity Science: Health
Charity Science: Health was founded by members of the effective altruism community with the explicit goal of creating a GiveWell top charity. Charity Science: Health plans to send SMS text reminders for vaccinations due to the strong evidence base they see for this program in increasing immunization rates. Good Ventures made a grant of $200,000 to support the first year of the organization’s work in India.

Because we have not yet vetted the relevant evidence closely, we remain unsure about whether we would recommend SMS reminders as a priority program. Charity Science: Health has been transparent and communicative with us, and we expect to learn from its work. Charity Science: Health is also a young organization with a very short track record, and we don’t anticipate evaluating it as a top charity until 2018 or 2019.

Mindset engagement for cash transfers
GiveDirectly, one of GiveWell’s top charities, provides unconditional cash transfers to very poor individuals in East Africa. In May 2016, Good Ventures made a $350,000 grant to Innovations for Poverty Action to support an RCT—in collaboration with GiveDirectly—testing whether “mindset engagement” approaches to cash transfers, such as watching an inspirational film or meeting with a counselor, affects the outcomes for cash transfer recipients by changing the framing of the transfer and thus how it is spent. The approaches are aimed at encouraging recipients to use the transfers to pursue their goals by increasing their sense of self-efficacy and understanding of their opportunities, which—according to the researchers’ theory—may have been adversely impacted by time spent in poverty. This study could influence the work of one of our current top charities (GiveDirectly) or our understanding of cash transfers as a priority program.

Incentives for immunization studies
In 2015, Good Ventures made two $100,000 grants to support further study of whether providing incentives for immunization could increase vaccination rates. These grants were made as part of our work to grow the body of evidence around promising programs that could become potential GiveWell priority programs.

The Incubation Grants were made to the Abdul Latif Jameel Poverty Action Lab (J-PAL) at the Massachusetts Institute of Technology and Interactive Research and Development (IRD) to support high-quality replications of a promising study on the impact of providing non-cash incentives, such as grocery vouchers, for parents to vaccinate their children. The replication studies are being conducted in India and Pakistan.

We are unsure when the results of these studies will be available.

Other work to support potential future top charities

Evidence Action, the parent organization of GiveWell top charity Deworm the World Initiative as well as No Lean Season, a GiveWell Incubation Grant recipient, recently announced a call for results of RCTs and other rigorous empirical studies that demonstrated a positive impact of an intervention benefiting poor households, and is planning to fund 3-6 of these proposals for further research. We’re excited to see this announcement and expect the results may further our understanding of potential GiveWell priority programs.

Full list of GiveWell Incubation Grants

A full list of grants we’ve recommended is available at www.givewell.org/research/incubation-grants.

If you know of a strong proposal for a potential GiveWell Incubation Grant, please email applications@givewell.org. We’d be particularly interested in new groups that work on promising programs for which we have not found charity implementers.

Notes

[1] In December, we recommended a grant of $900,000 to gui2de to scale up its Zusha! road-safety programs. This grant write-up is not yet public, but notes from our initial conversations with Zusha! are available here and here.

The post GiveWell Incubation Grants appeared first on The GiveWell Blog.

We’re planning to release updated top-charity recommendations in mid-November, and one of the questions our staff has been debating recently is whether to recommend New Incentives as a top charity.

We’ve decided that New Incentives doesn’t currently meet our criteria for a top charity because its program doesn’t have sufficient evidence supporting it. However, we have been extremely impressed with and think very highly of New Incentives’ staff and are considering how best to support them in the future and incentivize others to found an organization like they did.

In this post, we summarize the answers to the key questions we asked to determine whether New Incentives meets our criteria for a top charity recommendation and the options we’re considering for future support.

Background

New Incentives operates a conditional cash transfer (CCT) program in Nigeria to incentivize pregnant women to deliver in a health facility. New Incentives originally intended its CCT program to focus primarily on prevention of mother-to-child transmission (PMTCT) of HIV. However, under this model the program did not reach enough HIV-positive pregnant women to justify its operating costs, and in 2015, New Incentives expanded its program to target both HIV-positive women and HIV-negative women.

New Incentives was the first organization we supported as part of our experimental work to support the development of future top charities. It has been about two and a half years since New Incentives received its initial grant, and it now has a long enough track record implementing its program to be considered for a top charity designation.

Is New Incentives’ intervention evidence-backed?

New Incentives’ impact is made up of three components: (a) delivering cash to very poor people, (b) incentivizing HIV-positive pregnant women to deliver in clinics and get the medicines that prevent mother-to-child transmission of HIV, and (c) incentivizing pregnant women to deliver their babies in a health facility.

Because a relatively small portion of New Incentives’ beneficiaries are HIV-positive, because it costs New Incentives more than GiveDirectly to deliver each dollar, and because it is likely reaching individuals with higher incomes than GiveDirectly does, the impact that has the dominant effect on our view about whether or not New Incentives meets the standard we have for a top charity’s cost-effectiveness is the impact of facility delivery on neonatal mortality.

The evidence we have for the impact of facility delivery comes from (1) relevant randomized controlled trials (RCTs), (2) monitoring that New Incentives carries out, and (3) non-RCT evidence on the impact of facility delivery.

Overall, the evidence from the RCTs increases our confidence that an intervention that offers improved neonatal care could have a significant impact on neonatal mortality, but the evidence we have seen and New Incentives’ current monitoring of its program is insufficient to convince us that increasing the number of women who deliver at facilities has a similar impact.

Randomized controlled trial evidence

Two RCTs of low-intensity training programs for traditional birth attendants found significant (30-45%) reductions in neonatal mortality. These interventions are different than New Incentives’ intervention but may have a similar effect since they aim to increase the knowledge of traditional birth attendants so that they offer similar care to that which is offered in health facilities. We did not find any RCTs on facility delivery itself; these two RCTs are the most similar ones to New Incentives’ program that we identified. The interventions varied:

• In Gill et al. 2011, the intervention group received training and supplies related to common practices to reduce neonatal mortality immediately following birth. The study observed significant differences between the treatment and control group on practices such as drying the baby with a cloth and then wrapping it in a separate blanket (as opposed to using the same blanket), clearing the baby’s mouth and nose with a suction bulb (instead of a cloth), and using a pocket resuscitator (instead of mouth to mouth) (see Table 5, Pg. 8). We have not closely vetted this study but note some significant-seeming differences between the treatment and control birth attendants–in particular, the treatment group had significantly more education than the control group (see Table 1, Pg. 4).
• In Jokhio et al. 2005, the intervention group received supplies and 3 days of training focused on antepartum, intrapartum, and postpartum care, including activities such as: “how to conduct a clean delivery; use of the disposable delivery kit; when to refer women for emergency obstetrical care; and care of the newborn.” The intervention group was “asked to visit each woman at least three times during the pregnancy (at three, six, and nine months) to check for dangerous signs such as bleeding or eclampsia, and to encourage women with such signs to seek emergency obstetrical care.”

New Incentives’ monitoring

New Incentives’ staff interviews a nurse and conducts additional inspection at each health facility it considers working with. New Incentives reports the results of these interviews. Two questions are most relevant to our assessment of the similarity between the interventions studied in the RCTs discussed above and the care offered in facilities New Incentives works with.

New Incentives asks nurses at each health facility: 1) “What multiple steps do you take immediately after delivery?” and 2) “What are the essential steps immediately after birth in ensuring that the baby can breathe and is warm?”

For the first question, New Incentives counts how many of the following steps nurses say they take (without being prompted by the New Incentives staff member asking the question): a) Dry baby with cloth, b) Slightly rub baby, c) Clear airways, d) Use air mask if necessary, e) Regulate temperature (put on mother’s belly), f) Don’t know/refused to answer. For the second question, New Incentives captures a free form answer.

We have limited information about the differences in practices between the intervention and control groups in Jokhio et al. 2005, but we do have this information for Gill et al. 2011. (See Gill et al. 2011, Table 5, Pg. 8.) It does not appear that the way New Incentives evaluates answers to its first question can tell us whether nurses in the facilities with which it works follow the improved practices from Gill et al. 2011.

We aggregated the answers to the second question, and 17 of 54 answers explicitly mentioned using a bulb syringe or mucus extractor, which we would guess is equivalent to clearing the baby’s mouth and nose with a suction bulb in Gill et al. 2011 (another 11 mentioned ‘clear airways’ or ‘suck’ which might refer to the procedure used in Gill et al. 2011). We were not able to get additional relevant information from nurses’ answers to the second question.

New Incentives does not appear to ask questions that fully address the other major difference between the intervention and control groups in Gill: use of a resuscitation intervention.

The intervention offered by Jokhio et al. 2005 includes antenatal care in addition to intrapartum and postpartum care, and we don’t know what impacts each part of the intervention had.

Note that New Incentives does not systematically collect data on the type of care women who enroll in its program would have received had they not delivered in a facility, though it has done some limited surveys of traditional birth attendants in the areas it works in.

Non-randomized evaluations of the impact of facility delivery

We have not carefully reviewed these studies, and the studies we identified found mixed effects (including some studies finding higher neonatal mortality in facilities) but we have major questions about these studies’ ability to assess facilities’ causal impact.[1] In particular, women may be more likely to go to a facility for childbirth when they are experiencing complications, which could bias the results.

What is our best guess about New Incentives’ cost-effectiveness?

The most important questions in assessing New Incentives’ cost-effectiveness are (a) the impact its cash transfers have on rates of facility delivery and (b) the impact that increased facility delivery has on neonatal mortality.

New Incentives is conducting an RCT of its impact on (a) and preliminary results indicate that it had a significant impact on facility deliveries: 48% of women in the treatment group (i.e., all those who were offered the opportunity to enroll in the program even if they chose not to do so) delivered in a facility versus 27% in the control group. However, there are differences between the program studied by New Incentives’ RCT and its current program; the RCT only targeted HIV-positive women, so some portion of the impact may be attributable to educating women about the importance of PMTCT. The program studied in the RCT also provided larger cash transfers than New Incentives will provide in its ongoing program: the program originally gave 6,000 naira (approximately 19 US dollars) for enrollment, 20,000 naira for delivery, and 6,000 naira for an HIV test; the program currently gives 1,000 naira for enrollment and 10,000 naira for delivery.

As noted above, we have very limited information to rely on when forming an estimate of the impact of facility delivery on neonatal mortality, and we do not see the evidence from the RCTs described above as particularly relevant or informative.

However, in trying to arrive at our best guess of the impact of the program, we also considered the facts that:

• The interventions described in Gill et al. 2011 and Jokhio et al. 2005 are relatively low cost and of limited intensity, and they find significant decreases in neonatal mortality. This increases the plausibility that merely referring women to facilities for childbirth could have a similar, significant impact.
• Our intuition (supported by what appears to be conventional wisdom in the global health community) strongly implies that delivering in a facility (in general, without respect to the specific facilities New Incentives works with in Nigeria) is likely to lead to lower mortality than alternatives.

Philosophical value judgments

Based on the results from the RCTs, we would expect New Incentives’ program to primarily prevent deaths of very young children (largely those within the first days or week of life). In internal, staff discussions about New Incentives, we have asked ourselves how we value the lives of newborn children vs. the lives of those saved by malaria nets (the other life-saving intervention we currently recommend). We have not completed a thorough assessment of the ages at which people die from malaria, but our impression is that the median age of death is approximately 1.[2]

We believe there is no “right” answer to this question, but depending on one’s values, the answer could have a significant impact on the relative cost-effectiveness of New Incentives vs. the Against Malaria Foundation, and by extension our other top charities.

Key considerations include:

• One could simply sum the number of remaining years of life lost due to a death of a newborn vs a 1-year-old.
• One could focus solely on lives saved and treat all lives as equivalent.
• One might say that families and society have invested more in 1-year-olds and that 1-year-olds have more self-awareness and “personhood” than newborns, leading to valuing the 1-year-old more than the newborn.

Primarily for the last reason, the GiveWell staff who participated in these discussions tend to value 1-year-old lives over newborns, though our relative weights vary considerably.

Best guess cost-effectiveness estimate

Ultimately, we don’t have enough information to arrive at a reliable estimate of the impact of facility delivery on neonatal mortality. Our best guess is extremely rough, based primarily on intuitions formed based on limited data, and one that could easily shift significantly. We asked all staff who primarily work on GiveWell research to (a) guess the likely effect of New Incentives’ program on neonatal mortality and (b) enter the philosophical values discussed above. This yielded a median staff estimate that New Incentives was approximately as cost-effective as cash (in GiveDirectly’s program). Our cost-effectiveness model is here (.xlsx).

Is New Incentives transparent?

Yes – extremely. New Incentives has shared all of the information we have requested (and more) in a timely fashion. We feel that it is as good as any other organization we have ever engaged with on this criterion.

Options we’re considering for future support of New Incentives and/or its staff

We have discussed each of the following options with New Incentives and plan to let New Incentives’ preference drive our decision about which one to choose. In considering these options, we took into account (a) the likely direct impact funding would have and (b) the incentives that funding would create for others considering starting a new organization like New Incentives.

1. Recommend that Good Ventures (a foundation with which we work closely that has provided past funding for our experimental work) provide an “exit grant” of approximately $1.2 million to New Incentives. New Incentives relied heavily on funding we recommended in its scale up, and abruptly stopping funding could cause it significant harm. Our impression is that funders often give grantees exit grants to offer them time to comfortably adjust their plans for fundraising and spending; this has been GiveWell’s experience with support from institutional funders. We would plan to benchmark our recommendation to the level of support New Incentives could have expected from us over the next two years (January 2017 – December 2018) as of the last time Good Ventures made a grant (March 2016). $1.2 million represents half what we would have projected New Incentives spending to be in 2017 and 2018 as of March 2016. (It grew faster than we expected since March 2016, so this is less than 50% of its projected operating expenses.)
2. Recommend that Good Ventures agree to support some portion of New Incentives’ ongoing operations and a randomized controlled trial of New Incentives’ program’s impact on neonatal mortality. New Incentives’ program doesn’t seem cost-effective enough that we’d be willing to recommend that Good Ventures fully fund an RCT and New Incentives’ ongoing operations, but we’d consider recommending some, significant support (very roughly, we’d cap a recommendation at 50% of the total cost) if New Incentives could raise the rest of the funding elsewhere. This option would provide New Incentives with the opportunity to demonstrate that its program is more effective/cost-effective than we currently expect it to be as long as it is able to convince other funders to provide some support as well.
3. Provide support to New Incentives/the New Incentives team to do something new. If New Incentives or its staff were interested in starting a new charity aiming to be a GiveWell top charity or significantly changing its program to focus on something more cost-effective, we would recommend that Good Ventures provide support.

We hope to decide soon about which option to pursue.

[Added December 19, 2016: GiveWell’s experimental work is now known as GiveWell Incubation Grants.]

Notes
[1] We identified two relevant meta-analyses. Chinkhumba et al. 2014, a meta-analysis of six prospective cohort studied of perinatal mortality in sub-Saharan Africa found 21% higher perinatal mortality in home deliveries compared to facility deliveries (OR 1.21 [1.02-1.46]) using a fixed-effects model, but this difference was not significant using a random effects model (OR 1.21 [0.79-1.84]).

We are also concerned that studies limited to the perinatal period may not capture longer-term neonatal effects. Tura et al. 2013, a meta-analysis of 19 studies (of various methodology) of the effect of facility delivery on neonatal mortality, found mixed results. Pooled results from low- and middle-income countries showed 29% reduction in risk of neonatal death associated with facility delivery. However, results of the studies were highly heterogeneous. Of the 8 studies in sub-Saharan Africa, 4 found effect near the pooled mean, and the other 4 did not find a statistically significant effect. (Of the four that did not find a significant effect, two studies found a nonsignificant effect close to the pooled mean of all studies, and two found no effect.)

A retrospective study based on the demographic and health surveys in Nigeria found that facility delivery is associated with increased neonatal mortality (adjusted odds ratio 1.28 [1.11-1.47], Fink et al. 2015, Figure 1, Pg. 5).

[2] Here is one paper we found. We have not vetted this paper. The simple average age of death in it is approximately 1.2 years (see Table 1).

The post New Incentives update appeared first on The GiveWell Blog.

We wish we had more top charities, and as we look to the future we expect (and hope) that there will need to be more recommended charities in order to productively use all the donations that GiveWell-influenced donors are making. One of our major activities is trying to expand our top charities list – both by investigating charities that already exist, and by supporting activities (from new nonprofits to studies) that could eventually result in a larger set of evidence-backed programs and a larger set of top charities.

This post discusses types of charities that we would be excited to learn more about if they existed. We would also consider providing support to individuals trying to create the types of organizations described below. In a similar spirit to a request for startups, we’re sharing this list in the hopes that it might help us find out about such charities – or might help us find and support people looking to create them.

In brief, we would be excited to see:

• Charities that implement GiveWell’s priority programs: vitamin A supplementation, immunizations, conditional cash transfers, micronutrient fortification, or even bednets and deworming (since our top charities that focus on the latter two have limited room for more funding). More
• Charities implementing potential priority programs that are particularly challenging, particularly those revolving around (a) treatment of treatable conditions in a hospital or clinic setting; (b) behavior change for improving health. We see several hurdles to successfully focusing on such programs, but would be excited to see charities that overcome such hurdles. More
• Charities that collect or generate information and data relevant to our recommendations. Currently, we recommend charities based partly on the data they themselves collect and share. But we could potentially recommend an organization that does not, itself, collect and share strong monitoring data, if we had independent data showing its activities’ effectiveness. More

Charities that implement GiveWell’s priority programsThe first step in our research process is considering the independent evidence of effectiveness and cost-effectiveness of a program to determine whether we should classify it as a priority program. The programs that our top charities implement — bednet distribution, cash transfers and mass deworming programs — are among our priority programs.

There are cases where we have classified a program as a priority, but have not found a promising charity that focuses on that program.

We would be excited to see new charities implementing our priority programs who plan to publicly share significant monitoring and evaluation data and generally expect to be extremely transparent about their work.

Some specific examples of organizations we would be interested in:

• Providing vitamin A supplements to areas with high rates of vitamin A deficiency and child mortality. One illustrative example: the Central African Republic has a high child mortality rate (139 deaths per 1,000 children under 5, data from Gapminder here) and low rates of vitamin A supplementation (40%, data from the World Bank here). More on vitamin A supplementation and the questions we would ask a charity to answer here.
• Providing immunizations in areas with low coverage rates. An illustrative example: the Central African Republic has low rates of measles (49%) and DTP (diphtheria, tetanus and pertussis) immunizations (47%). (Data from UNICEF here.) We have not recently completed an intervention report on routine immunization but our report on maternal and neonatal tetanus immunization campaigns lays out the questions we would use to evaluate charities.
• Providing conditional cash transfers to encourage school attendance, clinic visits, etc. while also transferring wealth to low-income people. (Note that we currently support a charity taking this approach, in the hopes that it may become a top charity in the future.)

There are also some cases where we have identified charities that run priority programs, but are not completely satisfied with the evidence we’ve seen for the charities’ track records, either because we don’t find the case for impact compelling or because the charities have been hesitant to share information to the degree necessary. In these cases, we would be excited about organizations that implemented these programs and shared a significant amount of information about their work.

Possibilities in this category include:

• Organizations working on micronutrient fortification, such as salt iodization or folic acid fortification.
• Organizations working on immunizations campaigns for tetanus or measles.
• Organizations working on mass drug administration for neglected tropical diseases other than deworming.

Note that in some of the above cases, we have not completed our intervention reports, so it’s possible that we might conclude that a program does not have sufficient evidence of effectiveness or is not cost-effective enough to be a priority program. In general, we allocate our time with the goal of finding the combination of a strong program and strong organization; there are promising programs we have not completed our investigations of because we have not found promising charities running them.

Finally, we would be excited about organizations working on priority programs where we currently have top charities that may have limited room for more funding. In particular, we would be excited to see a new charity focusing on bednet distribution or deworming programs, while having a high willingness to collect and share data.

Note that we have intentionally not included areas like surgery, education, and family planning in this post. We have not prioritized research in these areas recently, but we do hope to revisit them in the future. Because we have not recently looked into these areas, we want to put more time into determining whether existing organizations may be able to meet our criteria before calling for new ones.

Charities implementing potential priority programs that are particularly challengingTwo types of promising programs are those (a) that focus on treating specific diseases or (b) that aim to promote behavior change. We have not completed intervention reports for either because both seem particularly challenging for charities to implement successfully.

Treatment programs

Treatment programs would focus on treating individuals after they contract a disease (e.g., malaria, HIV/AIDS, pneumonia, diarrhea, and tuberculosis). Unlike the programs we currently recommend, which target all members of a population (e.g., all women of childbearing age, all children under age 5), treatment programs are significantly more complicated. To receive treatment, (a) an individual generally must go to a clinic when s/he requires treatment, (b) be accurately diagnosed as having the condition, and (c) the clinic must have the necessary drug in stock. In some cases (e.g., HIV/AIDS or tuberculosis), the individual would have to return to the clinic and replenish his/her supply of the medication and also adhere to a long (or perpetual, in the case of HIV/AIDS) treatment regimen.

Because of the costs associated with providing treatments (training skilled diagnosticians or keeping drugs in stock), it seems unlikely that a charity should focus on providing just one of the above treatments. Possible approaches a charity could take include:

• Setting up high-quality clinics that provide treatments and other medical care. We have previously reviewed and recommended two organizations that follow this model: Partners in Health and Possible. Our guess is that clinics are unlikely to be as cost-effective as our current top charities.
• Running a program that involves community health workers to provide a limited range of treatments. This is somewhat similar, though not identical, to Living Goods’ model. Our impression is that there is a relatively large literature related to programs implemented by community health workers and the quality of the services they provide vary widely. Were we evaluating a charity implementing this model, we would be particularly focused on its monitoring and evaluation data as well as its cost-effectiveness.
• Providing treatments or diagnostics to clinics that would otherwise not have them. Were we evaluating a charity implementing this model, we would seek compelling evidence that the charity is causing the clinics to have access to treatments it otherwise would not have had, that the treatments are ultimately provided to people who need them (i.e., are accurately diagnosed as needing them) and that the recipients follow the prescribed regimen. Note that that this model seems similar to the some of what the Global Fund does. We evaluated the Global Fund in 2009 and 2010 but were not able to obtain the data we needed to recommend it.

We have not completed intervention reports for treatment programs, but we would likely consider them priority programs were we to find a charity effectively implementing one of the models above.

Behavior change

We have also not completed intervention reports for behavior change programs. Our impression is that programs to promote handwashing, breastfeeding or other health behaviors could be highly cost-effective because they can reach many people at low cost. (Note that Development Media International, a standout charity of ours, implements a behavior change intervention using mass media.)

The key challenge facing behavior change charities is demonstrating that their intervention causes behavior change and that that behavior change improves health. DMI is running a randomized controlled trial of its program to address this question, but very few charities are in a position to run a study like this. In GiveWell’s early years, we recommended PSI, a behavior change organization, but we changed its recommendation status because we no longer felt the evidence for its effectiveness was sufficiently strong.

We would be excited to evaluate a behavior change organization that could make a compelling case for its impact and is ready to share significant information about its activities with us.

Charities that collect or generate information and data relevant to our recommendationsThe types of charities discussed above are ones that could meet our criteria and receive a recommendation. We would also be interested in supporting groups that generate information or data that could inform our recommendations.

This could include groups that:

• Collect data that directly informs our views on current and potential top charities or the programs they implement. For example, Good Ventures provided funding (based on our recommendation) to IDinsight to conduct additional monitoring on the Schisotosomias Control Initiative’s programs. We could also imagine IDinsight, or a group like them, collecting and sharing better data that would inform our view of large-scale bednet distrubtions implemented by groups other than the Against Malaria foundation, salt iodization programs, or tetanus immunization campaigns. In all of these cases, we could recommend an organization that does not, itself, collect and share strong monitoring data, if we had independent data showing its activities’ effectiveness.
• Run randomized controlled trials (or replications of RCTs) of interventions that could be at least as cost-effective as our current priority programs. We’d be particularly excited about interventions that could plausibly be significantly more cost-effective than our current top charities.
• Qualitative research or journalism that informs our views of top charities or the programs they implement. This could include (a) articles about current top charities or the programs they implement (e.g., these pieces by Jacob Kushner that we commissioned in 2014 and 2015), (b) surveys of people served by the programs we recommend or the aid community in general, (c) research that directly addresses unanswered questions in our research (e.g., to what extent do individuals served by Development Media International’s program have access to clinics that can diagnose their conditions and provide them with medicine?), and/or (d) provides additional context on the lives of people living in extreme poverty, among others.
• GiveDirectly has suggested the idea of creating a facility for funding and implementing cash transfers as a control group for randomized controlled trials of development interventions. GiveDirectly has told us that this is something it doesn’t plan to currently prioritize and that it would be excited to see another organization undertake this. (It has expanded on this idea, suggesting a broader mandate for supporting cash transfer work, in this recent article.)

The post Charities we’d like to see appeared first on The GiveWell Blog.