You are here

Home » Research » GiveWell Incubation Grants » Results for Development—Childhood Pneumonia Treatment Scale-Up

Results for Development — Childhood Pneumonia Treatment Scale-Up

Published: August 2016

[Added December 19, 2016: GiveWell's experimental work is now known as GiveWell Incubation Grants.]

Note: This page summarizes the rationale behind a grant to Results for Development made by Good Ventures. Results for Development staff reviewed this page prior to publication.

Summary

As part of GiveWell’s general effort to support the creation of future top charities, in May of 2016 Good Ventures granted $6,400,000 to Results for Development (R4D) to support its Market Dynamics practice area’s pneumonia treatment program in Tanzania. The organization aims to use these funds to increase use of amoxicillin, the WHO-recommended first-line treatment for childhood pneumonia.

The intervention

Pneumonia is well-known to be a leading cause of death among young children worldwide.1 R4D told us that dedicated funding sources do not exist for pneumonia as they do for malaria, tuberculosis, and HIV/AIDS.2 As a consequence, countries often have a lower supply of pneumonia drugs in the public sector than is needed to treat their populations.3 In particular, funding in Tanzania for the child-friendly dispersible tablet formulation of amoxicillin, the WHO-recommended first line of treatment for pneumonia,4 had not been secured past mid-2016 despite a strong existing infrastructure for treatment.5 R4D intends to use this grant to fulfill as much unmet amoxicillin demand in Tanzania as is possible over the next couple of years.

Though we are somewhat uncertain on the details of R4D’s plan for our grant (beyond buying and distributing amoxicillin), our best understanding is that R4D intends to use our grant roughly as follows:

  • About 40%: buy amoxicillin dispersible tablet (amox DT) for the next couple of years and donate the product to the government of Tanzania for use in public health facilities. This funding is intended to be catalytic and time-limited.
  • About 25%: monitor and evaluate the intervention. (See below for more details on R4D's monitoring and evaluation plans.)
  • About 10%: test programmatic interventions (e.g. mentoring) in public healthcare facilities and private drug shops to improve proper diagnosis and dispensing of treatment for childhood pneumonia.
  • About 25%: hire and pay for R4D staff working on this project both in Washington, D.C. (where the organization is based) and in Tanzania.

R4D estimates that the treatment rate of pneumonia with amoxicillin was approximately 40% in 2014;6 we are uncertain about the accuracy of this estimate. The organization aims to increase the treatment rate to 55% by the end of the project in five years.7

Track record

Our impression is that the staff members at R4D working on this program have, in the past, focused their efforts on other types of work, such as analyzing the malaria bednet market to advise the Global Fund and other funders on bednet procurement.8 Starting in 2015, the R4D team launched a project to administer catalytic funding to purchase amox DT for the public sector in Ethiopia. However, our understanding is that R4D has not previously played a role similar to the one it aims to play with this grant.9 As a consequence, it has a limited track record on past projects directly relevant to this project.

Kanika Bahl, R4D's Managing Director and a member of the leadership team for this program, formerly worked at the Clinton Health Access Initiative, where she did work on direct implementation of similar projects (e.g. scaling pediatric AIDS drugs in developing countries).10

In general, we have a strong positive impression of Ms. Bahl and the R4D staff members with whom we corresponded as part of this investigation. We have communicated very well with these staff members, and they appear to place significant value on evidence-based decision-making and transparency.

If we eventually recommend R4D as a top charity, which is one of the main goals of this grant, that recommendation would rely on the track record that R4D built with these funds.

Monitoring and evaluation

In order to understand whether its activities are effectively improving the rate of pneumonia treatment in Tanzania, R4D plans to implement the following monitoring and evaluation activities:11

  • National levels: R4D plans to look at national data on the procurement of amoxicillin approximately every six months to see whether additional stock is entering the public sector.12 It also plans to survey private sector importers approximately every six months to determine whether they are purchasing more stock.
  • Availability data: R4D intends to survey health facilities to see whether amoxicillin is in stock and whether they have experienced stockouts in the past 30-90 days.
  • Correct dispensing: R4D plans to review public clinical records and clinician reports to determine whether amoxicillin dispensing has risen and whether the treatment was prescribed correctly. R4D and GiveWell are both uncertain whether this data will be high-quality, and it is plausible that this information will not be helpful.
  • Clinician understanding of appropriate pneumonia diagnosis and treatment practices: R4D plans to evaluate whether clinicians are retaining knowledge over the course of the mentoring sessions that it plans to implement. We do not expect that this data will be particularly relevant to our overall assessment of the grant’s success.

We are uncertain whether the above activities will be able to demonstrate that children sick with pneumonia actually receive treatment. R4D considered, but decided not to use, some other monitoring and evaluation activities that would have been aimed at answering this question:

  • Video-recording clinicians: Obtaining informed consent for each video-taped patient would be too difficult for this possibility to be feasible.
  • Surveying patients outside the clinic: R4D expects the number of treatments will be too low for this to be a cost-effective source of monitoring. R4D also suspects that poor patient recall and understanding could result in low-quality data.
  • Surveying parents at home about their children's treatment: Again, R4D suspects that any data collected in this manner could be low-quality due to poor recall and understanding.
  • A randomized controlled trial focused on child mortality: We expect that this would be expensive due to the required size of the trial, though it is possible that we could return to this possibility in the future.

Cost-effectiveness

We constructed a relatively rough cost-effectiveness model for R4D's pneumonia program, available here. Our current estimate of R4D's program puts it in the range of 50% to 200% as cost-effective as the mass distribution of long-lasting insecticide-treated nets.

There are a number of inputs to this cost-effectiveness analysis about which we are particularly uncertain:

  • The current pneumonia treatment rate in Tanzania
  • The level of increase in treatment rates that will be created by R4D's program
  • Non-R4D costs due to the program (which are currently excluded entirely)
  • The counterfactual: how would pneumonia treatment rates in Tanzania change in the short and long term in the absence of this grant?

We also have little sense of how cost-effective a future pneumonia treatment program outside of Tanzania might be. This question could become relevant if we considered recommending an organization implementing a similar pneumonia treatment program as a top charity.

Room for more funding

R4D considers this grant the first tranche of a five-year project that will cost about $19 million.

During the next 2.5 years, the organization hopes to find another funder who can replace its direct funding for amoxicillin, so that Tanzania has a sustainable source of funding for amoxicillin at the conclusion of the five-year project.13

We believe there may be substantial room for more funding for the broader intervention of scaling up evidence-backed, cost-effective health commodities, though we have not thoroughly considered this question nor carefully considered the cost-effectiveness of other well-known health commodities.

Grant fungibility

In our grant-making, we want to consider the question of fungibility: whether some other funder would have funded the project if we had not, in which case our grant would not actually lead to an increased pneumonia treatment rate.

Our impression is that this is unlikely to be the case. R4D told us that it has no other potential sources of funding.14 In addition, R4D is moving forward with a smaller grant than it originally proposed, and the organization also told us that it is not planning to try to raise additional funding because it does not have any reasonable prospects. R4D estimated a 25% or lower probability that, in the absence of this project, Tanzania would find another funding source for the amoxicillin this project plans to fund.15

Risks of the grant

We see a number of potential risks to the success of this grant:

  • Monitoring and evaluation data: Due to the complexity of the intervention, it is possible that the monitoring and evaluation data for R4D's program could turn out to be of lower quality than the data for our current top charities. In particular, we are uncertain whether we will be able to see high-quality data demonstrating that children with pneumonia actually receive amoxicillin. (See above.) This could weaken the case for R4D or a similar organization becoming a future top charity.
  • Current treatment rates: As stated above, we hold significant uncertainty regarding R4D's estimate for the current pneumonia treatment rate in Tanzania. It is possible that current treatment rates are higher than estimated, which could reduce the potential for this grant to raise treatment rates.
  • Counterfactual: It is possible that a similar project would be funded in the absence of this grant, though our current guess is that this is relatively unlikely. (See above.)
  • Transparency: We are uncertain about the level of comfort that officials in Tanzania will have with our expectations for transparency; it is possible that this could become an issue.
  • Future plans: We are uncertain what R4D would do after this grant if it is successful. It is possible that the organization's subsequent plans would not lead it to be a contender for a top charity recommendation.

Plans for follow-up

As stated above, R4D considers this grant to be the first tranche of a five-year project costing around $19 million. The organization intends to seek a second tranche of funding, which could range in size from $6 million to $13 million, about twelve months from now. At that point, we expect to have some data on treatment procurements and clinic availability, which will provide us with some additional information. We are uncertain, however, whether this data will be conclusive enough for us to have a confident assessment of the grant's progress.

We expect to receive updates from R4D every few months for the next year about this grant; we plan to publish summaries of these updates when warranted. We expect to receive major updates at baseline, at eight months, at eleven months, and at fourteen months, as the organization intends to collect and share procurement and clinic availability data on this approximate timeline.16

We plan to pay particular attention to these aspects of the grant in following up:

  • Operational updates: Has the program scaled as intended? Have any major challenges arisen?
  • Funding updates: Have any other funders expressed interest in funding the program?
  • Monitoring and evaluation updates: What does the monitoring data indicate about the program's impact? How high-quality is this data?

Internal forecasts

We are experimenting with recording explicit numerical forecasts of the probability of events related to our decision-making (especially grant-making). The idea behind this is to pull out the implicit predictions that are playing a role in our decisions, and to make it possible for us to look back on how well-calibrated and accurate those predictions were. For this grant, we are recording the following forecasts:

  • Good Ventures gives R4D a second grant of approximately the same size in 12 months: 70%
  • R4D is a top charity by the end of 2019: 25%

Our process

R4D contacted GiveWell in July 2015 about funding a pneumonia treatment program. Due to capacity constraints, we did not pursue the possibility. In January 2016, R4D contacted us again, and we spoke several times with the organization before deciding to make this grant. We also put significant time into constructing a cost-effectiveness model.

Sources

Document Source
GiveWell's cost-effectiveness analysis for R4D's Pneumonia Program Source
GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, and Thayer Rosenberg on January 27, 2016 Source
GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer on February 19, 2016 Source
Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer, phone conversation with GiveWell, April 14, 2016 Unpublished
Kanika Bahl, Cammie Lee, Thayer Rosenberg, Jean Arkedis, and Aileen Palmer, phone conversation with GiveWell, April 21, 2016 Unpublished
Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer, phone conversation with GiveWell, April 26, 2016 Unpublished
UN IGME - Levels & Trends in Child Mortality Source (archive)
USAID - Evaluation of Country-Level Procurement Constraints Source (archive)