This post will highlight GiveWell’s work with IDinsight, part of our Incubation Grants program to help grow the pipeline of potential future top charities and improve the quality of GiveWell’s recommendations. We previously highlighted the work of No Lean Season and Zusha!, Incubation Grant recipients and potential 2017 GiveWell top charities. Unlike these organizations, we don’t expect IDinsight to itself become a top charity. Instead, we hope it will help GiveWell support the development of more top charities and increase our understanding of the organizations we recommend.

IDinsight is an international NGO that aims to help its clients develop and use rigorous evidence to improve social impact. GiveWell is partnering with IDinsight to support organizations’ development of monitoring and evaluation information of the type we’re interested in. This is the first partnership of this kind for GiveWell.

Summary

Working with IDinsight is a major part of our GiveWell Incubation Grants program. We hope that IDinsight can (a) conduct randomized controlled trials (RCTs) of interventions that seem promising but have little evidence to back them up, (b) work with promising organizations or existing top charities to assess and improve their monitoring systems, and (c) conduct additional research work that could inform our recommendations (e.g., additional site visits or surveying beneficiaries about their preferences).

We don’t expect IDinsight’s work to influence our top-charity recommendations in 2017. We may have information from this partnership to inform our 2018 recommendations, and we expect it to influence our 2019 recommendations.

Background

We believe IDinsight fills a unique role in the development sphere. Unlike academics, who may be incentivized to focus on advancing the academic literature, IDinsight focuses on providing decision-relevant data and assessments; it serves nonprofits and policymakers who have to decide whether to implement intervention A or B, by, e.g., producing quick, low-cost, randomized controlled trial evidence. In addition to these types of “decision-focused evaluations,” IDinsight sets up “embedded learning partnerships” within governments and NGOs to answer priority policy questions using a broad suite of data and evidence tools. We think this kind of work is highly valuable and relevant to GiveWell’s mission of finding and identifying cost-effective, evidence-backed programs and charities.

We came across IDinsight early on in our GiveWell Incubation Grants program (then known as “GiveWell’s experimental work”) when we searched for organizations that could help us “bridge the gap between research and implementation”. GiveWell made its first Incubation Grant to IDinsight in September 2014. Since then, we have provided additional support to IDinsight in June 2016 to expand their general operations (in the hopes this would lead to more evidence and organizations of the type GiveWell is interested in recommending) and in October 2016. The latter grant is to support the creation of an IDinsight “embedded team” at GiveWell, and is the focus of this post. We recently recommended an additional grant to support the scale-up of the embedded team’s work.

Funding for IDinsight’s work with GiveWell comes from Good Ventures, a large foundation with which GiveWell works closely, and which has supported our Incubation Grants program.

Goals of our partnership with IDinsight

We hope IDinsight can help fill a gap in GiveWell’s evaluation process. One of our core criteria for recommending charities is high-quality monitoring and evaluation information to demonstrate a charity is having impact. Our impression is that many organizations—likely the majority of charitable organizations—do not have this type of information; in some cases, we think it’s likely that even excellent organizations may not be collecting this type of information due to the high cost, both monetary and in staff-hours, it requires.

We hope IDinsight will help fill this gap by working with promising potential top charities to develop these kinds of monitoring systems and/or to complete an impact evaluation of their work. We’re also interested in IDinsight working to strengthen monitoring systems for some of GiveWell’s current top charities, to help us improve our understanding of their impact.

Initial plans for the IDinsight “embedded GiveWell team”

We’re still early in exploring all of the possible ways in which GiveWell and IDinsight may work together. Two projects are most likely in the near term:

1. Helping GiveWell identify and develop a potential top charity working on cataract surgery. To do this, we think IDinsight will most likely build and implement a monitoring and evaluation system for a promising cataract charity. In August 2016, we published an intervention report announcing our view that cataract surgery was a GiveWell “priority program.” In other words, we believe that the strength of the evidence for and potential cost-effectiveness of the program is competitive with the other interventions we recommend, like distributing nets to prevent malaria and providing direct cash transfers to very poor households, and that we’d be interested in considering charities implementing cataract surgery programs for a top-charity recommendation.

   However, in our early conversations with organizations working on cataract surgery programs, our impression was that they didn’t yet have the type of monitoring and evaluation information on surgical outcomes and their impact on the number of surgeries carried out that we’d like to see as part of our charity review process. That’s why we’re working with IDinsight to support the identification or development of a GiveWell top charity working in this space.

   IDinsight is also undertaking a similar project focused on organizations providing surgeries to correct obstetric fistulas.

2. Conducting an impact evaluation of New Incentives’ immunization incentives program. We may ask IDinsight to work with New Incentives, a GiveWell Incubation Grant recipient and potential future top charity, to run a randomized controlled trial of its pilot program to provide conditional cash transfers to incentivize routine immunizations of infants.

   In February 2017, IDinsight traveled to Nigeria to visit New Incentives and observe their immunization incentives program as part of this project. IDinsight’s notes from the trip are available here.

Other potential projects with IDinsight include, but are not limited, to:

• Improving the Against Malaria Foundation (AMF)’s monitoring. AMF is currently our top recommendation to donors, although we in 2016 we wrote about weaknesses in its monitoring. IDinsight may work with AMF to improve its monitoring standards.
• Conducting an RCT on GiveWell Incubation Grant recipient Charity Science: Health‘s work. Charity Science: Health is a young organization that provides SMS reminders for vaccinations in India. We do not yet have a view on this intervention and are interested in potentially working with IDinsight to better understand Charity Science: Health’s work in this space and its impact on immunization rates.
• Informing subjective value judgments in GiveWell’s cost-effectiveness model. Cost-effectiveness plays a major role in GiveWell’s top-charity recommendations, and we rely on a number of highly uncertain and subjective tradeoffs between increasing income and various health outcomes in order to compare charities that work on different programs. Better understanding the preferences of individuals impacted by the work of the charities we recommend would improve our cost-effectiveness outputs. IDinsight may design and pilot a survey of beneficiaries on how they compare certain health and non-health interventions.

We’re looking forward to seeing what we learn from this partnership.