This is the third post (of six) we’re planning to make focused on our self-evaluation and future plans. The goal of this post is to update GiveWell’s followers on our plans for our traditional work in 2015 and to establish a general set of goals by which we can evaluate ourselves at the beginning of 2016.

As discussed in our previous post, in 2014, GiveWell’s traditional (“top charities”) work conducted a large amount of research while maintaining research quality and building substantially more capacity to conduct research in the future. The amount of money moved to our recommended charities continued to grow; we moved about $28 million to recommended charities in 2014.

This year, our primary goals are to:

• Build management and research capacity for GiveWell’s traditional work while further reducing senior staff time (note 1) spent on this work, primarily by reallocating Elie Hassenfeld’s management responsibilities related to GiveWell’s traditional work.
• Maintain our core research product by completing updates on all eight 2014 recommended charities and determining which of them should be recommended as top charities for the 2015 giving season.

Our secondary goals for 2015 are to:

• Continue to seek outstanding giving opportunities by reviewing 2-4 new charities and publishing 2-4 new intervention reports.
• Improve the cost-effectiveness analyses and room for more funding analyses in charity reviews.
• Finish and launch a redesigned GiveWell website.
• Make further progress on experimental work to “seed” potential recommended charities.

We expect our total output on “top charities” work to be roughly comparable to last year’s, despite a growing staff, because (a) a major focus of the coming year is training, and we expect to trade some short-term efficiency for long-run output; (b) we may be reallocating some capacity from our “top charities” work to the Open Philanthropy Project this year.

More details on some of these goals are below.

Building capacity

In 2015, we hope to build substantially more management and research capacity for GiveWell’s traditional work in order to move toward our goal of having a sustainable organization that is not dependent on past senior staff. As we have discussed before, building capacity is challenging and generally leads to reductions in capacity in the short term. This year, we plan to build capacity by:

• Training relatively senior staff to take on management roles by reallocating Elie Hassenfeld’s management responsibilities to them as much as possible. For example, Senior Research Analyst Natalie Crispin is currently performing all of Elie’s management responsibilities with respect to GiveWell’s 2015 charity reviews and charity updates. Elie is overseeing Natalie during this transition. Since Natalie is managing others on this work, she does not have as much time to directly do research work herself.
• Training relatively junior staff to do most charity updates, intervention reports, and new charity reviews. Most junior staff members are relatively new to this type of work.
• Continuing to hire and train new Research Analysts, Outreach Associates, and Conversation Notes Writers.

We expect that these efforts to build capacity will enable us to do more research – for both GiveWell and the Open Philanthropy Project – in the long run but will reduce the efficiency of our work in the short run, requiring more person-hours per unit of output than in 2014.

Building capacity to do intervention reports

As we wrote in our 2014 self-evaluation post, completing new intervention reports in 2014 was much more difficult and time-consuming than we had anticipated. This year, we are trying to build more capacity for completing these reports by training more staff to do intervention-related research and by improving our process for doing this research. Our ultimate goal is to have a process for completing a reasonable number of intervention reports that does not require substantial involvement from Elie.

We consider building capacity to do more intervention reports to be a high priority because we must be able to complete these reports in order to best prioritize new charities for investigation.

Charity updates

We plan to publish updates on all eight of our recommended charities in 2015. We are generally aiming to have conversations with each charity in February, May, and September that will each result in conversation notes and/or an update report (example). This update schedule may vary somewhat by charity. We are following this charity update schedule so that a) we learn about any major updates that might cause changes in a charity’s recommendation status as soon as possible and b) we spread out the work of refreshing our charity reviews over the course of the year.

As part of our February update calls with recommended charities (especially top charities), we will be gathering more information about charities’ room for more funding situations. As we mentioned in December 2014, we may update our recommended allocation to top charities to reflect any major changes in charities’ funding needs. We tentatively plan to publish such an update in April.

In our charity updates, in addition to our standard questions to follow up on each charity’s activities, we will be focusing in particular on whether any new information from our “standout” charities might lead them to be recommended as “top” charities by the end of the year.

Intervention reports

This year, we hope to complete 2-4 new intervention reports. The programs and program areas that we have preliminarily prioritized for investigation include:

• Nutrition programs (e.g., folic acid fortification and iron fortification)
• Immunization programs (e.g., immunization against measles and meningitis)
• Neglected tropical diseases programs (e.g., trachoma and onchocerciasis)
• Programs for which we believe a charity would apply if we determined the intervention to be a priority program (e.g., “Targeting the Ultra-Poor” (or “Ultra-Poor Graduation”) programs and voluntary medical male circumcision for the prevention of HIV)

We also plan to publish two intervention reports that are near completion: maternal and neonatal tetanus elimination and mass drug administration to eliminate lymphatic filariasis.

A major factor in prioritizing among intervention reports is determining which interventions seem to be most broadly similar to our other priority programs. We believe that such interventions are most likely to succeed in our current process. Factors that seem to be common among our priority programs include:

• The program has strong evidence of effectiveness (preferably from multiple high-quality studies)
• The program is very low-cost per person reached
• Studies of the program’s effects seem not to be overly dependent on the particular context in which the program was implemented (e.g., studies of health programs often seem more likely to be externally valid than studies of education programs because the mechanisms by which health programs have their effects are often more consistent across populations)
• The program is highly replicable for an implementing organization (i.e., a charity would face a low burden of proof to show that they were carrying out the same intervention that was studied and shown to be effective)
• The program has informative proximate outcomes (e.g., deworming pills taken, bed nets delivered and used, etc.) that can be fairly easily measured and monitored

New charity reviews

We plan to actively pursue evaluations of 2-4 new potential top charities this year. Our tentative plans for which charities we may evaluate are below. However, note that there are many reasons that new charity review prioritization could change during the year, such as learning new information from potential recommended charities and completing new intervention reports that change our views on which interventions are promising (several of the new charities that we may evaluate implement interventions for which we have not yet published intervention reports). We also plan to maintain our “open-door policy” for allowing any charity to apply for a recommendation.

As with previous years, we chose to mainly prioritize charities based on our best guess about whether they will become top-rated organizations. We also gave additional weight to organizations that we guessed have some chance of being substantially more cost-effective than our current top charities. The charities that we may evaluate include:

• Project Healthy Children (nutrition technical assistance)
• Micronutrient Initiative (nutrition technical assistance)
• Organizations that work on deworming or other neglected tropical diseases (e.g., Centre for Neglected Tropical Diseases, Sightsavers, The END Fund, Children Without Worms, and Helen Keller International)
• UNICEF MNTE Initiative (maternal and neonatal tetanus elimination)
• Other immunization organizations
• Other nutrition organizations

We have also reached out again to Nothing But Nets about applying for a recommendation because it distributes long-lasting insecticide-treated bed nets, which we consider to be one of the most cost-effective priority programs.

Experimental work

In 2015, we (in collaboration with Good Ventures) plan to continue the experimental work to “seed” additional top charities that we began in 2014, though we still do not consider this work to be a high priority. A few activities that we are considering in this area include:

• Providing funding to promising young charities, such as New Incentives, that could eventually become recommended charities.
• Funding additional research on and support for scale up for programs that a) could be priority programs if they were supported by additional studies and b) could be scaled up with additional support. The main way in which we are currently doing this is our funding of Evidence Action (early conversation, recent conversation, grant page). We have also investigated other potential partners and may take one or more on in the future.
• Funding additional independent monitoring that could increase our confidence in the success of recommended charities’ programs (e.g., deworming, salt iodization) or increase our confidence in other organizations’ ability to carry out priority programs (e.g., if we learned that standard government-led bed net distributions were high-quality, then we might recommend additional funding to large non-profits that fund government bed net distributions). We are initially planning to work with IDinsight on this project.

We plan to explore options and publish updates on our progress on this work throughout 2015.

Improving quality of charity reviews

In general, we feel that our charity reviews are high-quality. However, we believe that there are some ways in which they could be improved. In particular, in 2015, we hope to improve the cost-effectiveness analyses (CEs) (example) and “room for more funding” analyses (example) in our charity reviews, if we have the capacity to do so.

We feel that the quality of our CEs has been acceptable in the past, but we have identified tangible ways to improve them and feel that it is worth using some of our additional capacity to do this because such analyses are relatively important to our charity recommendations. In particular, we aim to:

• Generally improve the transparency and clarity of our CEs.
• Think more carefully about the major inputs that cause a substantial amount of variation in our CEs and ensure that we know as much as we can about those inputs. For example, the proportion of deworming pills that were given to children as part of Schistosomiasis Control Initiative (SCI)‘s campaigns is a relatively important parameter in our CE for SCI, but we did not have as much confidence in our understanding of this parameter as we could have at the end of last year.
• Ensure that we properly account for “leverage” considerations when appropriate (e.g., in our CEs for organizations such as Iodine Global Network and Deworm the World Initiative).
• Ensure that we are applying rules for including and excluding costs and benefits consistently across all CEs (e.g., making sure that we have captured all in-kind donations to all of our charities’ programs in our CEs).

Similarly, we would like to improve the “room for more funding” sections of our charity reviews. To achieve this, we plan to:

• Discuss charities’ funding needs with them earlier in the year so that we can gain as much clarity as possible about their funding situations.
• Standardize the questions that we ask charities about their room for more funding so that we can be more confident that we are making similar comparisons across organizations.

How large is the pipeline of potential top charities and priority programs?We are building a charity review organization with a substantial amount of capacity for conducting research, but it is unclear how many charities that we have not yet reviewed may be competitive with our current top charities.

One factor that affects our estimate of the size of the pipeline is that, over time, we have broadened our criteria and research process so that we are able to evaluate more types of potentially high-impact giving opportunities. For example, in the last two years we have begun to evaluate organizations that play an advocacy and advisory role to governments, such as Deworm the World Initiative and Iodine Global Network. We are also now open to evaluating components of “mega-charities” that are working to scale up potential priority programs, such as UNICEF Maternal and Neonatal Tetanus Elimination Initiative.

We have also seen that charities have been more willing to engage in our review process over time. This may be due to our growing money moved, growing influence, and generally improved incentives for charities to apply (discussed in our 2014 self-evaluation post).

These changes have had the effect of increasing the pipeline of potential charities and programs to evaluate and potentially allowing us to find more cost-effective giving opportunities than we had been able to find previously.

We are also trying to increase the pipeline of outstanding giving opportunities through our experimental work mentioned above (e.g., by funding New Incentives and replications of studies).

Currently, we do not have a strong sense of the overall number of potential top charities and potential priority programs that we have not yet evaluated, but we feel that there are substantially more promising charities and programs than we will be able to evaluate this year and believe that it is possible that the pipeline will grow over time.

Note 1: In this post, senior staff refers to Elie, Holden, and Alexander. Many staff took on additional responsibilities throughout 2014, so this refers to senior staff as of January 2014, not as of today.