This post reviews and evaluates last year’s progress on our traditional work of finding and recommending evidence-based, thoroughly vetted charities that serve the global poor. It has two parts. First, we look back at the plans we laid out in early 2015 and compare our progress against them, providing details on some of the most significant accomplishments and shortcomings of the year. Then, we reflect on the overall impact of our traditional work and critically evaluate some of our major strategic decisions. In our next post, we will cover our plans for GiveWell’s work in 2016.

SummaryIn brief, when evaluating ourselves against the goals we laid out in early 2015, we feel that we broadly achieved our primary goals for the year while we generally fell short on several of our secondary goals.

The overall impact of GiveWell’s recommendations continued to increase substantially in 2015, as we tracked more than $100 million that was donated to our recommended charities as a direct result of our research.

This self-evaluation post focuses primarily on how we have grown as an organization and the recent strategic decisions we have made since these are most relevant to thinking about the impact of recent GiveWell-focused work.

Our progress in 2015 relative to our plansIn our “2015 plan” blog post, we wrote:

This year, our primary goals are to:

• Build management and research capacity for GiveWell’s traditional work while further reducing senior staff time (note 1) spent on this work, primarily by reallocating Elie Hassenfeld’s management responsibilities related to GiveWell’s traditional work.
• Maintain our core research product by completing updates on all eight 2014 recommended charities and determining which of them should be recommended as top charities for the 2015 giving season.

Our secondary goals for 2015 are to:

• Continue to seek outstanding giving opportunities by reviewing 2-4 new charities and publishing 2-4 new intervention reports.
• Improve the cost-effectiveness analyses and room for more funding analyses in charity reviews.
• Finish and launch a redesigned GiveWell website.
• Make further progress on experimental work to “seed” potential recommended charities.We expect our total output on “top charities” work to be roughly comparable to last year’s, despite a growing staff, because (a) a major focus of the coming year is training, and we expect to trade some short-term efficiency for long-run output; (b) we may be reallocating some capacity from our “top charities” work to the Open Philanthropy Project this year.

We feel that we broadly achieved our primary goals for the year, while we fell short on several of our secondary goals.

Goals that we feel we accomplished include:

• Building management and research capacity for GiveWell’s traditional work while further reducing the time that Elie spent on this work. Specifically:
  • All GiveWell staff track how they spend time at work. In 2014, Elie spent 36% of his time on GiveWell’s traditional work, and in 2015, Elie spent 26% of his time on this work.
  • In 2014, Elie was primarily responsible for managing all research staff. In 2015, Natalie Crispin and Josh Rosenberg took on research management responsibilities, and at the end of the year, managed 10 research staff between them. Timothy Telleen-Lawton and Eliza Scheffler also took on management responsibilities, primarily for operations-focused staff.
  • Most of the other staff working on GiveWell’s traditional work in 2015 were relatively new. Because they were new, most of the time they spent was focused on training. We discuss staff time allocations in greater detail below.
• Maintaining our core research product. We completed and published updates on all four top charities from 2014. We also conducted deeper investigations of 3 out of 4 of our standout charities (Global Alliance for Improved Nutrition (GAIN), Iodine Global Network (IGN), and Development Media International (DMI)), as planned, in an effort to determine whether they should be recommended as top charities for the 2015 giving season. We ultimately did not recommend any of these charities as top charities. We did not publish our updates on GAIN and IGN, and those are still forthcoming.
• Improving our room for more funding analyses in charity reviews, which we wrote about extensively in our November 2015 blog post announcing our updated recommendations.
• Continuing to make limited but steady progress on experimental work to “seed” potential recommended charities via grants to Evidence Action’s No Lean Season Program, New Incentives, and two randomized controlled trials focused on incentives for immunization.

Areas in which we fell short include:

• We did not publish updates (or even reach tentative conclusions internally) about any new potential recommended charities, and we published only 2 new intervention reports. We engaged with Project Healthy Children, UNICEF, The END Fund, Episcopal Relief & Development’s NetsforLife® Program, and Sightsavers. (We published a page on Children Without Worms, but it was extremely short.) We published 2 new intervention reports — on vitamin A supplementation and tetanus immunization campaigns — but both of these largely relied on work we had completed in 2014.
• As discussed in our November 2015 blog post announcing our updated recommendations, we felt more confident in our cost-effectiveness analyses (CEAs) at the end of 2015 than we had in previous years, and we made some of the improvements we planned. Nevertheless, we don’t believe that we made as much progress as we had hoped for when we set the goal to improve our CEAs in early 2015. In particular, key inputs and judgment calls in our CEA were decided on by different staff members at different points in time, which made it more difficult than ideal for individual staff members to understand all details of the CEA. We hope to resolve this issue by making at least one staff member responsible for fully understanding and explaining all aspects of our core CEAs in the future.
• We did not finish and launch a redesigned GiveWell website. The GiveWell website is large and it took longer than the firm we worked with expected to complete a version that was working properly. The firm we contracted with also built the Open Philanthropy Project website, and in September 2015, we explicitly prioritized completing the Open Philanthropy Project (“Open Phil”) site before the GiveWell site.

Overall, we succeeded in passing significant responsibilities from Elie to others, but we saw somewhat less overall research output than we had hoped for. Part of this was because, in the process of transferring responsibilities, we made some mistaken decisions – i.e., in some cases staff put substantial work into assignments that we ultimately determined were not a fit for them.

Other self-evaluation questionsWhat was GiveWell’s overall impact in 2015?

The overall impact of GiveWell’s recommendations continued to increase substantially in 2015. We tracked more than $100 million that was donated to our recommended charities as a direct result of our research. Excluding Good Ventures’ giving, we moved more than $30 million to our recommended charities. (More details on our 2015 money moved will be in our forthcoming 2015 metrics blog post.) This was a major increase relative to 2014, when we tracked about $27.8 million in money moved to our recommended charities, with about $13 million coming from non-Good Ventures donors.

Does our impact justify our staff size, in absolute terms?

As mentioned above, in 2015 we moved more than $100 million to our recommended charities. Over the same period, we spent approximately $3.8 million on our operations, of which $1.3 million was spent on GiveWell’s traditional work and $2.5 million on the Open Philanthropy Project.

We previously wrote that we believe that expenses that are 15% of money moved are well within the range of normal, so we feel comfortable with the relative size of our operating expenses at this point.

Is recent growth in staff justified? Has growth in output matched growth in staff?

Staff size has grown significantly over the past few years: from 11 people at the end of 2013 to 32 at the end of 2015. We think it is reasonable to question whether this growth has been justified and productive, in light of the fact that:

• We haven’t significantly increased the number of new major reports (charities or interventions reviewed in depth). We estimate that we published 4 major new reports in 2013, 5 in 2014, and 6 in 2015 (details on our rough estimate in this Google sheet including year-to-date information for 2016).
• Most of our 2015 impact (in terms of money moved) likely could have been achieved with reduced research work in 2015. For example, if we had only stayed up-to-date on our past top charities, we would have been in a position to have the same top charities list as what we ultimately published at the end of 2015. The bulk of additional research work that we did in 2015 did not seem to have significant direct impact, largely because the new giving opportunities that we investigated did not lead to new top charities, though it would have been difficult to confidently predict that this would happen in advance (we reflect on whether we chose the best research priorities in 2015 below).

On the other hand:

• Since 2013, we have greatly expanded the Open Philanthropy Project, which is currently formally housed at GiveWell though we hope to separate the organizations in 2016. Much of our increase in staff size has been for the Open Philanthropy Project.
• Many of our most senior staff have switched over to working primarily on the Open Philanthropy Project, and senior staff are particularly challenging to replace. We don’t think we could have maintained the same output if we had simply hired one new person for each senior staff member who switched over.
• Capacity building – hiring, training and evaluating new staff – is itself a time-consuming and long-term project. In addition, increased staff size creates the need for more work in and of itself. For example: (1) in 2015, we outgrew our office and had to find new office space. Finding new office space was only necessary because of our staff size. (2) Because our staff is larger and we have a larger office, we now need an office manager. (Early in our history, we didn’t have an office, or we worked in a shared workspace where someone else played this role.)
• As our money moved grows, it becomes more worthwhile to be more thorough (as we believe we have been), and it becomes more worthwhile to try to find new top charities even if doing so is a long-term and uncertain proposition. While the work we did in 2015 did not lead to any immediate new top charities, we believe that some of it may have significant returns in the future (e.g., the “GiveWell experimental” work discussed above and the preliminary work that we did on new charities and interventions that we have not finished evaluating).

We previously discussed our reasoning about why we believe it is worthwhile to continue to expand GiveWell’s research capacity despite limited recent returns in last year’s self-evaluation post. This year, we decided to take a more in-depth look at specifically where our increased capacity has gone, while adjusting for staff seniority. Specifically, we roughly estimated the number of full-time, co-founder-equivalent people working on each area in each year. (Co-founder time was given a score of 1 but time spent by newer staff was scaled down significantly; for example, we multiplied time spent by entry-level staff in their first year at GiveWell by 0.1. These figures are far from precise, and are intended only to give a rough guess at our changes in total capacity over time.)

We’ve divided our work into several categories:

• GiveWell’s traditional research: time spent on evaluating new charities or interventions and updating our work on existing top charities or interventions.
• Open Phil: time spent on work related to the Open Philanthropy Project.
• Other output: work that directly furthers GiveWell’s and the Open Philanthropy Project’s missions but isn’t research related. Among other things, this includes work related to donation processing (e.g., opening the mail, entering donations into our database, sending thank you receipts), donor communication (answering donor questions, staying in touch with larger donors), and general outreach (giving talks, speaking to the media, and improving our web content). All of these categories of work have increased significantly as our public profile has grown.
• Overhead: work necessary due to the increased size of our staff, such as changing offices and having an office manager (as mentioned above).
• Capacity building: time spent by relatively senior staff to increase our future capacity. This includes time spent (a) recruiting (e.g., reviewing resumes, interviewing and evaluating candidates) and (b) training and evaluating new staff with the goal of increasing future capacity, not generating additional short-term research output. For example, in 2015, Elie estimates that he spent approximately 60% of his time building capacity.
• Training: time spent by newer staff being trained. These staff members are primarily working on projects where the goal is training and evaluation, not short-term research output.

The table shows that our overall capacity has increased significantly since 2012 (from approximately 3 co-founder equivalents then to 9 today). Most of that increase has gone to Open Phil (3.4 co-founder equivalents), capacity building (.8), and overhead (.9). The number of co-founder equivalents focused on GiveWell’s traditional work has stayed low (and relatively constant) since 2012.

Year	GiveWell trad-itional research	Other output	Capacity building	Train-ing	Open Phil	Over-head	Total co-founder equiv.	Total staff at year-end
2012	0.9	0.4	0.7	0	0.7	0.2	2.9	5.5
2013	0.8	0.6	0.5	0.3	1.4	0.3	3.9	11
2014	0.9	0.4	0.8	0.6	2.6	0.4	5.7	18
2015	0.9	0.7	1.5	1.1	4.1	1.1	9	32

Overall, we find this picture reasonable. While our work on GiveWell’s traditional research has stayed relatively constant in the short term, we think of it as a major benefit of increased staff that we’ve been able to do this while building the Open Philanthropy Project. And we also believe we’ve been laying the groundwork for future capacity increases, since much of our increased staff time has gone into capacity building itself.

Over time, as new staff train and are able to produce more, we hope that this picture begins to shift and that we can devote more capacity to GiveWell’s traditional research, other output, and Open Phil. Our guess is that we will start to see some of this shift in 2016.

Did we make mistakes in allocating staff capacity in 2015?

We see mistakes in our allocation of staff capacity in 2015. Specifically, we didn’t move as quickly as we should have to move staff from one type of work to another. We had several cases where we continued to try to train staff to work on research projects even though we should have already recognized that they were unlikely to be a good long-term fit for thIs type of research, and would be a better fit for another role. It is generally difficult to predict fit, so we classify this as a relatively easy-to-make mistake, though a costly one, since capacity building is a large time investment for relatively senior staff.

Did we make the right choices about how to spend the time that we had for research (i.e., did we choose the best research priorities)?

In addition to staying up to date on our top charities, we prioritized researching the below opportunities with the goal of maximizing our chances of identifying new top charities.

• Deworming organizations. At the end of 2014, we felt that deworming was one of the most promising interventions we were aware of, but the giving opportunities we had found to scale up deworming had limited room for more funding. In 2015, we prioritized finding new deworming organizations and began evaluations of the END Fund, Sightsavers, and Children Without Worms.
• Development Media International (DMI). At the end of 2014, we believed there was a significant probability that we would name DMI a top charity at the end of 2015. We prioritized work on DMI with this in mind, but the results from DMI’s randomized controlled trial were not in line with what they would have needed to be in order for DMI to become a top charity.
• Micronutrient fortification organizations. We prioritized this area because micronutrient fortification programs reach many people at relatively low cost and they are often backed by strong evidence of effectiveness. We prioritized evaluating salt iodization programs run by IGN and GAIN. We also began an evaluation of Project Healthy Children’s programs and invited the Micronutrient Initiative (MI) to apply for a recommendation. MI declined to participate.
• Bednet organizations. We invited Nothing but Nets to apply for a recommendation (it declined) and we began an evaluation of Episcopal Relief & Development’s NetsforLife® Program.

We believe these priorities were reasonable, given the amount of time we had available and the information we had at the time. However, as these investigations – and further investigations in 2016 – have had fairly disappointing results (in terms of not resulting in new top charities), we are rethinking our approach significantly going forward. More on this in a future post.

Have we failed to publish materials that we should?

We have struggled to publish information about many research projects that we have essentially completed. For example, we have formed views on several interventions for which we have not yet published intervention reports. One major reason for this is that we often try to resolve almost all of our major questions about an intervention before publishing a report on that intervention. A possible solution would be to publish more materials that explicitly note that we are sharing preliminary views that could be substantially affected by additional research. We plan to experiment with publishing more research about preliminary views we’ve reached in the future.

However, a continuing challenge in publishing our research work is that we have limited management capacity to review and sign off on public write-ups. We hope that building management and research capacity will steadily reduce this bottleneck over time.