Evaluation of Givewell Blogs

GiveWell is dedicated to finding outstanding giving opportunities and publishing the full details of our analysis. In addition to evaluations of other charities, GiveWell publishes substantial evaluation of our own work. This post lays out highlights from our 2015 metrics report, which reviews what we know about how our research impacted donors. Please note:

• We report on “metrics years” that run from February through January; for example, our 2015 data cover February 1, 2015 through January 31, 2016.
• We differentiate between our traditional charity recommendations, our work on the Open Philanthropy Project, and other charitable giving.
• More context on the relationship between Good Ventures and GiveWell can be found here.

Summary of influence: In 2015, GiveWell influenced charitable giving in several ways. The following table summarizes our understanding of this influence.

Total money moved: In 2015, GiveWell tracked $110.1 million in money moved to our recommended charities. Our money moved only includes donations that we are confident were influenced by our recommendations.

Open Philanthropy Project: As part of our work on the Open Philanthropy Project, we advised Good Ventures to make grants totaling $6.4 million. This was in addition to Good Ventures’ support for our recommended charities.

Money moved by charity: Our four top charities received the majority of our money moved. Our four standout charities received a total of $2.2 million.

Money moved by size of donor: In 2015, the number of donors and amount donated increased across each donor size category. In 2015, 95% of our money moved (excluding Good Ventures) came from about 15% of our donors, each of whom gave $1,000 or more.

Donor retention: The total number of donors who gave to our recommended charities or to GiveWell unrestricted increased about 60% year-over-year to 15,274 in 2015. This included 10,669 donors who gave to our recommended charities for the first time. Among all donors who gave in the previous year, about 40% gave again in 2015, up from about 33% who gave again in 2014.

Our retention was stronger among donors who gave larger amounts or who first gave to our recommendations prior to 2013. Of larger donors (those who gave $10,000 or more in either of the last two years), about 80% who gave in 2014 gave again in 2015.

GiveWell’s expenses: GiveWell’s total operating expenses in 2015 were $3.4 million. Our expenses increased from about $1.8 million in 2014 as the size of our staff grew, largely to support the Open Philanthropy Project. We estimate that about one-third of our total expenses ($1.1 million) supported our traditional top charity work and about two-thirds supported the Open Philanthropy Project. In 2014, we estimated that expenses for our traditional charity work were about $900,000.

Donations supporting GiveWell’s operations: Prior to 2013, GiveWell relied on a small number of donors to provide unrestricted support for our operations. Since 2013, we have asked more donors to support our operational costs and asked donors to support us at a higher level than we had in previous years. In 2015, we raised $4.9 million, up from $3.0 million in 2014. Several institutions and the six largest individual donors contributed about two-thirds of GiveWell’s funding in 2015.

Donor demographics: We continued to collect information on our donors. We found the picture of our 2015 donors to be broadly consistent with previous information. Based on reports from donors who gave $2,000 or more, we found:

• The most common ways that donors found us were via Peter Singer and personal referrals.
• About 70% of our donors are under 40, and about 60% work in technology or finance.

Web traffic: Unique visitors to our website increased by 12% in 2015 compared to 2014 (when excluding visitors driven by AdWords, Google’s online advertising product).

For more detail, see our full metrics report (PDF).

The post GiveWell’s money moved and web traffic in 2015 appeared first on The GiveWell Blog.

Over the past couple of years, we’ve put a lot of effort into hiring and training staff and we now have significantly more capacity to do research than we have in past years. Some of our increased capacity will support the Open Philanthropy Project, which we hope will be a separate organization by the end of 2016; its plans for the year will be discussed on the new Open Philanthropy Project blog. We also expect to have more capacity for GiveWell’s work of finding outstanding evidence-based charities.

At the same time, we have come to believe that the kind of work we’ve recently been doing to find top charities – deeply investigating the most promising-seeming charities we know of, based largely on which interventions they carry out – has limited promise. In past years – and at the beginning of this year – we hoped that these investigations would lead relatively quickly to new top charities. Now, we believe that we’ve already (previously) identified most of the strongest charities by our criteria, and there aren’t many strong candidates left (though there are a few that we continue to investigate, and we remain willing and eager to investigate further promising groups if we come across them). With that in mind, we have begun seeing more potential in other research priorities, such as supporting the development of new organizations and new evidence bases.

A future post will elaborate on why we’ve formed this view. This post focuses on laying out our plans for GiveWell’s research work in 2016, taking this view into account.

In brief, in 2016:

• We plan to focus much of our capacity on a small number of initiatives that are unlikely to result in new top charities in 2016, but which we hope will lead to new top charities that are competitive with our current top charities in 2017 or 2018.
• We plan to intensify our work following our current top charities and are tentatively planning to make site visits to distributions funded by the Against Malaria Foundation and work supported by Evidence Action’s Deworm the World Initiative.
• We are also planning a substantial project focused on the question of whether or not we should recommend that Good Ventures give significantly more than it has in the past to support insecticide-treated nets, arguably the most promising area we know of for substantial additional funding.
• We also hope to take on additional work (described in detail below) but plan to prioritize this work below the items listed above.
• We plan to put more staff time into donor outreach than we have in the past and discuss our priorities for that work below.

This plan represents a significant shift from previous years, when our primary goal was improving the list of top charities we published at the end of each year. We plan to write more about the reasoning behind this shift in a future post.

What we’ve done so far this year

In January and February 2016 we:

• Put significantly more effort into getting input on our plans from non-senior staff than we have in past years. To start, we asked staff and a small number of GiveWell followers to make probabilistic predictions about which charities would become top charities if we investigated them this year. The results of this exercise led to this initial list of possible priorities (listed in no particular order). This list represents the organizations that we would guess are most likely to become top charities at the end of 2016. When we later revised our plan, we held a series of staff meetings to discuss details of the plan and what the new plan might be missing.
• Had exploratory conversations with several charities on that list and others in the field of global health and development. The goal of conversations with charities was to explain our application process and ask them basic questions about their programs, monitoring and evaluation, and need for additional funding. The goal of conversations with others working in global health and development was to generate a list of additional organizations to contact.

Ultimately, this work made us more pessimistic that prioritizing work on all the organizations listed above would lead to new top charities by the end of 2016, as discussed in the introduction of this post, and we refined our plans for the year as a result. A future post will elaborate on this development.

Top priorities for research

Our top priorities are:

• Supporting the development of potential future GiveWell top charities: making grants to organizations that could become top charity contenders in the future or supporting research that could lead to more organizations that are a strong fit with our criteria. This work is unlikely to result in new top charities in 2016, but we hope it will lead to new top charities that are competitive with our current top charities in 2017 or 2018. This work might include:
  • Providing early stage funding to organizations that aim to scale up programs with strong evidence of effectiveness and cost-effectiveness. (For example, New Incentives or Evidence Action’s No Lean Season program.)
  • Funding research on programs that are candidates to become priority programs. (For example, this grant to support research on an incentives for immunization program.)
  • Funding organizations that run priority programs to increase or improve their monitoring, or funding a third party to do this monitoring.
• Considering additional funding for insecticide-treated nets: A significant funding gap exists for insecticide-treated nets, and this gap appears to be as cost-effective an opportunity as any other we have found. This project involves determining whether there are high quality opportunities to provide significantly more funding for insecticide-treated nets than we have in the past. It will involve conversations with the major bednet funders (e.g., Global Fund to fight AIDS, Tuberculosis, and Malaria and the President’s Malaria Initiative) and others familiar with how to identify funding gaps for bednets and what the options are for monitoring distributions. We have also been discussing with the Against Malaria Foundation (AMF) what it would take to quickly scale up AMF’s work. The goal of this work is to identify additional funding opportunities for funding insecticide-treated nets in 2016.
• Intervention prioritization: quick investigations on a large number of interventions with the goal of finding more priority programs. We have looked at many interventions historically, but regularly learn of programs that we do not know very much about.
• Current top charities: continuing to follow our current top charities and trying to answer our highest priority unanswered questions about these groups. More on this below.
• New evidence on deworming and bednets. The next round of follow up on a key deworming study is expected to be available later this year and could make a big difference to our view of deworming. We’re also looking more into the degree to which insecticide resistance may be reducing the impact of bednets.

Other research we will undertake if we have the time to do so

• Micronutrient fortification charities. Last year, we tried but were unable to find compelling evidence that the Iodine Global Network (IGN) or the Global Alliance for Improved Nutrition (GAIN) had successfully contributed to the impact of salt iodization programs (write-ups forthcoming). We also began investigating Project Healthy Children. We may continue some of these investigations this year and have also reached out to the Micronutrient Initiative and the Food Fortification Initiative.
• Neglected tropical disease (NTD) charities. We began investigating Sightsavers and END Fund’s work on deworming last year and may continue with those organizations this year and expand the investigations to cover multiple NTDs. We have reached out to the Carter Center and Helen Keller International (HKI) about their NTD programs. HKI declined to participate at this time.
• Surgery charities. We have had several conversations with organizations that work on cataract surgery and we may reach out to organizations that work on obstetric fistula surgery. Our initial impression from these conversations is that it will be very challenging to understand the impact that these charities’ programs have. We may also consider other surgical interventions (such as trachoma).
• Other organizations. If organizations apply for a recommendation and seem sufficiently promising, we will aim to review them.
• Publishing research we largely completed in 2015: updates on standout charities (GAIN, IGN, and Living Goods), interim reviews of charities we began investigating in 2015 (Sightsavers, END Fund, and Project Healthy Children), and intervention reports (folic acid fortification, surgery for cataracts, trachoma and fistula, measles immunization campaigns, mass drug administration for lymphatic filariasis, and “Targeting the Ultra Poor”).

Research we considered but do not expect to undertake

The following investigations are ones that we considered doing this year but don’t currently expect to get to. This could change if some of the higher priority work turns out to be less promising than expected.

• Mega-charities. We could try to work with one or more large organizations with very diverse programs to figure out how to scale-up work on one of our priority programs.
• Charities that work on programs that are probably more cost-effective than cash transfers but not by a large enough margin that it seems worth highly prioritizing work on them.
  • Voluntary medical male circumcision. We are interested in talking to PSI, the only major organization we know of working on this program, but do not plan to prioritize this program beyond that.
  • “Targeting the ultra poor” or “graduation” programs.
  • Lymphatic filariasis.
  • Incentives for immunization. We previously funded research on this program and have been working with IDinsight on a cost-effectiveness analysis.
• Immunization programs. We have put in a fair amount of work into looking for room for more funding for scaling up immunization programs and have largely failed to find opportunities (2012 write-up; recent example).

More detail on potential further research on current recommended charities

One of our top priorities for 2016 is continuing to follow our current top charities and trying to answer our highest priority unanswered questions about these groups. We moved over $100 million to these groups in 2015 and whether we recommend a similar (or greater) level of support in 2016, and how we recommend allocating funds among them, depends on answering: (1) what is our best estimate of the organization’s impact and cost-effectiveness? and (2) how much room for more funding do they have?

Top charities

In past years, we’ve updated our top charity reviews once a year, in November. This year, we plan to refresh these reviews twice, in June and November. As we have at the end of the year, we expect to reconsider what recommendation we make to donors about how to allocate donations amongst our top charities in June.

Summary of our research plans for each of our top charities (note that the strategy documents were written in February):

• Against Malaria Foundation (AMF). We plan to follow AMF’s progress closely in 2016. Key questions include (a) how quickly is AMF committing funding to new distributions, and (b) can we get a more detailed understanding of how data is collected in pre- and post-distribution surveys. More details here.
• Schistosomiasis Control Initiative (SCI). The amount of time we spend on SCI this year depends on whether we see a significant improvement in the quality of SCI’s financial information (how it has spent funds, how much funding it holds, and projected expenses). If it does not improve, we will likely deprioritize much additional work on SCI. If it does, we would be interested in exploring the research questions detailed here.
• Deworm the World Initiative. Of our top charities, we feel that there is the largest gap between what we could know and what we do know for Deworm the World. In particular, we’ve focused on Deworm the World’s work in India, because in the past most unrestricted funds were used in India. Going forward, unrestricted funds will largely be used in new programs. We aim to follow Deworm the World’s progress in new countries closely and to better understand its past work by learning more about its program in Kenya. Details here.
• GiveDirectly. Our main goals from following GiveDirectly are to see if the quality of monitoring remains high, it is able to enroll new recipients quickly, and we can learn more about the impact of its work with partners to make cash a baseline against which other development programs are judged. Details here.

Standout charities

• Development Media International. We’re not planning to consider DMI as a possible top charity in 2016. The results from a randomized controlled trial (RCT) of its program that DMI shared last year were not in line with what we would have wanted to see for DMI to become a top charity. More recently, DMI shared some additional results from the RCT (which are not yet public). We believe that taken together these results provide conflicting evidence for DMI’s impact. DMI stands out for its commitment to transparency and rigorous evaluation and we will consider working with DMI to continue to build the evidence base around behavior change through mass media. We see this as a long-term project that is unlikely to result in DMI’s being a top charity in 2016.
• Iodine Global Network. We are planning to follow up with IGN about a few case studies that IGN thought might provide additional evidence of its impact.
• The Global Alliance for Improved Nutrition (GAIN) – Universal Salt Iodization (USI) program. We’re not planning to consider GAIN’s USI program as a possible top charity in 2016. We have not been able to establish clear evidence of GAIN successfully contributing to the impact of iodization programs, and think it is unlikely that more work on this will be useful.
• Living Goods. It’s fairly unlikely that we will consider Living Goods as a possible top charity in 2016. We would revisit this if we were to see significant improvements in the rigor of Living Goods’ monitoring or if we significantly changed our cost-effectiveness estimate for its work.

Plans for donor outreach

We have not historically prioritized outreach at GiveWell, instead choosing to devote staff capacity primarily to our research work. Now, with the addition of new research staff as well as the continued growth of GiveWell’s donor base, we feel it is appropriate to dedicate more capacity to outreach for GiveWell in service of our mission to make our research available to help individuals decide where to give.

In 2016, we plan to have 1.5 staff members devoted to outreach related to GiveWell and the Open Philanthropy Project. Due to this being early on in our outreach work, we’re tentatively planning to reassess our priorities every month for the first half of the year, and then every quarter. As of the publication of this blog post, we expect the following to be top priorities for GiveWell outreach in 2016:

• Donor calls and meetings. We expect that connecting with individuals who have donated to GiveWell will be an important part of our outreach going forward, although as we’re relatively new to prioritizing this, we plan to survey donors about whether this is something that they find useful. We’re hoping to learn more about the donors who use our work and any questions or feedback they have, as well as to offer an opportunity for donors to stay up to date on GiveWell’s work. More here.
• Launching a redesigned website. The redesign will largely improve the look and feel of the site with some minor improvements in navigation and content organization.
• Improving GiveWell’s written communications. This includes:
  • Revisiting and refreshing content on our website (e.g., a recent update to our criteria page) to ensure it’s up-to-date and clearly presented, particularly for individuals who aren’t familiar with our research.
  • Publishing content to our blog, in the hopes of highlighting research and providing additional insight into our values, process, and findings. We will need to put more effort into writing blog posts in order to maintain our previous pace of about one blog post per week, since many types of blog posts that previously appeared here will now be appearing on the Open Philanthropy Blog.

The post GiveWell research plans for 2016 appeared first on The GiveWell Blog.

This post reviews and evaluates last year’s progress on our traditional work of finding and recommending evidence-based, thoroughly vetted charities that serve the global poor. It has two parts. First, we look back at the plans we laid out in early 2015 and compare our progress against them, providing details on some of the most significant accomplishments and shortcomings of the year. Then, we reflect on the overall impact of our traditional work and critically evaluate some of our major strategic decisions. In our next post, we will cover our plans for GiveWell’s work in 2016.

SummaryIn brief, when evaluating ourselves against the goals we laid out in early 2015, we feel that we broadly achieved our primary goals for the year while we generally fell short on several of our secondary goals.

The overall impact of GiveWell’s recommendations continued to increase substantially in 2015, as we tracked more than $100 million that was donated to our recommended charities as a direct result of our research.

This self-evaluation post focuses primarily on how we have grown as an organization and the recent strategic decisions we have made since these are most relevant to thinking about the impact of recent GiveWell-focused work.

Our progress in 2015 relative to our plansIn our “2015 plan” blog post, we wrote:

This year, our primary goals are to:

• Build management and research capacity for GiveWell’s traditional work while further reducing senior staff time (note 1) spent on this work, primarily by reallocating Elie Hassenfeld’s management responsibilities related to GiveWell’s traditional work.
• Maintain our core research product by completing updates on all eight 2014 recommended charities and determining which of them should be recommended as top charities for the 2015 giving season.

Our secondary goals for 2015 are to:

• Continue to seek outstanding giving opportunities by reviewing 2-4 new charities and publishing 2-4 new intervention reports.
• Improve the cost-effectiveness analyses and room for more funding analyses in charity reviews.
• Finish and launch a redesigned GiveWell website.
• Make further progress on experimental work to “seed” potential recommended charities.We expect our total output on “top charities” work to be roughly comparable to last year’s, despite a growing staff, because (a) a major focus of the coming year is training, and we expect to trade some short-term efficiency for long-run output; (b) we may be reallocating some capacity from our “top charities” work to the Open Philanthropy Project this year.

We feel that we broadly achieved our primary goals for the year, while we fell short on several of our secondary goals.

Goals that we feel we accomplished include:

• Building management and research capacity for GiveWell’s traditional work while further reducing the time that Elie spent on this work. Specifically:
  • All GiveWell staff track how they spend time at work. In 2014, Elie spent 36% of his time on GiveWell’s traditional work, and in 2015, Elie spent 26% of his time on this work.
  • In 2014, Elie was primarily responsible for managing all research staff. In 2015, Natalie Crispin and Josh Rosenberg took on research management responsibilities, and at the end of the year, managed 10 research staff between them. Timothy Telleen-Lawton and Eliza Scheffler also took on management responsibilities, primarily for operations-focused staff.
  • Most of the other staff working on GiveWell’s traditional work in 2015 were relatively new. Because they were new, most of the time they spent was focused on training. We discuss staff time allocations in greater detail below.
• Maintaining our core research product. We completed and published updates on all four top charities from 2014. We also conducted deeper investigations of 3 out of 4 of our standout charities (Global Alliance for Improved Nutrition (GAIN), Iodine Global Network (IGN), and Development Media International (DMI)), as planned, in an effort to determine whether they should be recommended as top charities for the 2015 giving season. We ultimately did not recommend any of these charities as top charities. We did not publish our updates on GAIN and IGN, and those are still forthcoming.
• Improving our room for more funding analyses in charity reviews, which we wrote about extensively in our November 2015 blog post announcing our updated recommendations.
• Continuing to make limited but steady progress on experimental work to “seed” potential recommended charities via grants to Evidence Action’s No Lean Season Program, New Incentives, and two randomized controlled trials focused on incentives for immunization.

Areas in which we fell short include:

• We did not publish updates (or even reach tentative conclusions internally) about any new potential recommended charities, and we published only 2 new intervention reports. We engaged with Project Healthy Children, UNICEF, The END Fund, Episcopal Relief & Development’s NetsforLife® Program, and Sightsavers. (We published a page on Children Without Worms, but it was extremely short.) We published 2 new intervention reports — on vitamin A supplementation and tetanus immunization campaigns — but both of these largely relied on work we had completed in 2014.
• As discussed in our November 2015 blog post announcing our updated recommendations, we felt more confident in our cost-effectiveness analyses (CEAs) at the end of 2015 than we had in previous years, and we made some of the improvements we planned. Nevertheless, we don’t believe that we made as much progress as we had hoped for when we set the goal to improve our CEAs in early 2015. In particular, key inputs and judgment calls in our CEA were decided on by different staff members at different points in time, which made it more difficult than ideal for individual staff members to understand all details of the CEA. We hope to resolve this issue by making at least one staff member responsible for fully understanding and explaining all aspects of our core CEAs in the future.
• We did not finish and launch a redesigned GiveWell website. The GiveWell website is large and it took longer than the firm we worked with expected to complete a version that was working properly. The firm we contracted with also built the Open Philanthropy Project website, and in September 2015, we explicitly prioritized completing the Open Philanthropy Project (“Open Phil”) site before the GiveWell site.

Overall, we succeeded in passing significant responsibilities from Elie to others, but we saw somewhat less overall research output than we had hoped for. Part of this was because, in the process of transferring responsibilities, we made some mistaken decisions – i.e., in some cases staff put substantial work into assignments that we ultimately determined were not a fit for them.

Other self-evaluation questionsWhat was GiveWell’s overall impact in 2015?

The overall impact of GiveWell’s recommendations continued to increase substantially in 2015. We tracked more than $100 million that was donated to our recommended charities as a direct result of our research. Excluding Good Ventures’ giving, we moved more than $30 million to our recommended charities. (More details on our 2015 money moved will be in our forthcoming 2015 metrics blog post.) This was a major increase relative to 2014, when we tracked about $27.8 million in money moved to our recommended charities, with about $13 million coming from non-Good Ventures donors.

Does our impact justify our staff size, in absolute terms?

As mentioned above, in 2015 we moved more than $100 million to our recommended charities. Over the same period, we spent approximately $3.8 million on our operations, of which $1.3 million was spent on GiveWell’s traditional work and $2.5 million on the Open Philanthropy Project.

We previously wrote that we believe that expenses that are 15% of money moved are well within the range of normal, so we feel comfortable with the relative size of our operating expenses at this point.

Is recent growth in staff justified? Has growth in output matched growth in staff?

Staff size has grown significantly over the past few years: from 11 people at the end of 2013 to 32 at the end of 2015. We think it is reasonable to question whether this growth has been justified and productive, in light of the fact that:

• We haven’t significantly increased the number of new major reports (charities or interventions reviewed in depth). We estimate that we published 4 major new reports in 2013, 5 in 2014, and 6 in 2015 (details on our rough estimate in this Google sheet including year-to-date information for 2016).
• Most of our 2015 impact (in terms of money moved) likely could have been achieved with reduced research work in 2015. For example, if we had only stayed up-to-date on our past top charities, we would have been in a position to have the same top charities list as what we ultimately published at the end of 2015. The bulk of additional research work that we did in 2015 did not seem to have significant direct impact, largely because the new giving opportunities that we investigated did not lead to new top charities, though it would have been difficult to confidently predict that this would happen in advance (we reflect on whether we chose the best research priorities in 2015 below).

On the other hand:

• Since 2013, we have greatly expanded the Open Philanthropy Project, which is currently formally housed at GiveWell though we hope to separate the organizations in 2016. Much of our increase in staff size has been for the Open Philanthropy Project.
• Many of our most senior staff have switched over to working primarily on the Open Philanthropy Project, and senior staff are particularly challenging to replace. We don’t think we could have maintained the same output if we had simply hired one new person for each senior staff member who switched over.
• Capacity building – hiring, training and evaluating new staff – is itself a time-consuming and long-term project. In addition, increased staff size creates the need for more work in and of itself. For example: (1) in 2015, we outgrew our office and had to find new office space. Finding new office space was only necessary because of our staff size. (2) Because our staff is larger and we have a larger office, we now need an office manager. (Early in our history, we didn’t have an office, or we worked in a shared workspace where someone else played this role.)
• As our money moved grows, it becomes more worthwhile to be more thorough (as we believe we have been), and it becomes more worthwhile to try to find new top charities even if doing so is a long-term and uncertain proposition. While the work we did in 2015 did not lead to any immediate new top charities, we believe that some of it may have significant returns in the future (e.g., the “GiveWell experimental” work discussed above and the preliminary work that we did on new charities and interventions that we have not finished evaluating).

We previously discussed our reasoning about why we believe it is worthwhile to continue to expand GiveWell’s research capacity despite limited recent returns in last year’s self-evaluation post. This year, we decided to take a more in-depth look at specifically where our increased capacity has gone, while adjusting for staff seniority. Specifically, we roughly estimated the number of full-time, co-founder-equivalent people working on each area in each year. (Co-founder time was given a score of 1 but time spent by newer staff was scaled down significantly; for example, we multiplied time spent by entry-level staff in their first year at GiveWell by 0.1. These figures are far from precise, and are intended only to give a rough guess at our changes in total capacity over time.)

We’ve divided our work into several categories:

• GiveWell’s traditional research: time spent on evaluating new charities or interventions and updating our work on existing top charities or interventions.
• Open Phil: time spent on work related to the Open Philanthropy Project.
• Other output: work that directly furthers GiveWell’s and the Open Philanthropy Project’s missions but isn’t research related. Among other things, this includes work related to donation processing (e.g., opening the mail, entering donations into our database, sending thank you receipts), donor communication (answering donor questions, staying in touch with larger donors), and general outreach (giving talks, speaking to the media, and improving our web content). All of these categories of work have increased significantly as our public profile has grown.
• Overhead: work necessary due to the increased size of our staff, such as changing offices and having an office manager (as mentioned above).
• Capacity building: time spent by relatively senior staff to increase our future capacity. This includes time spent (a) recruiting (e.g., reviewing resumes, interviewing and evaluating candidates) and (b) training and evaluating new staff with the goal of increasing future capacity, not generating additional short-term research output. For example, in 2015, Elie estimates that he spent approximately 60% of his time building capacity.
• Training: time spent by newer staff being trained. These staff members are primarily working on projects where the goal is training and evaluation, not short-term research output.

The table shows that our overall capacity has increased significantly since 2012 (from approximately 3 co-founder equivalents then to 9 today). Most of that increase has gone to Open Phil (3.4 co-founder equivalents), capacity building (.8), and overhead (.9). The number of co-founder equivalents focused on GiveWell’s traditional work has stayed low (and relatively constant) since 2012.

Year GiveWell trad-itional research Other output Capacity building Train-ing Open Phil Over-head Total co-founder equiv. Total staff at year-end 2012 0.9 0.4 0.7 0 0.7 0.2 2.9 5.5 2013 0.8 0.6 0.5 0.3 1.4 0.3 3.9 11 2014 0.9 0.4 0.8 0.6 2.6 0.4 5.7 18 2015 0.9 0.7 1.5 1.1 4.1 1.1 9 32

Overall, we find this picture reasonable. While our work on GiveWell’s traditional research has stayed relatively constant in the short term, we think of it as a major benefit of increased staff that we’ve been able to do this while building the Open Philanthropy Project. And we also believe we’ve been laying the groundwork for future capacity increases, since much of our increased staff time has gone into capacity building itself.

Over time, as new staff train and are able to produce more, we hope that this picture begins to shift and that we can devote more capacity to GiveWell’s traditional research, other output, and Open Phil. Our guess is that we will start to see some of this shift in 2016.

Did we make mistakes in allocating staff capacity in 2015?

We see mistakes in our allocation of staff capacity in 2015. Specifically, we didn’t move as quickly as we should have to move staff from one type of work to another. We had several cases where we continued to try to train staff to work on research projects even though we should have already recognized that they were unlikely to be a good long-term fit for thIs type of research, and would be a better fit for another role. It is generally difficult to predict fit, so we classify this as a relatively easy-to-make mistake, though a costly one, since capacity building is a large time investment for relatively senior staff.

Did we make the right choices about how to spend the time that we had for research (i.e., did we choose the best research priorities)?

In addition to staying up to date on our top charities, we prioritized researching the below opportunities with the goal of maximizing our chances of identifying new top charities.

• Deworming organizations. At the end of 2014, we felt that deworming was one of the most promising interventions we were aware of, but the giving opportunities we had found to scale up deworming had limited room for more funding. In 2015, we prioritized finding new deworming organizations and began evaluations of the END Fund, Sightsavers, and Children Without Worms.
• Development Media International (DMI). At the end of 2014, we believed there was a significant probability that we would name DMI a top charity at the end of 2015. We prioritized work on DMI with this in mind, but the results from DMI’s randomized controlled trial were not in line with what they would have needed to be in order for DMI to become a top charity.
• Micronutrient fortification organizations. We prioritized this area because micronutrient fortification programs reach many people at relatively low cost and they are often backed by strong evidence of effectiveness. We prioritized evaluating salt iodization programs run by IGN and GAIN. We also began an evaluation of Project Healthy Children’s programs and invited the Micronutrient Initiative (MI) to apply for a recommendation. MI declined to participate.
• Bednet organizations. We invited Nothing but Nets to apply for a recommendation (it declined) and we began an evaluation of Episcopal Relief & Development’s NetsforLife® Program.

We believe these priorities were reasonable, given the amount of time we had available and the information we had at the time. However, as these investigations – and further investigations in 2016 – have had fairly disappointing results (in terms of not resulting in new top charities), we are rethinking our approach significantly going forward. More on this in a future post.

Have we failed to publish materials that we should?

We have struggled to publish information about many research projects that we have essentially completed. For example, we have formed views on several interventions for which we have not yet published intervention reports. One major reason for this is that we often try to resolve almost all of our major questions about an intervention before publishing a report on that intervention. A possible solution would be to publish more materials that explicitly note that we are sharing preliminary views that could be substantially affected by additional research. We plan to experiment with publishing more research about preliminary views we’ve reached in the future.

However, a continuing challenge in publishing our research work is that we have limited management capacity to review and sign off on public write-ups. We hope that building management and research capacity will steadily reduce this bottleneck over time.

The post GiveWell’s progress in 2015 appeared first on The GiveWell Blog.

In addition to evaluations of other charities, GiveWell publishes substantial evaluation of itself, from the quality of its research to its impact on donations. We publish quarterly updates regarding two key metrics: (a) donations to recommended charities and (b) web traffic. This post is being published late due to staff focusing on updating GiveWell’s charity recommendations in the fourth quarter; it also includes a preliminary view of our money moved since the end of our third quarter.

Preliminary estimate of 2015 money moved (since February 1, 2015)

As of early January 2016, we have tracked about $98 million in money moved to our recommended charities. Excluding Good Ventures, we have tracked about $28 million (of which, roughly half has come from donors giving $1 million or more).

These data are preliminary. We expect that in some cases we are currently overstating our impact (e.g. due to double counting or incorrect attribution of our influence) and in other cases we are understating our impact (since there are several weeks left in our metrics year and there are delays entering data); overall, we would guess that we are currently underestimating our annual money moved. We plan to publish our annual metrics (covering February 1, 2015 – January 31, 2016) in March, at which point we will have more confidence in our data and be able to share more details.

GiveWell’s web traffic / money moved through Q3 2015

The tables and chart below present basic information about our growth in money moved and web traffic in the first three quarters of 2015 compared to the previous two years (note 1).

Money moved and donors: first three quarters

Money moved by donors who have never given more than $5,000 in a year increased about 80% to $2.44 million. The total number of donors in the first three quarters increased about 80% to about 8,300 (note 2). These growth rates are reasonably consistent with the growth we previously reported in our first and second quarter metrics.

Web traffic through October 2015

Growth in web traffic excluding Google AdWords increased about 25% in the first three quarters. Last year, we saw a drop in total web traffic because we removed ads on searches that we determined were not driving high quality traffic to our site (i.e. searches with very high bounce rates and very low pages per visit).

GiveWell’s website receives elevated web traffic during “giving season” around December of each year. To adjust for this and emphasize the trend, the chart below shows the rolling sum of unique visitors over the previous twelve months, starting in December 2009 (the first period for which we have 12 months of reliable data due to an issue tracking visits in 2008).

We use web analytics data from two sources: Clicky and Google Analytics (except for those months for which we only have reliable data from one source). The raw data we used to generate the chart and table above (as well as notes on the issues we’ve had and adjustments we’ve made) is in this spreadsheet (note 3, on how we count unique visitors).

Note 1: Since our 2012 annual metrics report we have shifted to a reporting year that starts on February 1, rather than January 1, in order to better capture year-on-year growth in the peak giving months of December and January. Therefore, metrics for the “first three quarters” reported here are for February through October.

Note 2: Our measure of the total number of donors may overestimate the true number. We identify individual donors based on the reported name and email. Donors may not share all of this information or may update it (for example, using a different email), in which case, we may mistakenly treat a donation as if it was made by a new donor. We plan to investigate how large of an overstatement there may be and possibly adjust the total for our next annual metrics report.

Note 3: We count unique visitors over a period as the sum of monthly unique visitors. In other words, if the same person visits the site multiple times in a calendar month, they are counted once. If they visit in multiple months, they are counted once per month.

The post Update on GiveWell’s web traffic / money moved appeared first on The GiveWell Blog.