This year, Dr. Kevin Croke, a post-doctoral fellow at the Harvard School of Public Health, released a study that we consider an important addition to the evidence for deworming children. The study (Croke 2014) followed up on a randomized controlled trial (RCT) of a deworming program in Uganda and found higher scores on tests of literacy and numeracy in children living in treatment areas compared to the control 7 to 8 years later. This finding reinforces the findings of the only other RCT examining the long-term effects of deworming, which we had previously considered to be relatively strong but still had substantial reservations about. By providing a second data point that is free of some of our previous concerns, Croke 2014 substantially changes our view of the evidence.
Two of our top charities, the Deworm the World Initiative (DtWI) led by Evidence Action and the Schistosomiasis Control Initiative (SCI), focus on deworming. We have not yet concluded our examination of Croke 2014, but at this point we think it is likely to lead us to view these charities as more cost-effective.
Overview of Croke 2014
Croke 2014 follows up on an RCT that involved 48 parishes (the administrative level above the village and below the sub-county) in 5 districts of Uganda selected based on the prevalence of worms in the districts. Half of the parishes were randomly assigned to a treatment group and the other half to a control. In all the districts, community organizations delivered basic health services, like vitamin A supplementation, vaccination and growth monitoring, through regular Child Health Days (CHDs). Children in the treatment group received albendazole (a deworming drug) during CHDs in addition to the other services offered, while children in the control just received the usual services.
Croke analyzed surveys conducted by an education nonprofit several years later that happened to sample 22 of the parishes in the original RCT. He compared children living in the treatment parishes sampled who were 1 to 7 years old (the age group offered albendazole) at the time of the program to children of the same age living in the control parishes and found children in the treatment group had scores about 1/3 of a standard deviation higher on tests of literacy and numeracy.
Strengths and significance
Few other studies have rigorously examined the long-term effects of deworming. Up until now, we’ve relied heavily on two studies: (a) Bleakley 2004, a study of the Rockefeller Sanitary Commission’s campaign to eradicate hookworm in the American South in the early 20th century; (b) a series of studies in Kenya, in which school deworming was rolled out on a purposefully arbitrary (randomization-like) basis, and children who received more years of deworming were compared to children who had received fewer. These studies suggest the possibility that deworming children dramatically improves their productivity later in life by subtly improving their development throughout childhood. In our view, the case for deworming largely rests on these long-term, developmental effects, because the intervention seems to have few obvious short-term benefits.
Having two relatively recent RCTs from Sub-Saharan Africa increases our confidence in long-term benefits far more than having just one RCT, especially because we have had substantial reservations about the RCT in Kenya – some of which seem notably less applicable to Croke 2014. Specifically:
- The earlier RCT was a trial of “combination deworming” – treatment of both schistosomiasis (with praziquantel) and soil-transmitted helminths (with albendazole). Croke 2014 looks only at albendazole. This is particularly important because one of our current top charities – the Deworm the World Initiative – operates largely in India, where only albendazole is used.
- Regarding the earlier study, we also thought it was plausible that efforts to encourage students to attend school in order to receive treatment might have accounted for some of the effect found in Baird et al 2012 (the follow-up of Miguel and Kremer 2004). The intervention examined in Croke 2014 appears far less likely to introduce other positive changes into the treatment group, because it involves the addition of albendazole to an existing program rather than an intensive program of deworming in schools in the treatment compared to the absence of any program in the control.
- We worried that the results of Miguel and Kremer 2004 (the RCT in Kenya) might not generalize to other areas, because of extraordinary flooding caused by the El Niño climate pattern during the study and abnormally high infection rates in the study area (more). Croke 2014 appears to have had a lower (though still high) initial prevalence of infections (the programs selected districts based on high rates of worms found by Kabatereine et al 2001, which estimated that 60% of children ages 5-10 were infected, primarily with hookworm). El Niño may still have affected the study, however, because the parishes examined in Croke 2014 are very close (some within about 10 miles) to the district in which Miguel and Kremer 2004 took place. The program evaluated in Croke 2014 started about 2 years after El Niño, but we’re not sure whether this amount of lag time would lead to lower or higher infection rates.
In our current cost-effectiveness analyses for deworming, we have a “replicability adjustment” to account for the possibility that Baird et al 2012 wouldn’t necessarily hold up on replication, as well as an “external validity adjustment” to account for the fact that most deworming programs likely take place in less heavily infected areas. We will be revisiting both of these adjustments, resulting in improved estimated cost-effectiveness for deworming.
Remaining questions
We still have some concerns about the evidence.
- El Niño may have affected the parishes examined in Croke 2014 just as it affected the schools in Miguel and Kremer 2004, potentially causing unrepresentatively high infection rates and limiting the generalizability of both studies.
- Though Croke 2014 finds a large increase in test scores, Baird et al 2012 does not.
- We worry about the sensitivity of the results to the outcome and control variables used in regressions and about selective reporting of results.
- We also worry about publication bias. Perhaps other parish-level surveys would have supplied other outcomes. We wonder if other analyses employing a similar methodology that did not find an effect would have been published.
- The study included a relatively small number of clusters. Croke 2014 reports on a few different regressions and methods of calculating the standard error of the treatment effect, which lead to different estimates of that standard error. In one more conservative analysis, for instance, the effect on the combined literacy and numeracy test scores is significant at the 90% confidence level.
- Finally, we still can’t articulate any mechanism for the long-term benefits of deworming supported by data (we haven’t seen notable impacts on weight or other health or nutrition measures).
Bottom line
We have not yet concluded our examination of Croke 2014, though we have looked it over closely enough to feel that it is very likely to result in a substantial positive revision of our view on deworming, and therefore of our views on two current top charities as well.
In combination with the earlier study, Croke 2014 represents a major update regarding the case for deworming; we’re very glad to see this new evidence generated, and hope that it will become a prominent part of the dialogue around deworming. We intend to do our part by updating our content for this giving season.
Comments
“Regarding the earlier study, we also thought it was plausible that efforts to encourage students to attend school in order to receive treatment might have accounted for some of the effect found in Baird et al 2012”
Is that unique to the study? If it happens for deworming in general, then it’s a perfectly valid reason to support deworming.
Daniel – good point. We don’t know how the intensity of the RCT encouragement efforts compares to the intensity of normal encouragement efforts, though our prior is that the RCT encouragement efforts would be stronger.
In Croke’s study:
Few parishes are observed (24 per group), and there aren’t many schools per parish, so it seems like the effect could conceivably be accounted for by an uneven distribution of school quality between the groups. The study doesn’t look at average scores by school, so it’s hard to know if this is going on.
(A friend of mine noticed this; I’m just sharing it here)
I was suprised to read that this is only the second RCT looking at the long term effects of de-worming. What are Give Well’s thoughts on funding similiar studies given the impact that for example this study appears to have on our understanding of cost effectiveness? Are there other areas where funding an RCT would have a disproportionate impact because of the lack of existing evidence? Would there be value in Give Well drawing up a wish list of studies it would like to see and making it public?
Thanks for the comments.
Ted – In addition to randomization making it less likely for large differences in school quality to drive the effect, the age groups exposed to the most years of deworming seem to have the largest difference in test scores between treatment and control (See Figure 1 in Croke 2014). You might expect the oldest age groups to have a large difference in test scores too if the treatment parishes had better schools but they don’t seem to.
Tristam – We’re not sure that the costs of an additional RCT studying the long-term effects of deworming would be worth it given that the cost of deworming is so low and it would take around 10 years before the results would be available. During this time, it’s possible that that mass drug administration campaigns would have already reached most of the children that need deworming. The advantage of Croke 2014 is that it combines existing evidence to generate a new finding (and accordingly doesn’t require a long delay between treatment and results). We have thought about other areas where additional evidence could be particularly helpful to us, and are working with Evidence Action and IDinsight on related projects.
Like Tristram, I was also surprised to discover there have been so few RCTs of deworming, particularly since GiveWell state “We have a high standard for evidence: we seek out programs that have been studied rigorously and repeatedly”.
This seems in direct contradiction of the relative weakness of the evidence for deworming. Obviously this extra RCT helps, but GiveWell were recommending deworming before this.
It seems like there must be charities out there that have stronger evidence to support them than the current recommendations *but* are less cost-effective. So do GiveWell try to weight them by the probability that they will have an impact? Would a program that was estimated to have 25% chance of working be ranked above one that had 90% chance of working but that was 10x less cost-effective than the first (all else being equal)?
For example if the cost-effectiveness of the first program was given a value of ‘100’, the expected value of supporting this program would be 25. The second program, being 10x less cost-effective, would have a value of 10, with an expected value of 9.
Matt – thanks for the comment.
In our cost-effectiveness estimates, we follow a process very similar to the one you describe. The adjustments we make based on our guesses at external validity and replicability are discussed in this blog post on the cost-effectiveness of our 2012 top charities. We’ve continued to use a similar process in assessing the cost-effectiveness of our current top charities; our comparison spreadsheet is available from this 2013 blog post.
As a minor addendum, there have been a number of shorter term RCTs of deworming, but they don’t show the impacts that the longer-term studies have; we discuss that evidence here. I think there are very few RCTs of any interventions in developing countries that follow up more than five years later, so I’m not sure we should expect more than 2 long-term deworming RCTs.
Thanks Alexander.
I’ve just noticed that Evidence Action are reporting another study on deworming, this time looking at externalities. More specifically, it looks at the impact on children whose older siblings have previously been dewormed. Apparently they “found that there is strong evidence of large cognitive improvements among children whose siblings were dewormed. These were not even children who were treated themselves – they were the untreated younger siblings.”
Have GiveWell looked at this yet?
http://evidenceaction.org/the-evidence-for-impacts-of-deworming-grows/
Hi Matt – we discuss a previous version of Ozier’s paper here. Because it draws on the same underlying experiment in Kenya as Baird et al. 2012, we don’t see it as changing our “external validity” or “replicability” adjustments in the same way that evidence from other places or other experiments does.
To Jake and the rest of the GiveWell Team
As discussed here is my blog on the Croke et al paper. https://www.givingwhatwecan.org/blog/2014-12-08/some-questions-cognitive-benefits-deworming
Please feel free to comment, repost ect.
Comments are closed.