We’ve recently encountered some new (to us) information about studies we have relied on in our evaluation of the Schistosomiasis Control Initiative (SCI), which is one of our top charities (and will remain one after our upcoming refresh). We won’t be able to fully investigate all the implications of this information by the end of giving season, but at this point we provisionally hold that:

• We are less confident in SCI’s track record of reducing worm infection prevalence than previously. In assessing SCI’s track record, we have relied on a set of studies showing dramatic drops in worm infection prevalence and intensity coinciding with SCI-run deworming programs; we now believe that most of these studies do not show what we had thought they showed. Specifically, it appears that most of the studies we’ve relied on involved treating children for worms as part of the study (i.e., children who tested positive for infection were treated by research personnel), so improvements measured over time could be attributable to the activities of the people carrying out the study rather than necessarily to the coverage of the national control program. We do not believe this represents an error or purposeful deception – the lead author on the study straightforwardly confirmed this issue to us and defended it based on ethical considerations. However, we believe that the published version of the study was (perhaps unintentionally) not clear on this point.
• We plan to review other evidence of SCI programs’ effectiveness; we have not focused on collecting such evidence because of our impression that the studies we were using were sufficient. It is now a high priority for us to solicit and review such evidence. On November 28 (in response to a draft of this post sent Nov. 11), SCI sent us reports for Burundi and Niger that, on preliminary review, look strong and encouraging to us, though we have not been cleared to share the reports publicly and have not yet done a thorough examination. We plan to review these further and to solicit other monitoring and evaluation data.
• We plan to continue recommending SCI through this year’s giving season. We feel that SCI represents an outstanding giving opportunity, one of the best we are aware of for donors seeking to support evidence-backed programs serving the global poor. More

New information about studies of national deworming programsIn our 2012 review of SCI, we emphasized a set of panel studies in which children at multiple sites were tested for infection (schistosomiasis and in some cases other conditions) before the start of an SCI-led control program, then followed up and tested again one and two years later. The studies show a fairly consistent picture of dramatic drops in disease prevalence. We acknowledged multiple limitations to these studies, but felt that they gave relatively strong suggestive evidence that the SCI-led control programs had had their intended effects.

We recently discovered – via an investigation that started when Sally Murray and Rob Wiblin of Giving What We Can alerted us to the paper, Does mass drug administration for the integrated treatment of neglected tropical diseases really work?: assessing evidence for the control of schistosomiasis and soil-transmitted helminths in Uganda – that in one of these studies (Kabatereine et al. 2007), when children tested positive for infection in the course of the study, they were then directly treated. We confirmed this point over email with Dr. Narcis Kabatereine, the lead author of the study. In the discussion that followed, SCI stated:

The data in Uganda in this study was collected as part of the initial studies supported by the Bill and Melinda Gates Foundation to evaluate the feasibility and impact of mass drug administration for SCH. Due to the nature of the study, which involved drawing of blood and ultrasonography, ethical considerations demanded that treatment had to be offered by investigators. Findings from these and additional studies confirmed the link between SCH infections and morbidity. Data from these studies also provided very valuable information on other aspects of SCH control. Ongoing monitoring and evaluation protocols in Uganda and all other SCI assisted countries were able to use a different methodology and did not require the treatment of all positive cases. (Children in sentinel sites are tested approx. 1 month prior to MDA so that any positive cases will be treated as part of the MDA. This ensures that the sentinel sites accurately reflects the impact of the programme as a whole but also does not leave known positive children untreated, which would be unethical.) Therefore from 2006 onwards studies assessing the impact of SCH mass treatment did not include the treatment of identified positives and therefore were not subject to the theoretical bias that may have been inherent in the original protocols.

On November 28 (in response to a draft of this post sent Nov. 11), SCI sent us reports for Burundi and Niger following the post-2006 methodology that, on preliminary review, look strong and encouraging to us, though we have not been cleared to share the reports publicly and have not yet done a thorough examination. We plan to review these further and to solicit other monitoring and evaluation data.

We feel that a study using the original methodology (not the post-2006 methodology) – treating children who test positive for infection – could easily find the observed declines in schistosomiasis, even if there were no national control program running at all, because such declines could be caused by the practice of treating children for infection as part of the study (and then following up with the same children in later years). Therefore, we believe that the results of such a study cannot be taken as a reflection of the quality or success of a national control program such as those led by SCI.

We were previously unaware that Kabatereine et al. 2007 had involved treating children as part of the study. Dr. Kabatereine pointed out to us in an email that the study did intend to communicate this, pointing to the quote that opens the “Findings” section: “We enrolled 4351 children from 37 schools, of which 2815 (64.7%) were traced and treated at one year follow-up and 1871 (43.0%) at two year follow- up.” Dr. Kabatereine also stated that phrases such as “Anthelmintic treatment delivered as part of a national helminth control programme” imply “Anthelmintic treatment delivered as it is done in a national helminth control programme” (e.g., same drugs, same dosage and frequency).

We now see that the phrase “traced and treated” in the above quote should have raised a flag for us. However, this phrase could have referred to treatment by the national control program (“and treated” seems to refer only to followup rounds, not to the first round of measurement), and it appears in what we see as an unexpected place in the paper: in the opening of the “Findings” section, where the emphasis appears to be on how many children were examined and followed up in each round and what the results were. As far as we can tell, this issue is not noted in any of the sections on methodology or ethics. In addition, the introduction and conclusion of the paper do not mention this issue and strongly seem to us to imply that the intent of the paper was to examine the impact of the national control program, as opposed to the impact of treatment delivered along the same protocol as the national control program. We are inclined, therefore, to chalk the fact that we did not notice this issue up to a lack of clarity in the paper, rather than to a shortcoming on our part. We recognize that others may disagree. For reference, a link to the paper in question is available here.

The upshot is that we no longer see studies such as this one as demonstrating the effectiveness of a national control program as implemented.

We plan to review other evidence of SCI programs’ effectiveness; we have not focused on collecting such evidence because of our impression that the studies we were using were sufficient. It is now a high priority for us to solicit and review such evidence. On November 28 (in response to a draft of this post sent Nov. 11), SCI sent us reports for Burundi and Niger that, on preliminary review, look strong and encouraging to us, though we have not been cleared to share the reports publicly and have not yet done a thorough examination. We plan to review these further and to solicit other monitoring and evaluation data.

Continuing to recommend SCIThis represents a potentially substantial update regarding the evidence of SCI’s track record; we will need to reassess the case for its track record thoroughly. The fact that we weren’t aware of crucial information about how at least one study was conducted reflects concerns we have earlier noted (here and here) about communication difficulties with SCI.

However, we plan to continue recommending SCI through this giving season, because:

• We would still guess, given all the information we have – including the new reports we have seen as well as our impression that school-based deworming programs are among the more straightforward interventions (the latter is based partly on site visits and partly on several years of ongoing conversations with deworming-focused charities) – that the programs led by SCI are relatively effective.
• Our criteria for top charities are shifting to put more emphasis on potential leverage (which we feel we have made progress in understanding) and less emphasis on organization-specific monitoring and evaluation. We will be writing about this more in a future post. By our modified criteria, SCI remains outstanding.

We have substantial work to do in rethinking the evidence regarding SCI’s track record. With that said, we continue to feel that SCI represents an outstanding giving opportunity, one of the best we are aware of for donors seeking to support evidence-backed programs serving the global poor.