Note: Consistent with our usual practices, SCI reviewed a draft of this post prior to publication, but the final product is ours.

We wrote last year about reevaluating studies we relied on in our evaluation of the Schistosomiasis Control Initiative (SCI), which is one of our top charities. We noted at the time that we were planning to continue learning about the studies. We have now revisited these studies and discussed them in depth with SCI. We have two main takeaways from this investigation:

1. We now feel that some of the panel studies we previously relied on in our evaluation of SCI do not in fact provide evidence that SCI’s national programs have reached a large proportion of children targeted. We wrote last year that in one of the four studies, participants received extra deworming treatment from researchers if they were found to be infected with worms. We learned this year that participants in at least one of the other three studies received treatment separately from and under closer supervision than other students in the country. Where participants received more treatment or more careful treatment than other students in the country, we believe that the results of the studies do not reflect the treatment coverage achieved by the national programs.
2. We are concerned about SCI’s external communications around these panel studies. We first published the view that all of SCI’s panel studies provided evidence of effective national programs in 2009, along with our interpretation of how the studies were carried out. It is concerning to us that (a) the published papers on the studies, which had SCI staff as coauthors, imply or explicitly say that the studies reflect the performance of the national programs, (b) SCI did not correct this interpretation, which we published and asked them to review, and (c) when we asked SCI about the methodological issue specifically, SCI leadership gave us information that SCI program staff later contradicted. 

SCI has told us that it has standardized and made significant improvements to its procedures for more recent monitoring, including treating children at sentinel schools as part of MDAs. SCI recently shared with us a more recent panel study and two studies of MDA coverage rates. We plan to write about these studies in our upcoming review of SCI.

On the whole, we continue to view SCI as an outstanding giving opportunity, and it will likely maintain its status as a top charity when we refresh our recommendations in December.

Our updated views on four SCI panel studies

In our 2011 and 2012 recommendations of SCI, we emphasized a set of four panel studies from Uganda, Burkina Faso, Niger, and Burundi. In each of the studies, a set of sentinel schools was chosen, and children at each sentinel school were tested for worm infections before the start of an SCI-led control program. The same children were followed up and tested again in subsequent years. The studies show a fairly consistent picture of dramatic drops in disease prevalence among the study participants. We acknowledged multiple limitations to these studies, but felt that they gave relatively strong suggestive evidence that the SCI-led control programs had reached a large proportion of the children they targeted.

Last year, we learned that in the Uganda study (Kabatereine et al. 2007), study participants who tested positive for infection in the course of the study were directly treated by the researchers. Thus, improvements measured over time could be attributable to the activities of the researchers carrying out the study rather than necessarily to the coverage of the national control program.

This year, we have revisited the methodology used in the other three studies through conversations with SCI. We have learned that the Burkina Faso study has similar methodological issues to the Uganda study. We are unsure whether researchers participated in treatment in the studies in the other two countries.

In the Burkina Faso study, which began in 2004, drug administration at the sentinel sites was not part of the national mass drug administration (MDA) program. Researchers working for the national schistosomiasis control program supervised teachers as they administered deworming treatment to all children in the sentinel schools on the day researchers took samples, a few weeks before the MDA. SCI told us that the national control program staff ran the study in this way to meet ethical guidelines laid out by the countries’ ethics boards as well as to produce more impressive results than might have been obtained by treating children as part of the national MDA. In addition to their ethical commitment and strong desire to ensure that all children tested were treated, the researchers were more knowledgeable about deworming than the district-level health workers who administered treatment in the MDA. Supervision from the researchers may have caused teachers to do a more thorough job administering deworming treatment. For this reason, and because drug administration in sentinel schools was logistically separate from the MDA, we do not consider this panel study to reflect the coverage achieved by the national MDA.

We are unsure whether participants in the Niger study received treatment in the same manner as the other students throughout the country, or whether they were treated separately under the supervision of the researchers as in Burkina Faso. SCI leadership originally told us that participants were treated in the same way as other students in the country. SCI’s current Program Manager for Niger and Burkina Faso (who was not the Program Manager at the time of the study) later said that participants were treated separately under the supervision of the researchers (note 1). After we discussed the Program Manager’s comments with SCI leadership, SCI contacted Dr. Amadou Garba, National Coordinator of the Schistosomiasis and Soil-transmitted Helminth Control Program in Niger at the beginning of the panel study. According to SCI, he stated that participants were treated as part of the MDA. We believe that participants were most likely treated as part of the MDA, but feel that we cannot be sure given the conflicting positions expressed by SCI staff.

Note that we believe that each SCI staff member we spoke to told us what they believed to be the truth about the methodology of the studies, and we do not think that any of them intentionally misled us.

SCI’s Program Manager in Burundi confirmed SCI leadership’s statement that the children in the sentinel schools there received treatment in the same manner as children throughout the country (note 2).

Note that other substantial concerns about these studies remain even in cases where children in sentinel schools were treated as part of the national MDAs. As we wrote in our 2012 review of SCI, we are concerned because the sentinel sites selected may not be representative of the whole country, and because only about half of the students initially surveyed were followed up. We are also concerned that treatment teams and teachers may have known which schools were sentinel schools and may have administered deworming treatment more carefully in the sentinel schools (note 3). We have had these concerns since first evaluating the studies and they should not be taken as an update, but it is important to note that they are still live concerns.

Additionally, given that it took us years to discover the issues regarding different protocols for treating children at sentinel sites, we feel that it is reasonably likely that the Niger and Burundi studies have weaknesses that we have not yet discovered.

We have seen little monitoring data outside of the panel studies that we believe shows that national programs supported by SCI reach a high proportion of the children they attempt to treat. Thus, we are unsure of the impact of SCI’s programs. Also, considering that the Burkina Faso study does not seem to reflect the coverage achieved by the MDA, we now have somewhat lower confidence that SCI has been able to effectively use its research results to improve its programs.

SCI’s external communications about the four studies

Our conversations with SCI staff about the panel studies have reduced our confidence in SCI’s external communications, particularly its communication with us. There are three reasons for this:

1. The published papers on the studies, which had SCI staff as coauthors, either imply or explicitly say that the Uganda and Burkina Faso studies reflect the performance of the national programs, even though we now believe that the studies do not necessarily reflect the performance of the national programs (note 4).
2. We treated the panel studies as central evidence of the impact of SCI’s MDAs in our 2011 and 2012 reviews of SCI. SCI vetted those reviews but did not note to us that in some studies, children in the sentinel schools were treated separately from the national MDAs.
3. This year, we asked SCI’s leadership about whether the Niger and Burkina Faso studies had methodological issues similar to the Uganda study. They told us, “All students were treated as part of the national treatment program (at the same time and with the same treatment strategy) as the purpose of the sentinel sites was to assess the impact of the national control program.” When we later spoke with program staff who had been involved in the studies, they contradicted this picture (though we believe that all SCI staff told us what they believed to be the truth).

We previously noted difficulties communicating with SCI. We feel that our struggle to communicate effectively with SCI about the panel studies was more serious than previous difficulties. We credit SCI with connecting us with staff who could provide more detailed answers to our questions, even where those answers contradicted SCI leadership. Still, the fact that it took this much time and effort to gain this information reflects poorly on our ability to communicate with SCI.

More recent studies

The Uganda and Burkina Faso studies began in 2003 and 2004 respectively. SCI has continued to collect monitoring data as part of its ongoing programs. SCI has told us that it has standardized and made significant improvements to its procedures for more recent monitoring, including treating children at sentinel schools as part of MDAs. SCI recently shared with us a more recent panel study and two studies of MDA coverage rates. We have begun to analyze these new studies and feel that they likely provide some degree of additional evidence that national programs supported by SCI achieve relatively high coverage. We plan to write about these studies in our upcoming review of SCI.

Bottom line

Our concerns about SCI’s evidence of effectiveness and its external communications discussed in this post cause us to take a less positive view of SCI. However, we continue to believe that the program SCI supports, combination deworming, is among the most cost-effective programs we have considered and that the program has room for more funding globally. We recently wrote about a new study that seems to bolster the evidence for the long-term effects of deworming. In addition, SCI recently sent us the more recent studies discussed above and hosted us for three days of meetings with staff to update and expand our understanding of SCI’s work. When we refresh our top charity rankings later this year, we will likely include SCI.

Note 1: “In 2004 both Niger and Burkina Faso received funds from the Bill & Melinda Gates Foundation through the SCI to establish a national Schistosomiasis and STH control program. In addition to the financing of mass drug administration (MDA), a research lab was built in each country with the purpose of monitoring and evaluating the impact of the national deworming program. Assessment of the MDA was carried out through a six-year sentinel site survey from 2004-2010. Statistical power calculations were performed to ensure that the correct number of schools and children per school were sampled to be representative of all schools receiving the same treatment strategy. At baseline (in 2004), each of the sentinel schools were sampled by a team from the research lab who arrived in the morning to take samples from 150 students selected at random, which were then taken back to the lab for analysis. In the afternoon, one or two lab staff remained at the school to supervise the deworming drug administration and to ensure that all students in the school were treated. Ethical review boards in the countries mandated that the members of the lab team personally ensure treatment of children who had been sampled for the study. The lab staff who administered treatment were aware that the school was a sentinel site. These same children were then followed up each sub sequential year. If there were any children that were lost-to-follow up then new children who were entering into the first year of school at sentinel schools were recruited into the study. They were then representative of non-treated children. Children in the sentinel schools received treatment under the supervision of the lab staff each year. The lab staff were unaware of which students had been found to have worms to avoid preferential treatment. Children who were found to have worms were not given any extra treatment beyond the treatment given to the entire sentinel school. The sentinel sites were treated two weeks prior to the national MDA to ensure that the schools were not treated twice. SCI were able to then monitor registers from the national campaign showing which schools had received treatment in the MDA to ensure that the sentinel schools were not twice treated. For the non-sentinel schools, the National Schistosomiasis and STH Control Program were responsible for coordinating the mass treatment for all schools in each district. Cascade training was carried out where teachers were trained at the district level on how to administer the deworming drugs. The teachers in turn were responsible for treating the children with supervision from district level health staff. The district level health staff were in turn trained and supervised by the central level Ministry of Health.” Non-verbatim summary of a conversation with Anna Phillips on May 27, 2014.