Comparison of administrative and survey data for estimating vitamin A supplementation and deworming coverage of children under five years of age in Sub‐Saharan Africa

Introduction

Vitamin A deficiency and parasitic worm infections are common in developing countries and pose serious threats to the health, growth and development of millions of African children 1, 2. Twice‐yearly vitamin A supplementation (VAS) and deworming of young children are key public health strategies for controlling these problems and are regularly provided in most Sub‐Saharan African countries 2-4. However, for these interventions to have a public health impact, adequate coverage is required. For VAS, evidence suggests an impact on child mortality when at least 80% of target children are reached 5.

VAS and deworming administrative coverage figures generated from health system reports compiled from tally sheets completed by health workers during distribution campaigns are generally available and timely data for monitoring coverage and are therefore the most commonly used indicators of programme performance. However, concerns exist surrounding the quality of administrative data, particularly the potential for systematic overestimation of coverage due to various factors that may affect the coverage calculation. Concerns about the reliability of health system‐reported data have led to the implementation of representative population‐based household surveys to provide complementary coverage measures. Comparisons of administrative and survey‐based coverage estimates from vaccine programmes indicate higher coverage reported from health facility tally sheets 6.

Since 2010, Helen Keller International (HKI) has conducted post‐event coverage (PEC) surveys in several African countries to estimate VAS and deworming coverage. These surveys provide a method to validate administrative figures and are important for identifying barriers to achieving high coverage. Comparisons of administrative and PEC survey data have revealed sizable discrepancies in VAS coverage in the African context 7, 8. However, this has not been rigorously evaluated, and little is known about coverage differences between data sources for specific VAS delivery strategies and child age groups. There is also limited knowledge regarding the level of agreement between administrative and survey data for deworming coverage in the region. In this article, we report on a multicountry study that compared administrative coverage with PEC survey estimates for VAS and deworming campaigns for children <5 conducted over six years in Sub‐Saharan Africa.

Methods

Two delivery strategies were employed for the provision of vitamin A supplements and deworming treatment. Door‐to‐door distribution was used in Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Niger, Senegal and Sierra Leone. In DRC, Kenya, Mozambique, Nigeria and Tanzania, the delivery of VAS and deworming occurred through health facilities and outreach distribution points for remote communities, known as a ‘fixed‐site plus outreach’ distribution model. Country campaigns were implemented in twice‐yearly rounds.

Administrative coverage figures for VAS and deworming campaigns conducted from 2010 to 2015 were obtained from health authorities in the 12 countries. These data, recorded by service providers on tally sheets during campaign distributions, represented the proportion of children supplemented/treated in each campaign and were calculated based on the number of doses administered and the estimated number of children in the target age population. PECs are representative household surveys that employ random sampling of 30 clusters selected using probability proportional to size sampling and 30 households with children 6–59 months of age per cluster based on the WHO EPI cluster sampling methodology 9. Surveys were administered by HKI‐trained enumerators at the household level to caregivers of age‐eligible children (VAS: 6–59 months; deworming: 12–59 months) within 6 weeks of the end of the most recent campaign. VAS and deworming status were recorded based on documentation in the child's health card or from caregiver recall. Required approvals were obtained in each country for the use of administrative data and for conducting the PEC surveys.

Descriptive statistics were calculated for VAS and deworming coverage. Paired t‐tests were used to examine VAS and deworming coverage differences for each administrative‐survey dyad. Independent sample t‐tests were used to compare VAS and deworming coverage between data sources for the two distribution strategies (door‐to‐door and fixed‐site plus outreach), the twice‐yearly campaign rounds and two child age groups (6–11 and 12–59 months). Mean coverage estimates are reported with 95% confidence intervals, and α = 0.05 was used to indicate statistical significance. Only dyads with available data from both administrative records and PEC surveys were included in the analysis. SPSS software version 20.0 (IBM Corporation, Armonk, NY) was used for all analyses.

Results

Administrative coverage data were compared with PEC survey estimates for 52 VAS and 34 deworming dyads. Health system‐reported coverage was higher than PEC estimates in 47 of 52 (90%) VAS campaigns and was <10% higher in 16 dyads, 10–20% higher in 16 dyads, 21–50% higher in 12 dyads and >50% higher in three comparisons (Table 1). Discrepancies >30% were observed in eight of 12 (67%) countries. Mean (±SD) VAS administrative coverage for children 6–59 months was 96.7% ± 21.8% and highly variable across individual country campaigns, ranging from ~12% (Guinea) to ~175% (Burkina Faso) and exceeding 100% in 28 of 52 (54%) rounds (Table 1, Figure 1). Average PEC survey coverage for the 6‐ to 59‐month age group was 80.6% ± 16.4% across the 52 VAS campaigns. The mean difference in coverage between the two data sources was 16.1% (95% CI: 9.5–22.7; P  <  0.001). Average differences between administrative and PEC survey data across all respective surveys were ≥20% in seven countries (Figure 2).

For deworming, administrative coverage exceeded PEC survey estimates in 31 of 34 (91%) comparisons. The magnitude of difference was <10% higher in two dyads, 10–20% higher in 10 dyads, 20–50% higher in 13 dyads and >50% higher in six comparisons (Table 1). Gaps between administrative and survey coverage were >30% for at least one campaign in seven of 10 countries. Health system‐reported coverage for children 12–59 months averaged 97.9% ± 39.9% and was also variable, ranging from ~20% to ~270% in campaigns in DRC and Senegal, respectively, and surpassing 100% in 17 of 34 (50%) rounds (Table 1, Figure 3). Mean deworming coverage calculated from the PEC surveys was 68.1% ± 21.6% (Figure 3) and, in the paired‐sample analysis, the mean difference in deworming coverage between the two data sources was 29.8% (95% CI: 16.9–42.6; P  <  0.001).

In the analysis by delivery strategy, mean (±SD) VAS administrative coverage was 100.1% ± 21.2% for door‐to‐door (n  =  37) distribution and 88.4% ± 21.8% for the fixed‐site plus outreach (n  =  15) model. Mean PEC survey coverage estimates for VAS were 87.9% ± 9.5% for the door‐to‐door and 62.5% ± 16.1% for the fixed‐site plus outreach approaches (Table 2). Average differences in coverage between administrative and PEC survey data were 12.2% ± 22.5% for door‐to‐door and 25.9% ± 24.7% for fixed‐site plus outreach supplementation (P  =  0.06). For deworming, health system‐reported coverage averaged 106.1% ± 43.7% for door‐to‐door (n  =  23) and 80.8% ± 24.1% for fixed‐site plus outreach (n  =  11) delivery. Mean PEC coverage estimates were 77.9% ± 10.7% for the door‐to‐door strategy and 47.7% ± 24.7% for fixed‐site plus outreach distribution (Table 2). Mean differences between administrative and PEC survey coverage were 28.1% ± 43.5% for door‐to‐door and 33.1% ± 17.9% for fixed‐site plus outreach supplementation (P  =  0.64).

In the analysis by age group, VAS administrative coverage was higher than PEC survey estimates in 37 of 49 (76%) comparisons for the 6‐ to 11‐month age group and 45 of 48 (94%) comparisons for the 12‐ to 59‐month age group. Mean (±SD) tally sheet coverage was 92.3% ± 20.1% for children 6–11 months in the door‐to‐door group and 114.8% ± 36.3% for this age cohort supplemented through the fixed‐site plus outreach model. PEC survey coverage estimates for the 6‐ to 11‐month age group were 86.0% ± 12.7% and 64.6% ± 18.8% for the door‐to‐door and fixed‐site plus outreach strategies, respectively (Table 3). For these infants, mean coverage differences between administrative and PEC survey data were 5.1% ± 20.1% in the door‐to‐door group and 52.9% ± 40.1% in the fixed‐site plus outreach group (P  =  0.001). For children 12–59 months, health system‐reported VAS coverage was 104.1% ± 15.3% for the door‐to‐door group and 84.7% ± 23.1% for the fixed‐site plus outreach group. PEC survey coverage estimates for these older children were 88.2% ± 9.3% for door‐to‐door and 62.8% ± 16.4% for fixed‐site plus outreach supplemented children (Table 3). Mean differences in coverage between the data sources were 16.2% ± 18.8% for door‐to‐door and 26.8% ± 18.6% for fixed‐site plus outreach (P  =  0.08).

Child VAS and deworming campaigns are conducted twice‐yearly in most countries, generally in May‐June (round 1) and November‐December (round 2). In the dyads included in our study, mean (±SD) VAS tally sheet coverage was 94.9% ± 25.5% for round 1 (n  =  23) and 98.1% ± 18.7% for round 2 (n  =  29) campaigns. PEC survey coverage estimates were 80.6% ± 16.3% for round 1 and 80.6% ± 16.8% for round 2 campaigns (Table 4). Mean differences in coverage between administrative and PEC survey data were 14.4% ± 25.1% for round 1 supplementation and 17.6% ± 22.9% for round 2 distributions (P  =  0.64). For deworming, average tally sheet coverage was 105.1% ± 47.5% for round 1 (n  =  16) and 91.5% ± 31.7% for round 2 (n  =  18) distributions. Mean PEC survey coverage was 71.7% ± 18.8% for round 1 and 65.0% ± 23.9% for round 2 campaigns (Table 4). Coverage differences between the data sources averaged 33.4% ± 45.7% for round 1 and 26.5% ± 27.9% for round 2 supplementation (P  =  0.60).

Discussion

In this study, we compared coverage figures from health facility reports and post‐campaign household surveys for 52 VAS and 34 deworming campaigns. Overall, we detected poor agreement between the data sources for the proportion of children <5 reached with both interventions. For VAS and deworming, respectively, ~60% and ~90% of dyads examined showed a bias above 10 percentage points in the direction of administrative data, our a priori determination of programmatic significance. Moreover, on average, PEC survey coverage was 16% lower than administrative coverage for VAS and 30% lower for deworming. In 25% of the VAS comparisons, administrative data showed coverage >80%, while the surveys produced estimates below this threshold. Notwithstanding these substantial data source discrepancies, importantly, PEC survey estimates indicated VAS coverage >80% in two‐thirds of the administrative‐survey dyads, suggesting a child survival benefit of the intervention in some areas.

Our findings are consistent with previous research showing higher VAS coverage reported in health system data than in estimates generated through independent household surveys. Dhillon et al . 10 showed national figures derived from aggregated tally sheet data overestimated VAS coverage by ~30% in Tanzania. A study by Katcher et al . 7 revealed higher VAS coverage reported in administrative records than PEC surveys, in 46 of 49 (94%) VAS campaigns in 13 African countries, with percentage differences ranging from <1% to 170%.

In our study, differences between administrative coverage and survey estimates were larger where the provision of VAS and deworming occurred through a fixed‐site plus outreach delivery model. This was not unexpected as data recording may be less accurate where distribution takes place during routine health facility and outreach activities in which health workers are responsible for providing multiple health‐promoting interventions to various populations. This is in contrast to door‐to‐door delivery, where dose administration and documentation may be easier, and eligibility criteria (e.g. age and residence) more strictly enforced. Higher VAS coverage in a fixed‐site plus outreach programme context has been observed in the African region. Maïmouna et al . 11 showed ~ 60% lower VAS coverage in a post‐campaign survey, compared to facility‐reported coverage, during a Child Health Week campaign in Niger. The lack of difference in administrative and survey coverage between the twice‐yearly campaign rounds in our study suggests little seasonal variation in coverage tabulations in these settings.

Although specific factors underlying the discordance in coverage between facility reports and survey data were not investigated in our study, parallels can be drawn between our findings and studies on childhood immunisation coverage in the African context that have shown higher administrative coverage as compared to survey‐based estimates 12-14. Challenges in estimating coverage from routine data have been attributed to a lack of data accuracy and completeness resulting from systematic errors in recording doses administered (numerator) by service providers, inaccurate target population (denominator) figures from which to compute coverage estimates, miscalculations in compiling and aggregating data and purposeful over‐reporting of coverage where incentives and/or pressures from authorities are utilised to achieve performance targets 15-18.

While inflated target population estimates make coverage rates appear falsely low, underestimated denominators produce misleadingly high percentages. Outdated population data are commonly used in the absence of current national data and robust vital registration systems in low‐income countries 19. This is a particular challenge at the subnational level where intracountry population growth rates may vary due to large population movements, as in much of Sub‐Saharan Africa. In such areas, denominator values for coverage indicators are commonly calculated based on population projections from the latest census data 20. These projections involve applying correction adjustments derived from assumptions about the rate of population growth including fertility, mortality and migration patterns. In a study by Brown et al . 21 involving data from 47 Sub‐Saharan African countries, proportions of children <5 reported by national immunisation programmes were compared with corresponding values computed from United Nations 22 population estimates. The results revealed higher coverage rates calculated from nationally reported data, with ≥10% differences in 15 (32%) countries and ≥ 20% differences in seven (15%) countries. In addition to the likelihood of denominator‐related challenges in the countries included in our study, the fact that coverage levels exceeded 100% in more than 50% of the VAS and deworming rounds we examined suggests tally sheets may have included doses administered to children outside the target age rage and/or to those residing outside enumerated areas, leading to inflated numerators and higher than actual coverage. A study in Tanzania revealed 24% of children were erroneously administered deworming treatment under 12 months of age 10. Deworming tablets are also routinely provided to pregnant and lactating women in many countries, with distribution often recorded in child tally sheets, which may contribute to this finding.

A strength of our study is that the results are based on a large multicountry data set for VAS and deworming coverage in the Sub‐Saharan African region. Comparable survey methodologies were used across countries and across time periods within countries. Although surveys are prone to sampling errors, the strong trend towards lower PEC survey coverage suggests minimal measurement error. Notwithstanding the sizable discrepancies observed between administrative and survey data for both VAS and deworming coverage, our study was likely underpowered to detect significant differences between the two delivery strategies due to the small sample size and underlying data variability, which likely resulted in a loss of precision in both groups. However, the low overall and strategy‐specific agreement between data sources has programmatic implications which should be considered. A limitation of our study is not all administrative‐survey comparisons represented the same country‐year dyads. However, we do not believe this alters the interpretation of our results as the divergent trend is apparent across time periods and geographic contexts. Further, as recall bias and social desirability bias are inherent in survey‐based research, PEC coverage estimates may over or under‐represent actual practice, depending on the knowledge level and extent of false reporting by respondents. As the PEC surveys did not distinguish VAS and deworming status based on caregiver recall or documentation in the child's health card, we are not able to speculate on the potential impact of recall bias in our study.

Health system reporting is the primary mechanism for assessing and monitoring coverage achieved during VAS and deworming campaigns. The consistently higher administrative coverage observed in our study suggests substantial and widespread overestimation of health facility data for both interventions in Sub‐Saharan African countries, with potentially larger differences depending on the delivery approach used. These findings have important public health implications as fewer children may actually be protected against vitamin A deficiency and parasitic worm infections. False reassurances from erroneously high coverage figures may inform misguided policies and programme decisions surrounding child health priorities. Examples such as Cameroon and Tanzania are particularly obvious, where coverage has been reported as reaching ~100%, but has shown to be below the 80% threshold for reduction in child mortality.

Administrative coverage data are routinely used by programme managers and donors to assess programme performance. However, health system and survey data may provide very different pictures of coverage for vitamin A supplementation and deworming treatment and, consequently, suggest very different courses of action. Reliance on health facility data alone may mask low coverage and prevent measures to improve programmes in the African region. Countries should periodically validate administrative coverage estimates with population‐based methods and identify factors influencing gaps between routinely collected and survey data, which may not be homogeneous across countries. Finally, such measures should continue in parallel with investments in improving health information collection and management in resource‐poor African settings.