Accuracy and quality of immunization information systems in forty-one low income countries

Introduction

Routine immunization is one of the most cost-effective public health interventions (The World Bank 1993) to reduce child mortality (Jones et al. 2003). Global immunization coverage of systematic vaccines has been steadily increasing since the eighties (WHO 2006a,b). However, global figures mask great inequalities between geographical regions and population sectors (Pearson 2003; WHO 2006a,b). It has been estimated that almost 18 million infants have not received the first dose of Diphtheria-Tetanus-Pertussis vaccine (DTP), half of them living in Southeast Asia and one third of them in Africa (Anonymous 2006).

The GAVI Alliance (GAVI), launched in the year 2000, is one of the global health partnerships that have emerged in recent years aiming at improving access to known effective health care interventions (Walt & Buse 2000). GAVI’s mission is to save children’s lives and to protect people’s health through the widespread use of vaccines (GAVI; http://www.vaccinealliance.org/General_Information/About_alliance/index.php). It focuses on the 72 countries (in 2006) with a Gross National Income (GNI) per capita below 1000 USD (GAVI; http://www.vaccinealliance.org/Support_to_Country/Who_can_Apply/index.php), where most of the unimmunized children live (WHO 2006a,b).

GAVI’s support to countries includes immunization services (ISS), injection safety, new and underused vaccines, health systems strengthening and civil society organization support (GAVI; http://www.gavialliance.org/support/what/index.php). ISS is provided in two phases: an investment phase (years one and two) and the reward phase (from year three onwards). During the latter, countries receive 20 USD per additional infant who has received DTP3 (third dose of DTP) as compared to baseline figures (GAVI; http://www.gavialliance.org/support/what/iss/index.php). However, this reward is contingent to providing evidence that data reported by countries are reliable, as assessed with Data Quality Audits (DQA).

The aim of any initiative to improve immunization is to increase coverage up to a level where all children are protected against the targeted diseases. However, it has been increasingly recognized that good quality information for decision-making is essential to increase coverage (Bchir et al. 2006; Papania & Rodewald 2006). Surveillance, monitoring and evaluation are integral components of successful immunization systems (WHO/UNICEF 2005).

The DQA is a survey methodology developed by WHO which estimates the robustness of immunization reporting systems. The main outcome of DQAs is the estimation of the Verification Factor (VF), which expresses the accuracy of the reporting system by estimating the proportion of DTP3 immunizations that can be traced through the reporting system, from the vaccine delivery points up to the national coverage estimates. GAVI partners agreed that countries with a VF of 80% or more would ‘pass’ and receive the reward while countries that ‘fail’ are required to produce their own plans to improve the reporting system and are encouraged to conduct a second DQA 2 years later (The LATH Consortium 2001).

A previous paper (Ronveaux et al. 2005) reviewed the methodological issues of DQA and focused on the aggregated outcomes of the DQAs conducted up to 2003. In this report, we present individual countries’ performance of the DQA carried out in 41 countries up to 2005, and explore patterns of performance among countries. We also show the changes in the reporting system in those countries that failed the first DQA and undertook a second one. As DQA do not aim at estimating immunization coverage, we will not do any comparison with other methods to estimate immunization coverage.

Methods

The DQA is a standard methodology developed by the WHO; it is carried out by independent companies after an open tender process. Two external consultants travelled to each country and engaged with two staff members of the national immunization programmes to conduct the DQAs over a period of 2 weeks.

In each country, a multistage sampling procedure was followed: first, four districts were randomly selected with probability proportional to the reported doses of DTP3 administered in the previous year; secondly, in each of the four districts, six Health Units (HU), where immunizations are administered, were randomly selected (total of 24 HUs per country). This weighted representative sampling was designed to fit with what could be reasonably achieved within the resources and timeframe of the DQA. Districts and HUs with unsolvable access problems which make them non-eligible were excluded from the sampling process. Reasons for exclusion were mainly security situations or major geographical barriers that could not be overcome within the timeframe of a field visit. DQAs with a proportion of unreachable districts greater than 20% have been excluded from some of the analyses and indicated in the text.

The DQAs have two outcome measures: the VF and the Quality Scores (QS). The period audited was the full calendar year previous to the date when the DQA took place. In each district, the VF is calculated by dividing the number of DTP3 vaccinations administered during the audited year as recounted in the HUs records filled at the very moment when children are vaccinated by the annual DTP3 vaccinations reported in the HUs reports found at the health district offices (the usual next tier in the reporting system). This quotient is adjusted for the weight of the six selected HUs in relation to the whole number of HUs in the district. This is finally extrapolated to the national level as the weighed average of districts VFs. The methods and mathematical expressions have been described in detail elsewhere (WHO 2003; Ronveaux et al. 2005).

A VF less than 100% indicates that the reports at district level showed more DTP3 administrations than those that could be recounted at HU level (‘over-reporting’); a VF over 100% suggests that not all DTP3 doses recounted could be traced in the reports at district level (‘under-reporting’).

The QSs were based on a series of questions and observations undertaken at each level of the immunization programme: national, district and HU. They covered topics such as recording and reporting of immunization data, keeping of vaccine ledgers and information system design. Each question correctly answered was assigned one point. An average QS ranging from 0 to 5 was obtained for the national level, for each one of the four districts and for each one of the 24 HUs (some questions for each level of the system differed).

Finally, auditors provided feed-back to immunization staff at all levels and suggested recommendations addressing the most relevant issues identified.

Results

Forty-seven DQAs were conducted between 2002 and 2005 in 41 countries: 30 African, 10 Asian and one Caribbean. Twenty-one countries failed the DQA (VF less than 80%), and six of those conducted a second DQA (total 47 DQAs).

The proportion of non-eligible districts for sampling was higher than 20% in nine of the 41 DQAs: 55% in Yemen, 45% in Nepal, 43% in Myanmar, 41% in Congo, 34% in Afghanistan, 32% in Lesotho, 32% in Sudan’s second DQA, 25% in DR Congo and 22% in Mali. A total of 1082 HUs were surveyed in 188 districts in the 47 DQAs. Table 1 summarizes the country profiles and DQA framework.

The VF (data accuracy) was below the threshold value (80%) in 46% of the DQAs (median of the VF 83%, inter-quartile range (IQR) 23%) (Figure 1). Excluding those DQAs with high proportion of unreachable districts: 50% had a VF below 80% and the median of the VF was 80% (IQR 33%). In Nigeria it was not possible to estimate the VF due to lack of data. Two DQAs showed VFs above 100%, indicating under-reporting (the deviation from 100% was marginal: 100.2% and 106.4%).

VF 95% confidence intervals (CI) were wide, especially in countries with low VFs, reflecting the great variability of the DTP3 recounted-reported quotient among districts. DQAs with VFs above 95% showed very narrow CIs, suggesting homogeneity in the VFs among districts. There was a significant correlation between VFs and the widths of its CIs (rho = −0.679, P < 0.001).

Table 2 shows a selection of the questions to assess the quality of the immunization reporting system in each tier, with the percentage of countries, districts and HUs that correctly answered them, excluding DQAs with a high proportion of unreachable districts.

In theory, immunization reporting mechanisms can be integrated within the national health management information systems or can be set apart as a parallel vertical reporting system only for immunization. DQAs showed that reporting of immunization data from the HU to the district level was integrated in 61% of the DQAs; and from the district to the national level in 55% of DQAs.

Computers to manage immunization data were used in all national immunization programme offices and in 41% of district offices. In almost three quarters of the DQAs, immunization data was used to provide feed-back from the national to the district immunization offices; slightly more than half of districts provided feed-back to the HUs under their catchment area. Immunization data was also compiled in some type of publication in 82% of national immunization offices and 58% of districts. However, immunization monitoring charts or tables could only be seen in less than half of the national immunization offices, 59% of districts and in a smaller proportion of HUs. DTP1-3 drop-out rates were monitored in a lesser proportion at all three levels. A relatively high proportion of HUs had some immunization reports or primary recording forms available, but only two-thirds had a complete set of reports from the previous year.

The use of consistent denominators is essential to obtain accurate immunization coverage figures. Almost all national immunization programmes used different figures in different years, reflecting the change in population size. However, this was not the case in districts with 87% using the same figures in different years. In 82% of the DQAs, denominators complied with the WHO recommended definition. In only 14% of the DQAs it was found that districts were using consistent denominators to those assigned by the national immunization programmes to each district within a country.

The presence of guidelines for different immunization related procedures was variable. At district level, guidelines seemed to be more available than at national level.

Vaccine ledgers to manage vaccine stocks could be found in the majority of national immunization programmes, in district offices and HUs holding vaccine stocks; however, a smaller proportion were updated. Vaccine wastage calculations could be confirmed in almost one-third of national programmes and district offices and in two-thirds of HUs.

The answers to these questions were used to estimate QS for each level of the immunization reporting system. The median QS at national level was 3.3 out of 5.0 (inter-quartile range 0.7), 3.3 out of 5.0 (inter-quartile range 1.1) in the 152 districts and 3.1 out of 5.0 (inter-quartile range 1.6) in the 912 HUs across all districts and countries.

Correlation analyses

Figure 2 is a scatter chart depicting one ‘bubble’ per DQA, with the X and Y axis showing the aggregated HU and district QSs respectively. The size of the ‘bubbles’ is proportional to the QS measured at national immunization headquarters.

There was a significant correlation between QSs measured at HU and at district levels (rho = 0.865, P < 0.001). Larger ‘bubbles’ tended to be found towards the upper right corner of the chart suggesting a significant correlation of national QSs with district and HUs scores (rho = 0.525, P < 0.001 and rho = 0.4843, P = 0.002 respectively).

Figure 2 identifies countries with consistent poor or good performances. Central African Republic, Haiti, Lao, Madagascar, Mauritania, Mozambique and Nigeria are in the lower left corner with small size bubbles: showing poor QS at all levels. At the far right upper end, Tanzania, Burkina Faso, Guinea (second DQA) and Kenya (second DQA) show the highest scores (DQAs with a high proportion of unreachable districts, excluded).

We also explored to what extent there could be examples of good quality districts (good district QS) in the poorest performing countries (poor national QS). QSs at national level significantly correlated with those of the best performing district in each country (rho = 0.408, P = 0.004). Looking at pairs of national-QS and best district QS in that country data, there were several cases of outstanding performance at district level in countries with very poor national QSs (Ethiopia, Tajikistan and Yemen) and also cases of consistent poor national QSs with even the best districts also poorly performing (Central African Republic, Haiti, Madagascar, Mauritania and Nigeria).

The VF did not show any significant correlation with national QS (rho = 0.211, P = 0.202). On the contrary, there were significant correlations with districts and HUs QS (rho = 0.703, P < 0.001 and rho = 0.726, P < 0.001 respectively) (DQAs with a high proportion of unreachable districts, excluded).

Discussion

Conclusion

DQA is a systematic methodology to describe in depth data quality issues and to provide recommendations to address them. DQAs can reveal a number of crucial problems that affect the quality of immunization data and provide countries with an opportunity to identify the weakest parts in the collection, transmission and use of information. Basic recording and reporting practices at the periphery of the system, alongside design aspects (e.g. denominators), have been identified as key factors that need to be tackled. DQAs also provide insights from all tiers of the health system, identifying good practices in some HUs and districts even in countries poorly performing as a whole. Those HUs and districts can become drivers to improve reporting mechanisms in the countries. DQAs have been adapted into a self-assessment tool (WHO 2004b) and can be simplified to assess specific aspects of the information system. In whatever form, DQAs bring data quality issues to the top front of the agenda to improve the monitoring of immunization coverage. Furthermore, the DQA methodology could be considered to address data quality issues across the spectrum of national disease control programmes (The Global Fund 2007) so data quality remains a priority to help improve planning and service delivery based on accurate coverage estimates.