INTRODUCTION
Vaccination campaigns targeting disease elimination aim to achieve very high levels of coverage, up to 95% in every district. To rapidly assess areas that are not reaching these very high levels of coverage during vaccination campaign activities, rapid house-to-house monitoring (RHHM) has been proposed [Reference Dietz1]. RHHM has the advantages of being practical and fast, but its statistical value may be difficult to interpret since the sample is not randomly selected (it uses a convenience sample approach).
The World Health Organization (WHO) has been using the lot quality assurance sampling (LQAS) method to classify areas of interest (lots) as having reached acceptable or unacceptable levels of vaccination coverage [2]. LQAS uses two coverage thresholds, an upper threshold (UT) and a lower threshold (LT), and a decision value (d) in a sample (N) of the population. The area between the UT and the LT is known as the ‘grey area’ and is the area of statistical uncertainty typical of LQAS designs: if the number of unvaccinated individuals found in N is higher than d, then we reject the lot as not having reached acceptable vaccination coverage (i.e. real coverage in the lot is below UT); if it is equal to or less than d, then we accept it as acceptable coverage (i.e. real coverage in the lot is above LT). The alpha error is the probability of accepting a lot with an unacceptable proportion of unvaccinated individuals, while the beta error is the probability of rejecting a lot with acceptable vaccination coverage [Reference Lemeshow and Taber3]. The larger the grey area, the smaller the sample size needed to assess the lot with equal precision [Reference Pezzoli, Andrews and Ronveaux4].
LQAS classification traditionally relies on simple random sampling (SRS) and small sample sizes (N<20), this is feasible if the lots are small areas (e.g. a city neighbourhood) where individuals can be easily sampled randomly, but it can become very time-consuming if the lots are large [Reference Tawfik, Hoque and Siddiqi5, Reference Corbella and Grima6]. Instead of SRS, cluster sampling can be used in the lots as an alternative method to reduce the number of site visits needed to complete data collection [Reference Pezzoli, Andrews and Ronveaux4]. Previous studies have explored the applications of clustering the LQA sample to assess global acute malnutrition using field data and computer simulations [Reference Deitchler7–Reference Deitchler, Deconinck and Bergeron9]. A similar approach based on computer simulations to assess the precision of LQAS plans divided into smaller clusters (Clustered-LQAS) has also been explored to assess vaccination coverage [Reference Pezzoli, Andrews and Ronveaux4]. Clustered-LQAS has proven to be useful in assessing vaccination coverage at the health district level after completion of immunization activities [Reference Pezzoli10], but has not previously been used to monitor performance before the end of a campaign in order to identify areas that require mop-up activities and ensure achievement of objectives.
Between 4 and 11 May 2009, Cameroon conducted a polio and yellow fever (YF) vaccination campaign. The target population for YF vaccination was 90% of the 7·5 million individuals aged ⩾9 months (except pregnant women) in 62 health districts at risk for YF [Reference Briand11]. The target population for the oral polio vaccination (OPV) was 100% of the 2·9 million children aged between 0 and 59 months in all 173 health districts of Cameroon.
YF is a mosquito-borne viral haemorrhagic fever endemic in the equatorial regions of Africa and South America [12–Reference Barnett14]. Resurgence of YF virus in recent years poses a public health threat due to the potential for devastating urban epidemics [Reference Wiysonge15, Reference Adeleke16]. Cameroon is one of the 12 African countries supported by the Yellow Fever Initiative led by the WHO and the United Nations Children's Fund (UNICEF) with funding from the Global Alliance for Vaccines and Immunization (GAVI) [Reference Briand11, Reference Wiysonge15, Reference Kuniholm17]. Poliomyelitis (polio) is a viral disease that affects the nervous system [Reference Dutta18]. Polio is targeted for worldwide eradication but, as of 2010, remains endemic in Afghanistan, India, Pakistan, and Nigeria, with sporadic cases exported to other countries [Reference Dutta18–Reference Singh, Das and Dutta21]. At the time of the vaccination campaign, Cameroon had not reported any polio cases since 2006, although cases occurred in neighbouring Nigeria, Chad, and Central African Republic [22].
The purpose of our study was to use the clustered-LQAS technique for mid-campaign vaccination coverage monitoring, and thereby assess districts in Cameroon at risk of low YF or polio vaccination coverage in order to implement timely measures to improve performance before the end of the campaign.
MATERIALS AND METHODS
Definitions
For the purpose of the clustered-LQAS survey, we defined the health districts in Cameroon as ‘lots’. We defined an individual vaccinated against YF as a person aged ⩾9 months, not pregnant, presenting a YF vaccination card from the current campaign or prior vaccination. We defined an individual vaccinated against polio as a child aged between 0 and 59 months presenting the indelible ink mark on the fingernail at the time of OPV vaccination. In the absence of card or ink mark, we recorded a verbal history of vaccination.
Clustered-LQAS plans
As the YF vaccination coverage target was 90%, we used this level for the UT and 75% as the LT, selecting a decision value of seven in a sample of 50 individuals. Assuming SRS, the alpha value of this plan was 5% and the beta value 24%. For polio we set the UT to 95% and the LT to 85%, with d=3 and N=50. Assuming SRS, the alpha and beta values of the plan used for polio were 5% and 12%, respectively.
In the field, we did not use SRS but we divided each sample (N=50) into five clusters of 10 individuals according to the clustered-LQAS methodology [Reference Pezzoli, Andrews and Ronveaux4]. To investigate the effect of clustering, we recalculated alpha and beta by conducting simulations where each cluster's true coverage was sampled from a binomial distribution with a mean of LT or UT and standard deviations (s.d.) of 0·05 or 0·10 (Table 1) [Reference Pezzoli, Andrews and Ronveaux4].
Sampling plans used to evaluate coverage of the two vaccines, clustered lot quality assurance sampling survey of yellow fever and polio vaccination coverage, Cameroon, May 2009
N, Sample size; d, decision value; LT, lower threshold; UT, upper threshold; OPV, oral polio vaccine; YFV, yellow fever vaccine; s.d., standard deviation.
We calculated the operating characteristic (OC) curves of the sampling plans used to assess YF (Fig. 1) and OPV (Fig. 2) coverage according to SRS and the two 5×10 clustered designs (s.d.=0·05 and s.d.=0·10).
Operating characteristic curves for lot quality assurance sampling rule-rejecting programmes with more than seven defectives in a sample of 50, according to simple random sampling (SRS) and the two 5×10 clustered designs (s.d.=0·05 and s.d.=0·10).
Operating characteristic curves for lot quality assurance sampling rule-rejecting programmes with more than three defectives in a sample of 50, according to simple random sampling (SRS) and the two 5×10 clustered designs (s.d.=0·05 and s.d.=0·10).
Lot selection
As the approach was designed to detect low vaccination coverage during the campaign to allow time for intervention, all 62 target districts were assessed before the campaign based on local data. The resources available did not allow us to assess all 62 target districts, we therefore decided to prioritize the districts for intervention. We selected a subset of districts (17/62) at risk for low coverage, of which 12 were selected before the campaign and five once the campaign had begun.
Based on the indicators used by WHO [23], we defined seven criteria for risk of low coverage:
(1) Routine infant administrative YF vaccination coverage in 2008 with s.d. above (i.e. above 100%) or below the mean coverage in the 62 districts targeted by the campaign, reflecting a possible problem with population estimates in the district, which would make administrative coverage data unreliable.
(2) Routine OPV-3 administrative coverage in 2008 with s.d. above or below the mean coverage in the districts.
(3) A district population estimate with s.d. above the mean population in the districts, reflecting large or densely populated districts that may present unique challenges.
(4) A drop-out rate between the first and third doses of routine infant DTP-HepB vaccine above 10%, reflecting problematic routine immunization services [Reference Vandelaer, Bilous and Nshimirimana24].
(5) The district has a border with a neighbouring country; in this case, cross-border influx of persons seeking vaccination could mask a low coverage in the resident population of the district.
(6) District with a complex situation, such as conflict, hosting refugees or internally displaced persons or other circumstances that could challenge campaign implementation.
(7) Previous campaigns resulted in low immunization coverage in the district; some districts may regularly have difficulties achieving national programme targets for a variety of reasons.
We assigned a point each time a district fulfilled one of the criteria and selected the 12 districts with the highest rank, ensuring that all the target regions of Cameroon and the two main cities Yaoundé and Douala, were represented by at least one district. We selected the remaining five districts from those reporting the lowest preliminary YF administrative coverage on the second day of the campaign (Table 2).
Classification for being at risk of low coverage and yellow fever vaccination coverage in the 17 districts retained for the clustered lot quality assurance sampling survey, Cameroon, May 2009
YFV, Yellow fever vaccination.
* Numbers 1–3 reflect the number of risk criteria for low vaccination coverage, assessed prior to the campaign.
† LC-D2, Lowest administrative YFV coverage at campaign day 2.
Sampling and data collection
Since our objective was to use clustered-LQAS for mid-campaign evaluation, we conducted the survey during the last 4 days of the campaign (days 5–8). For each antigen, we interviewed 50 eligible individuals per lot (district) for vaccination status, divided into five clusters of 10. In each district we randomly selected five health areas. In each health area we randomly selected a locality (village or neighbourhood). Next, we randomly selected the first household in the cluster according to geographic random sampling: we drafted a map of the locality, divided it into smaller sectors according to existing divisions (streets, rivers, etc.), and randomly selected one sector where we chose the most central household to start the survey. In the selected household, we interviewed for YF vaccination status only one eligible individual randomly selected and all eligible individuals for OPV. If the individuals selected were aged <10 years we asked a parent or caregiver to answer the questions on their behalf. We then moved to the right of the household to select the subsequent households following a predetermined step. Based on the experience from previous national household surveys in Cameroon and on the proportion of each target group in the population, the step used was nine households for YF and three for polio in rural areas; 18 for YF and six for polio in urban areas.
Since the campaign was in progress, we ensured that the survey was conducted in the localities already covered by vaccination activities, by instructing surveyors to obtain this information at district level. If a selected locality had not yet been covered, surveyors waited until the last day of the evaluation to cover it with the clustered-LQAS survey. We used two standardized questionnaires for data collection.
We did not stop sampling once the number of unvaccinated individuals exceeded d in the lot (i.e. the lot was rejected), and we always completed the sample (N=50).
Administrative vaccination coverage
Tally sheets were used to record the number of persons vaccinated each day and data were transmitted daily through each level of the health system. Thus administrative coverage figures were available day by day during the campaign.
Planning the clustered-LQAS surveys in the field
The LQAS study team planned the mid-campaign evaluation in collaboration with the campaign coordination committee at central level. Campaign technical supervisors were sent to coordinate the campaign at regional level and were fully aware of the purposes of the LQAS surveys. During the mid-campaign evaluation, the LQAS survey teams were instructed to participate at the daily vaccination campaign meetings at district level and discuss the daily results of LQAS with the health officers. The purpose of these meetings was to interpret all information from the field to guide the final efforts of the campaign. LQAS supervisors also sent telephone text messages on the performance of each health district undergoing evaluation every day and especially during an emergency when the district was rejected (as soon as d was exceeded in the lot). In the rejected lots, the campaign coordination committee recommended immediate mop-up activities and also considered an extension of vaccination activities.
Data analysis
The statistical simulations for calculation of the clustered-LQAS plans were run with Stata [25]. We entered data in Epidata [Reference Bohigas and Lauritsen26] and analysed it with Stata [25]. During the campaign, in order to take corrective action based on objective information, we considered as unvaccinated any individual not presenting proof of vaccination. At a later stage, we also analysed whether accepting verbal information on vaccination status would have affected our decision.
We analysed the impact of the measures taken following the outcomes of the clustered-LQAS on YF administrative coverage. We compared districts with or without intervention with regards to the mean preliminary YF administrative coverage reported at day 5 of the campaign and the final administrative coverage at the end of the campaign in the two groups, using the Wilcoxon/Mann–Whitney test with P⩽0·05 as threshold for statistical significance.
We also calculated the standard error (s.e.) from the mean lot coverage for the proportion vaccinated in the five clusters as a measure of inter-cluster variability and checked if this exceeded the s.d. of 0·05 and 0·1 from the mean lot coverage hypothesized in the statistical simulations. We used the maximum standard errors found in the lots as the measure of inter-cluster variability to recalculate alpha and beta with the simulation program.
RESULTS
Evaluation of OPV coverage
Of the 17 surveyed districts, 14 (82·4%) were rejected for low OPV coverage on day 5 of the vaccination campaign due to exceeding the threshold of three unvaccinated individuals, based on documented vaccination status. Allowing verbal report of vaccination, we would have rejected 11 (64·7%) (Table 3).
Oral polio vaccination coverage findings of clustered lot quality assurance sampling survey, Cameroon, May 2009
OPV, Oral polio vaccine; s.e., standard error; R, rejected; A, accepted.
Evaluation of YF vaccination coverage
Ten districts (58·8%) were rejected for low YF vaccination coverage by the LQAS rule due to exceeding the threshold of seven unvaccinated individuals, based on documented vaccination status. Allowing verbal report of vaccination, we would have rejected seven districts (41·2%) (Table 4).
Yellow fever vaccination coverage findings of clustered lot quality assurance sampling survey, Cameroon, May 2009
YFV, Yellow fever vaccination; s.e., standard error; R, rejected; A, accepted.
Control measures
Measures were taken at district level to increase OPV and YF vaccination coverage before the end of the campaign in the rejected lots and, given the high proportion of rejected districts across Cameroon, a 2-day extension of the campaign was recommended nationally to increase vaccine coverage.
Intervention analysis
The final administrative coverage was 96·3% nationally for OPV and 100·5% for YF. The mean increase in the final YF administrative vaccination coverage levels compared with day 5 was higher in the 10 districts initially rejected [mean increase 51·3%, 95% confidence interval (CI) 27·3–75] than in the seven accepted ones (mean increase 31·5%, 95% CI 16·9–46·1), although the difference was not statistically significant. The initially rejected districts appeared to have a significantly higher (P=0·05) final YF administrative vaccination coverage (mean administrative coverage 106·4%, 95% CI 95·3–117·4) compared to the accepted ones (mean administrative coverage 89·7%, 95% CI 77·9–101·5) (Table 2).
Inter-cluster variability
Among the districts evaluated for YF vaccination coverage, the s.e. of the distribution of the binary variable vaccinated/unvaccinated in the five clusters within the lots ranged between 0 and 0·19. It exceeded 0·1 in 23·5% (4/17) of lots; in 41·2% (7/17) it was between 0·05 and 0·1, and in 35·3% (6/17) it was below 0·05.
Among the districts evaluated for OPV coverage, the s.e. of the distribution of the binary variable vaccinated/unvaccinated in the five clusters within the lots also ranged between 0 and 0·19. In 10/17 (58·8%) lots it exceeded 0·1; in 5/17 (29·4%) it was between 0·05 and 0·1, and in 2/17 (11·8%) it was below 0·05. Setting 0·19 inter-cluster variability in the simulations, gave alpha=17% and beta=26% in the YF plan; while in the OPV plan alpha was 20% and it was not possible to calculate beta since 0·19 variability around the 95% upper threshold did not fit in the simulations.
DISCUSSION
During the YF and polio vaccination campaign in Cameroon, we used clustered-LQAS [Reference Pezzoli, Andrews and Ronveaux4] to assess vaccination coverage in 17 lots (health districts) purposely selected as being at higher risk for inadequate coverage. The proportion of districts rejected as having low coverage was high: 59% for YF vaccination and 82% for OPV, using the vaccination card or fingernail mark as proof of vaccination. Based on these findings a 2-day extension of the campaign was recommended nationally by the Ministry of Health to increase vaccination coverage. The districts initially rejected for low YF vaccination coverage presented a significantly higher final administrative YF vaccination coverage compared to those initially accepted, suggesting that interventions to increase coverage at district level and nationally may have been effective.
We used clustered-LQAS for mid-campaign evaluation; while this approach allows time to implement mop-up activities, undertaking an assessment based on final targets before the end of the time allocated to achieve the target will evidently result in underestimation of coverage. To overcome this limitation, we instructed surveyors to first survey the selected localities already covered by vaccination, especially during the first day of the evaluation. It was, however, difficult in some cases to establish which localities were covered at lower administrative levels while in the field.
Relying on verbal confirmation of vaccination status, we would have rejected fewer districts: 41% for YF vaccination and 65% for OPV. The possible overestimation of vaccination coverage using verbal confirmation is a potential problem during the evaluation of vaccination activities [Reference Suarez, Simpson and Smith27, Reference Bolton28]. We therefore opted to base operational conclusions on reliable vaccination information to increase the likelihood of timely evidence-based intervention. The frequent absence of documented proof of vaccination was surprising since the evaluation was conducted during the campaign when card retention should still be high, suggesting a failure in ensuring adequate supply, delivery and use of vaccination cards and finger-marking during the campaign.
We explored dividing lots into five clusters, rather than using SRS, to increase the feasibility and timeliness of LQAS conducted at district level and recalculated the statistical errors using computer simulations [Reference Pezzoli, Andrews and Ronveaux4]. Although this did not happen in Cameroon, with decision values of three and seven in a 5×10 sample, it may be difficult to justify the decision on the whole lot if d is exceeded only in one cluster, especially in the case of villages selected as clusters completely missed by vaccination. To overcome this limitation, it is important to interpret the clustered-LQAS findings with other data including the administrative coverage and local information.
In a previous experience with this approach, we were reassured of the homogeneity of the territory through calculating inter-cluster correlation (ICC) of field data [Reference Pezzoli10]. In this study, clusters were too few and small to calculate any reliable ICCs. Therefore, we accounted for clustering in each lot by calculating the s.e. of the proportion of vaccinated in the clusters, as a measure of inter-cluster variability, which exceeded 0·1 in 23·5% of the lots assessed for YF coverage and in 58·8% of the lots assessed for OPV coverage. Assuming that the maximum inter-cluster variability in the lots was really s.e.=0·19, this would have produced unreliable decision rules with alpha up to 17% and beta up to 26% (yellow fever plan) and with alpha up to 20% and beta theoretically up to 100% (OPV plan). This confirms that clustered-LQAS is unreliable if the territory under study is very heterogeneous with regard to immunization coverage and that the s.d. from the mean lot coverage is >0·1 in the clusters [Reference Pezzoli, Andrews and Ronveaux4].
The higher proportion of districts rejected for low OPV coverage suggests that the polio campaign was less successful than the YF campaign in reaching the target. However this may also depend on the sampling plan used for OPV. To keep N to a manageable size, allowing also for clustering, we had to be more tolerant towards the risk of error [Reference Myatt and Bennett29]. We gave priority to keeping alpha lower rather than beta, especially in the OPV plan, preferring to be strict in not accepting districts with low coverage (a risk for the population), rather than to reject districts with good coverage (a risk for the healthcare system) [Reference Sandiford30]. Such a study design may have produced a higher proportion of false positives (districts with high coverage rejected by the LQAS rule) rather than false negatives (districts with low coverage accepted by the LQAS rule) [Reference Bilukha and Blanton31]. In addition the polio plan had a shorter grey area (10%) that increased the risk of error considerably and also the sampling strategy used, which recommended sampling all the individuals in the target group in the selected household, rather than selecting one randomly, may have increased clustering and the likelihood of error.
To select lots for assessment, we calculated the s.d. from the mean administrative vaccination coverage in all the districts targeted by the campaign to assess if districts may have been presenting vaccination coverage below or above that expected, thus being at risk of low vaccination coverage. Although this approach is standardized and reproducible, it may be more practical to agree on fixed vaccination coverage thresholds that should always be used in different settings in order to increase the practicality of the risk assessment. The pre-selection of two thirds of districts according to available information allowed for maximizing efficiency in preparing the survey, while reserving the option of deploying teams to additional districts based on early vaccination data also ensured the immediate operational relevance of the exercise.
Although our study suggests that the clustered-LQAS survey may have been effective in improving coverage (especially in rejected districts), it is difficult to demonstrate this statistically given the small sample of 17 districts and the fact that at-risk districts were purposefully selected. It would be useful to check if results of other monitoring tools are consistent with the clustered-LQAS findings and compare the results of the simulation with real field data [Reference Deitchler, Deconinck and Bergeron9]. This was not possible as household and external convenience monitoring were not done in the same districts as the clustered-LQAS survey, and the final vaccination coverage survey was administered at regional not district level [32], therefore we opted to compare the two different plans used (OPV and YF) with each other.
This is the first experience using clustered-LQAS in the field since this new variant of the methodology has been standardized [Reference Pezzoli, Andrews and Ronveaux4]. The use of clustered-LQAS for mid-campaign evaluation was successful in monitoring the progress of the campaign in at-risk districts, allowing authorities to implement timely interventions in the rejected lots. Reasons why a district was rejected may have been: poor micro-planning, failure of social mobilization, or accessibility problems.
Dividing the survey sample in each lot into five small clusters increased operational feasibility, while respecting the scientific rationale underpinning the LQAS methodology. We recommend the use of clustered-LQAS as an operational tool to guide the final efforts of vaccination campaigns. To ensure sufficient precision of the clustered-LQAS plans we recommend keeping the grey area in the plans to at least 15% if users want to divide the sample in five clusters. We recommend clustered-LQAS to evaluate immunization programmes that perform well, where it is unlikely to find villages completely missed by vaccination activities. To improve accuracy of the assessment, the survey should be implemented as close to the end of the campaign as possible, while still allowing time for operational decision-making to implement mop-up activities. We recommend improved planning and monitoring for the logistics and distribution of proof of vaccination (cards or ink mark as appropriate) and inclusion of card retention as a key message for the public during social mobilization. The LQAS component should be included in the planning of the campaign and the tools adapted accordingly (e.g. plan extra days in the tally sheets for administrative vaccine coverage monitoring in case the district is rejected). Further refinement of this method may be necessary to decide on intermediate coverage thresholds lower than the campaign coverage target, based on the experience of the country. Further studies could compare the use of clustered-LQAS with other monitoring tools in the same target population to verify consistency of results [Reference Dietz1, Reference Deitchler, Deconinck and Bergeron9]. The results of the clustered-LQAS conducted during the campaign could also be validated with the results of the post-campaign coverage surveys [Reference Bennett33] if these are planned in the same territory to evaluate the campaign. The performance of clustered-LQAS plans divided in more than five clusters could also be tested in the field.
ACKNOWLEDGEMENTS
The study was co-funded by the WHO Yellow Fever Initiative, under a grant from the GAVI Alliance and the WHO Immunization, Vaccines and Biologicals Department. The authors thank the health workers and enumerators in Cameroon who implemented the survey and made the study possible. We are grateful to Dr William Perea for reviewing the survey protocol and to Dr Fenella Avokey for support.
DECLARATION OF INTEREST
None.
