INTRODUCTION
Estimation of influenza vaccination effectiveness (VE) is challenging for the following reasons: (a) Predominant influenza virus types, subtypes and phenotypes change from one season to the next, necessitating a new vaccine targeting different strains in most seasons. As a result, VE has to be re-estimated in every season. (b) Influenza vaccination is now recommended for every person aged >6 months in the USA and many other countries have broad recommendations, making randomized, placebo-controlled clinical trials unethical. Observational studies therefore often become the only option. (c) Confounding and bias are often present in these observational VE studies. (d) It is not easy to find all or most influenza patients in a given community, as symptoms are usually not severe and many patients do not seek medical care to alleviate them. (e) Symptoms of influenza are non-specific; hence many patients who develop an acute respiratory illness (ARI) are not infected with an influenza virus. (f) Special laboratory tests are required to confirm influenza infection, and these tests are not 100% sensitive and specific, causing misclassification bias. Vaccination status may also be misclassified. For all these reasons, observational studies to estimate influenza VE have to be designed very carefully to avoid, or at least to minimize, the various sources of bias.
In this article we evaluate and compare two commonly used case-control study designs for estimating VE against seasonal or pandemic influenza illness. In both study designs, individuals who report to a clinic, or to a member of a network of clinics, because of an ARI and test positive for an influenza virus are considered cases. In the (ordinary) case-control design (CCD), a control is an asymptomatic person randomly selected from the source population when a case is identified. In the test-negative design (TND), ARI patients who test negative for an influenza virus serve as controls. The TND [Reference Skowronski1, Reference Orenstein2] is relatively new and has become very popular because (a) it is more convenient and (b) it accounts for bias resulting from differences in the propensity of seeking medical care. However, the accuracy of influenza VE estimates resulting from this study design has not been evaluated while accounting for all potential sources of bias. In addition, we are not aware of any study comparing these two case-control designs side by side.
Below we present a summary of the main sources of bias in influenza VE estimates from case-control studies.
(a) Ascertainment of cases (selection bias). A person who develops an ARI may or may not seek medical care. In both CCD and TND studies, only persons seeking medical care for ARI can be tested and be considered cases. This subset of cases who seek care for ARI may not be a representative sample of all cases.
(b) Confounding by propensity of seeking medical care. The likelihood of seeking medical care may be related to a person's vaccination status, as vaccinated individuals may be more health conscious so that their probability of seeking care for ARI may be different from that of unvaccinated persons. In CCD studies, only persons seeking medical care for ARI can be considered cases, while controls are selected from the entire population. This may confound the association between vaccination status and being considered a case and result in underestimation of VE. This source of confounding bias is avoided in TND studies, as both cases and controls are persons seeking care for ARI.
(c) Probabilities of non-influenza ARI may depend on vaccination status. In TND studies, individuals with non-influenza ARI serve as controls. Therefore, the TND may produce biased estimates of VE unless vaccinees and non-vaccinees are equally likely to develop non-influenza ARI. The validity of this assumption has not yet been confirmed. On one hand, De Serres et al. [Reference De Serres3] used data from randomized clinical trials (RCTs) to argue that this assumption is usually satisfied. On the other hand, a recent randomized influenza vaccine trial [Reference Cowling4] found that vaccinees had a significantly increased risk of virologically confirmed non-influenza infection (that may lead to ARI) compared to those who received placebo.
(d) Other confounders. Confounders such as health status, age, exposure, education, socioeconomic status, may be associated with both the likelihood of being vaccinated and the likelihood of becoming infected, developing ARI and seeking medical care.
(e) Misclassification bias. As already mentioned, even the best diagnostic tests for influenza viruses are not 100% sensitive and specific. Vaccination status may also be misclassified.
The goal of this article is to evaluate and compare the bias and precision of VE estimates resulting from TND and CCD studies when the outcome of interest is symptomatic influenza. Specifically, we will (a) evaluate the bias of each of the VE estimates by comparing the expected value of the estimate with the true VE, and (b) evaluate the standard errors of the VE estimates as functions of the total sample size. To conduct these evaluations and comparisons we developed a detailed stepwise probability model of the process involved in collecting data in these studies and obtaining VE estimates. The model will allow us to derive both general and numerical results under different scenarios.
METHODS
We first describe the real-life process involved in conducting the two types of studies and obtaining the estimates of VE. We then describe the model we develop to mimic this process.
The study population
The source population for both types of case-control studies consists of all individuals receiving most of their medical care at a single clinic or at a specific network of clinics. Since influenza VE varies by age, we can assume that the model pertains to a subpopulation corresponding to a single age group.
The study designs
When a member of the study population develops an ARI, s/he may decide to report to a clinic for treatment. At the clinic, the healthcare provider may ask the person to be tested for influenza viruses. If the person agrees then a swab is taken and sent to a laboratory for testing. In both study designs, a person who tests positive is eligible to be considered a case. In a TND study, an individual who tests negative is eligible to be considered a control. In a traditional CCD study, controls are individuals who have not developed ARI prior to their inclusion in the study. Usually, one or more controls are selected immediately after a case is identified. In both study designs, the vaccination status of every case or control was determined from manual or electronic records.
Outcome of interest and true VE
In this work we evaluate estimates of VE against symptomatic influenza, sometimes also called ‘influenza illness’. A person is considered a true case of symptomatic influenza if s/he has ARI and is infected by an influenza virus. The true VE is defined as 1 minus the ratio of the probability of this outcome in vaccinees and non-vaccinees.
Estimation of VE and bias of VE estimates
We only consider estimates of VE that are not adjusted for possible confounders. In case-control studies, VE is usually estimated as 1 minus the odds ratio (OR) of being vaccinated in cases vs. controls. The bias of the estimate is defined as the difference between the expectation of the estimated VE and the true VE.
The model
The model we developed for comparing the estimates from the two study designs follows the scheme described above with a few simplifications. We assumed that (a) when a person seeks medical care for ARI then her/his probability of being tested for influenza viruses does not depend on vaccination status or on the actual cause of ARI (influenza/non-influenza), (b) given a person's symptoms and influenza infection status, the sensitivity and specificity of the test do not depend on the tested person's vaccination status or on the probability that s/he seeks medical care for ARI, (c) a person's vaccination status is determined without error, and (d) controls in a CCD study are selected at random from all asymptomatic individuals who receive their medical care at the facilities enrolling cases.
Our model consists of four steps, where the value of a single variable is determined at each step. The distribution of this variable may depend on the values of the variables from the previous steps. Below we define the four steps, the associated variables and the probabilities determining each variable's distribution.
Step 1: Vaccination. A person may be vaccinated against influenza. We define a binary variable V, where V = 1 for a vaccinated person, and denote α = P(V = 1).
Step 2: Infection and ARI. During the influenza season, a person may become infected with an influenza virus. Both influenza infected and uninfected individuals may develop an ARI. Since our outcome of interest only involves symptomatic individuals, we ignore the influenza infection status of asymptomatic persons. We therefore define a variable E for the illness/infection status with three categories as follows: E = 0 indicating no ARI, E = 1 for ARI without influenza infection (i.e. an ARI resulting from a different pathogen), and E = 2 for ARI and influenza infection (symptomatic influenza). Since the distribution of E depends on the vaccination status, V, we denote β v = P(E = 1|V = v), γ v = P(E = 2|V = v) for v = 0, 1 with β v + γ v ⩽ 1. Here we assume the ‘leaky vaccine’ model [Reference Haber, Longini and Halloran5], where a vaccinee has a lower probability of becoming infected than a non-vaccinee. We also developed an alternative model assuming that the vaccine has an ‘all-or-none’ effect [Reference Haber, Longini and Halloran5], i.e. some of the vaccinees are completely protected against infection while the vaccine does not reduce the susceptibility of the remaining vaccinees.
Step 3: Seeking medical care for ARI. A person with ARI may seek medical care and in this case s/he is tested for influenza viruses. We define a binary variable M, with M = 1 for a person seeking medical care for her/his ARI. The probability of this event depends on E (only individuals with an ARI seek medical care), and it may be different for ARI patients with and without an influenza infection. In addition, the conditional distribution of M given E may depend on V to allow confounding due to the fact that a vaccinated person may be more or less likely to seek medical care compared to an unvaccinated person. We therefore define
$$\delta_{ev} = P(M = 1 \vert E = e,\,\,V = v)\,\quad for \; \,e = 1,\,\,2\;\; \,{\rm and}\;\; \,v = 0,\,1,$$
(note that P(M = 1|E = 0) = 0.)
Step 4: Testing for influenza viruses. Although only individuals who seek medical care for ARI are tested for influenza viruses, it will be convenient to define a binary variable T as the (possibly unobserved) test result for any person with an ARI, regardless of whether or not s/he is actually tested. Therefore we define T = 1 (T = 0) if a person would test positive (negative) for influenza if tested. Because of assumption (b) above, the probability of testing positive given the person's influenza infection status does not depend on V and M. We therefore denote τ e = P(T = 1|E = e) for e = 1, 2. Note that τ 1 is 1 minus the test's specificity and τ 2 is the test's sensitivity in persons with ARI.
Our model has a total of 11 parameters (Table 1), which specify the conditional distribution of each variable in terms of the values of the variables determined in the previous steps. The true VE against symptomatic influenza is VE T = 1 − RR T , where
$$RR_T = P(E = 2 \vert V = 1)/P(E = 2 \vert V = 0) = \gamma_1/\gamma_0. $$
Notation used in this paper
ARI, Acute respiratory illness; TND, test-negative design; CCD, case-control design; w.r.t., with respect to.
Estimates of VE in our model
As stated earlier, the estimate of VE from a case-control study is 1 − OR in the C × V table corresponding to the individuals included in the study, where C is a binary indicator of case/control status with C = 1 for a case. For convenience, the TND and CCD studies will be represented by the letters A and B, respectively.
In a TND study, the case/control variable is denoted C A , where {C A = 1} = {M = 1, T = 1} and {C A = 0} = {M = 1, T = 0}. Then the estimate of VE is: VE A = 1 − OR A , where
$$\eqalign{&{\rm OR}_A \cr & =\displaystyle{{P(C_A = 1,\,\,V = 1 \vert M = 1) \cdot P(C_A = 0,\,\,V = 0 \vert M = 1)} \over {P(C_A = 1,\,\,V = 0 \vert M = 1) \cdot P(C_A = 0,\,\,V = 1 \vert M = 1)}},}$$
Note that all the probabilities are conditional upon M = 1 as only individuals who seek medical care for ARI can be included in the TND study.
In a CCD study, the case/control variable is denoted C B . Cases are defined in the same way as in the TND study, i.e. {C B = 1} = {M = 1, T = 1} = {C A = 1}. Controls are individuals included in a random sample drawn from all the asymptomatic individuals. In other words {C B = 0} is a random subset of {E = 0}. In addition we define a binary variable B indicating whether or not a person is included in the CCD study, i.e. {B = 1} = {C B = 1 or C B = 0}. The VE estimate is based on the odds ratio in the C B × V table when all the probabilities are conditional upon B = 1: VE B = 1 − OR B , where
$$\eqalign{&{\rm OR}_B \cr &= \displaystyle{{P(C_B = 1,\,\,V = 1 \vert B = 1) \cdot P(C_B = 0,\,\,V = 0 \vert B = 1)} \over {P(C_B = 1,\,\,V = 0 \vert B = 1) \cdot P(C_B = 0,\,\,V = 1 \vert B = 1)}}},$$
Note that in a real-life study, the odds ratios are estimated from the relative frequencies of the corresponding events, rather than from their (unknown) probabilities. Therefore the model-based estimates of VE defined above are actually the expected values of the observed estimates. For convenience we will continue to refer to them as ‘the VE estimates’. As stated earlier, the bias of an estimate is the difference between the expected value of the estimate and the true VE. In Supplementary Appendix 1 we derive general expressions for the bias of the VE estimates from each study designs in terms of the model's parameters.
Standard errors of the estimates
In Supplementary Appendix 3 we use approximations based on the ‘Delta method’ to the standard errors of odds ratios [Reference Agresti6] to derive expressions for the standard errors of both VE estimates in terms of the parameters and the corresponding sample size(s). For evaluating the standard errors we consider the observed odds ratios, where the probabilities are replaced by the observed relative frequencies.
Determining the values of the parameters
We distinguish between biological and non-biological parameters. The biological parameters are the probabilities of non-influenza and influenza ARIs in non-vaccinees and vaccinees, i.e. β 0, β 1, γ 0, γ 1. We used data from RCTs from a recent review paper [Reference Osterholm7] and other sources. We found five publications where the numbers of vaccinated and unvaccinated RCT participants who developed ARI with and without influenza infection could be determined. In all these RCTs, influenza infection was confirmed via culture or RT–PCR. From these publications we identified a total of 14 comparisons of an active influenza vaccine and a placebo in a specific influenza season, as some of the publications included RCT data from more than one season or RCTs with more than one active vaccine. For each of the comparisons we obtained estimates of the four biological parameters from the numbers of influenza and non-influenza cases of ARI in vaccinees and non-vaccinees. A list of these comparisons and the corresponding observed frequencies and estimates of the biological parameters is presented in Supplementary Appendix 2.
Regarding the non-biological parameters, the proportion of vaccinees (α) does not affect the bias of any of the VE estimates; however it affects their standard errors. According to the most recent Centers for Disease Control and Prevention (CDC) publication [8], influenza vaccine coverage in the USA in the 2011–2012 season ranged between 30% and 70%. The probability of seeking medical care for ARI has been estimated to be between 0·20 and 0·50 [Reference Ferdinands and Shay9]. We used 0·30 as the baseline value of this probability for unvaccinated non-influenza ARI cases. We then allowed the probability of seeking medical care to be higher or lower for vaccinated cases and for influenza-infected ARI cases. The sensitivity and specificity of the test for influenza viruses were assumed to range from 95% to 100% (J. M. Ferdinands, unpublished data). A list of all the model's parameters, their values and other notation is provided in Table 1.
RESULTS
The results under the leaky vaccine model and the all-or-none model were identical. We now introduce additional notations that will be helpful for the presentation of the results (see Table 1 for a full list of the notations used in this paper). First, we define a few probability ratios comparing vaccinees and non-vaccinees: ρ β = β 1/β 0, ρ γ = γ 1/γ 0, ρ δ 1 = δ 11/δ 10, ρ δ 2 = δ 21/δ 20. In addition we denote the probability of not having ARI by η v = 1 − (β v + γ v ) = P(E = 0|V = v), v = 0, 1, and define ρ η = η 1/η 0. Finally, we define the cross-product ratio θ δ = δ 10δ21/δ 11 δ 20 = ρ δ 2 /ρ δ 1 .
Next, we introduce three assumptions that will simplify the interpretation of both the algebraic and the numerical results:
Assumption 1 (A1). The influenza test has perfect sensitivity and specificity, i.e. τ 1 = 0, τ 2 = 1.
Assumption 2 (A2). The probability of non-influenza ARI is independent of vaccination status, i.e. β v = P(E = 1|V = v) does not depend on v, or β 0 = β 1. As we stated in the Introduction, this assumption is essential for the validity of VE estimates from TND studies as persons with non-influenza ARI serve as controls in these studies.
Assumption 3 (A3). The vaccine-related relative increases or decreases in the probability of seeking medical care for ARI are the same for ARI patients with and without influenza infection, i.e. δ 11/δ 10 = δ 21/δ 20, which is equivalent to ρ δ1 = ρ δ2 . While this assumption allows the probability of seeking medical care for ARI to depend on vaccination status and type of infection (influenza or non-influenza), the ratio of these probabilities between vaccinees and non-vaccinees does not depend on the type of infection leading to ARI. We will refer to this assumption as ‘homogeneity of the probability ratios’ of seeking medical care for ARI.
Table 2 presents algebraic expressions for the bias of the VE estimates from the two study designs under three combinations of the above assumptions. These expressions for the bias can be easily derived from the general expressions given in Supplementary Appendix 1. From the results in Table 2 we learn that the VE estimate from a TND study is unbiased when all three assumptions are satisfied. Note that in this case, the probability of seeking medical care may depend on vaccination status as long as the homogeneity assumption holds. On the other hand, in order for the VE estimate from a CCD study to be unbiased one must make the additional assumption that the vaccine does not affect the likelihood of developing ARI (ρ η = 1). The assumption is unlikely to hold as long as the vaccine protects against influenza infection which is usually associated with an increased risk of ARI.
Bias of vaccine effectiveness estimates under various assumptions*
* See text for definitions of the assumptions.
Numerical assessments of the bias of VE estimates
Numerical values of the bias of VE estimates based on TND and CCD studies are presented in Tables 3–6 for all the 14 comparisons of RCT participants who received an influenza vaccine or a placebo (see Methods section and Supplementary Appendix 2). The bias is defined as the difference between the estimated and true VE. For example, if the true VE is 0·6 (60%) and the estimated VE is 0·68 (68%) then the bias is 0·08.
In Table 3 we consider the scenario where all three assumption (A1–A3) are met, i.e. perfect sensitivity and specificity, β 1 = β 0 and ρ δ 1 = ρ δ 2 (homogeneity of ratios of probabilities of seeking medical care for ARI), and we consider the bias for different values of the common value ρ δ of ρ δ 1 and ρ δ 2 (ρ δ is the ratio, comparing vaccinees and non-vaccinees with respect to the probability of seeking medical care for ARI; under A3 this ratio does not depend on whether the ARI resulted from an influenza or a non-influenza infection). As expected from our general considerations above, the TND-based estimate is always unbiased when the three assumptions are satisfied. The CCD-based estimate has a positive (negative) bias when vaccinees are less (more) likely than non-vaccinees to seek medical care for ARI. Since one would expect vaccinees to be more health conscious than non-vaccinees, they may also be more likely to seek care for ARI (i.e. ρ δ > 1), hence the CCD-based estimate is likely to underestimate the true VE.
Bias of vaccine effectiveness (VE) estimates under assumptions A1, A2, A3 for various values of ρδ *
* We assume that the diagnostic test has perfect sensitivity and specificity, the probability of non-influenza acute respiratory illness (ARI) does not depend on vaccination status and the ratio, comparing vaccinees and non-vaccinees with respect to the probability of seeking medical care for an ARI does not depend on whether the ARI resulted from an influenza or a non-influenza infection. This ratio is denoted ρ δ. VE A and VE B are the VE estimates from test-negative design and case-control design studies, respectively. The table's rows correspond to the comparisons of vaccinated and unvaccinated randomized clinical trial participants (see Table A1 in the Supplementary Appendix).
In Table 4 we still assume perfect sensitivity and specificity and β 1 = β 0 but we omit the homogeneity assumption ρ δ 1 = ρ δ 2 . Thus, we explore the impact of the deviation from the homogeneity assumption (A3). As we mentioned earlier, the ρ δ values measure the excess ‘risk’ of seeking medical care for ARI in vaccinees vs. non-vaccinees. Hence θ δ = ρ δ 2 /ρ δ 1 compares these excess risks when ARI results from an influenza or a non-influenza infection. The bias of the TND-based estimate is positive (negative) when θ δ is less than (greater than) 1. Regarding the CCD-based estimate, the algebraic value of the bias decreases as θ δ increases.
Bias of vaccine effectiveness (VE) estimates under assumptions A1, A2 for various values of θδ *
* We assume that the diagnostic test has perfect sensitivity and specificity and the probability of non-influenza acute respiratory illness (ARI) does not depend on vaccination status. θδ = ρδ2 /ρδ1 where ρ δ 1 is the ratio, comparing vaccinees and non-vaccinees with respect to the probability of seeking medical care for an ARI resulting from a non-influenza infection, and ρ δ 2 is similarly defined for an ARI resulting from an influenza infection. VE A and VE B are the VE estimates from test-negative design and case-control design studies, respectively. The table's rows correspond to the comparisons of vaccinated and unvaccinated randomized clinical trial participants (see Table A1 in the Supplementary Appendix).
In Table 5 we examine the effect of departure from the assumption (A2) that the probability of developing a non-influenza ARI is independent of vaccination status, i.e. β 1 = β 0. We still assume perfect sensitivity and specificity of the influenza test and homogeneity of the probabilities of seeking medical care for ARI. Comparing VE estimate based on a TND study across the three values of ρ β = β 1/β 0, we observe that the algebraic value of the bias decreases as ρ β increases and that the bias is positive when ρ β > 1. The absolute bias of a TND study-based VE estimate due to unequal probabilities of non-influenza ARI may be quite substantial, especially when ρ β < 1. The effect of departure of ρ β from 1 on the bias of VE estimates from CCD studies is much smaller than the effect on TND study-based estimates. Departure of ρ β from 1 may be a result of viral interference.
Bias of vaccine effectiveness (VE) estimates under assumptions A1, A3 for various values of ρβ *
* ρβ = β1/β0, the ratio of the probabilities of a non-influenza acute respiratory illness (ARI) in vaccinees and non-vaccinees. We assume that the diagnostic test has perfect sensitivity and specificity and that the vaccination-related ratios of probabilities of seeking medical care for ARI are homogeneous, i.e. ρ δ 1 = ρ δ 2 . VE A and VE B are the VE estimates from test-negative design and case-control design studies, respectively. The table's rows correspond to the comparisons of vaccinated and unvaccinated randomized clinical trial participants (see Table A1 in the Supplementary Appendix).
In Table 6 we examine the effects of lack of 100% sensitivity and specificity of the influenza test. We still assume that the probability of a non-influenza ARI does not depend on vaccination status (β 1 = β 0) and that the ratios of the probabilities of seeking medical care for ARI are homogeneous. We observe that misclassification of the test results indeed decreases the algebraic value of the bias. Reducing the test's specificity from 1·00 to 0·95 has a much more pronounced effect on the bias than a similar reduction in the test's sensitivity, thus confirming the results of Orenstein et al. [Reference Orenstein2].
Bias of vaccine effectiveness (VE) estimates under assumptions A2, A3 for various values of the sensitivity and specificity of the diagnostic test*
Se, Sensitivity; Sp, specificity.
* We assume that the probability of a non-influenza acute respiratory illness (ARI) does not depend on vaccination status and that the ratios of probabilities of seeking medical care for ARI are homogeneous, i.e. ρ δ 1 = ρ δ 2 . VE A and VE B are the VE estimates from test-negative design and case-control design studies, respectively. The table's rows correspond to the comparisons of vaccinated and unvaccinated randomized clinical trial participants (see Table A1 in the Supplementary Appendix).
Standard errors of VE estimates
As we can see from the results in Supplementary Appendix Table A2, the standard errors of VE estimates from TND and CCD studies are usually quite similar, with no clear rule for predicting which study design provides more precise estimates.
DISCUSSION
We developed a model describing the process that generates data for an observational study aimed at estimating VE against symptomatic influenza. The process involves four steps: vaccination, developing infection and illness, seeking medical care for ARI and testing for influenza viruses. The bias and standard error of VE estimates based on ordinary case-control studies and on test-negative studies can be written in terms of the model's parameters. Therefore this model facilitates the evaluation and comparison of the two study designs in terms of their accuracy and precision.
Several models and methods for evaluating the bias of influenza VE estimates from TND studies have been proposed in the past [Reference Orenstein2, Reference De Serres3, Reference Ferdinands and Shay9–Reference Jackson and Nelson11]. The current approach has the following advantages compared to the previous publications: (a) it accounts for more sources of bias than any of the earlier approaches (e.g. the recent paper by De Serres et al. [Reference De Serres3] evaluates the bias of TND-based VE estimates but it does not account for bias related to different health-seeking behaviours of vaccinated and unvaccinated individuals), (b) our model can be used to assess the bias of VE estimates from both TND and CCD studies, and (c) it allows the evaluation and comparison of standard errors of the estimates.
We found that the TND study-based VE estimate is unbiased under the following conditions: (A1) the diagnostic test has perfect sensitivity and specificity, (A2) the probability of non-influenza ARI does not depend on vaccination status, and (A3) the ratio, comparing vaccinees and non-vaccinees, of the probabilities of seeking medical care is the same for influenza and non-influenza ARI patients. The bias of the CCD study-based estimates is very small if these three assumptions hold. When assumptions A2 and A3 hold, but assumption A3 is violated then it may be difficult to compare the biases of the estimates as the comparison depends on the odds ratio θ δ which is usually unknown (Table 4). When assumption A2 is violated, i.e. the probability of non-influenza ARI depends on vaccination status, then TND-based estimates may be severely biased. In this case, the bias of VE estimates from CCD studies is less affected by the possible inequality of the probabilities of non-influenza ARI, compared to the bias estimated from TND studies (Table 5).
In this work we considered the bias of VE estimates without adjusting for any covariates. Both estimates are based on odds ratios and can be adjusted for known covariates. As we have seen, a very important potential confounder is the propensity of seeking medical care for influenza and non-influenza ARI. Most influenza VE studies do not collect the information that would allow adjusting for this confounding effect.
In order to assess the bias in a real-life influenza VE study one has to estimate the parameters underlying the various sources of bias. Accurate estimates of the biological parameters can only be obtained from carefully designed randomized studies, which are usually expensive and unethical. On the other hand, behavioural parameters, such as probabilities of seeking medical care for ARI, can be obtained from observational studies. As suggested by our results, a high correlation between vaccination status and the propensity of seeking medical care (e.g. older persons are more likely to be vaccinated and to seek medical care) may result in substantial bias. Estimation of this correlation should not be too difficult.
Our study has some limitations: (a) We assumed that every person who seeks medical care for ARI has the same probability of being tested for influenza viruses, regardless of vaccination status. (b) We assumed that the test's sensitivity and specificity does not depend on vaccination status or on the propensity of seeking medical care. (c) We assumed that vaccination status is determined without an error. (d) We considered ‘symptomatic influenza’ as the only outcome of interest as we believe that this is the most important outcome from a public health perspective. Using different outcomes, such as ‘influenza infection’ or ‘medically attended influenza’ would affect the results of our study (Q. An, Ph.D. dissertation). (e) Our model does not account for the infection transmission process generating cases of influenza and non-influenza ARI. (f) All the parameters in our model remain unchanged throughout the influenza season. We could eliminate the first three limitations by including additional parameters in the model, but it would be very difficult to determine the values or reasonable ranges for these parameters. In addition, including more parameters in the model would make interpretation of results more difficult. Addressing limitations (e) and (f) would involve assumptions about the contact and the transmission processes and about temporal trends in the values of the parameters. The transmission dynamics could have an important effect on our results, especially in the ‘leaky vaccine’ case, as the ratio of the incidence rates of infection comparing vaccinees and non-vaccinee would vary over time. In the future we plan to use a detailed agent-based stochastic simulation model to evaluate the bias and precision of influenza VE estimates while incorporating these processes and additional real-life factors.
SUPPLEMENTARY MATERIAL
For supplementary material accompanying this paper visit http://dx.doi.org/10.1017/S0950268814002179.
ACKNOWLEDGEMENTS
We thank three anonymous reviewers for their helpful comments. Dr Haber's research was supported by the National Institute of Allergies and Infectious Diseases of the National Institutes of Health (NIH) under Award R01AI110474, and by IPA 1110376-05 with the Centers for Disease Controls and Prevention (CDC). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the CDC.
DECLARATION OF INTEREST
None.
