Skip to main content Accessibility help
Hostname: page-component-559fc8cf4f-sbc4w Total loading time: 0.935 Render date: 2021-02-25T11:23:50.694Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false, "newCitedByModal": true }

Rapid assessment of influenza vaccine effectiveness: analysis of an internet-based cohort

Published online by Cambridge University Press:  12 September 2011

London School of Hygiene and Tropical Medicine, London, UK
London School of Hygiene and Tropical Medicine, London, UK
Institute for Scientific Interchange (ISI), Torino, Italy
Institute for Scientific Interchange (ISI), Torino, Italy
Institute for Scientific Interchange (ISI), Torino, Italy
London School of Hygiene and Tropical Medicine, London, UK
E-mail address:
Rights & Permissions[Opens in a new window]


The effectiveness of influenza vaccination programmes is seldom known during an epidemic. We developed an internet-based system to record influenza-like symptoms and response to infection in a participating cohort. Using self-reports of influenza-like symptoms and of influenza vaccine history and uptake, we estimated vaccine effectiveness (VE) without the need for individuals to seek healthcare. We found that vaccination with the 2010 seasonal influenza vaccine was significantly protective against influenza-like illness (ILI) during the 2010–2011 influenza season (VE 52%, 95% CI 27–68). VE for individuals who received both the 2010 seasonal and 2009 pandemic influenza vaccines was 59% (95% CI 27–77), slightly higher than VE for those vaccinated in 2010 alone (VE 46%, 95% CI 9–68). Vaccinated individuals with ILI reported taking less time off work than unvaccinated individuals with ILI (3·4 days vs. 5·3 days, P<0·001).

Original Papers
Copyright © Cambridge University Press 2011


Many countries routinely vaccinate against influenza. However, as vaccines must be developed before the influenza season, mismatches between the circulating and vaccine strains can occur. It is therefore seldom clear during the season how effective the vaccine is at reducing the risk of death, serious illness, influenza-like illness (ILI), laboratory-confirmed disease, absenteeism, or other adverse events.

Measuring vaccine effectiveness (VE) can be challenging because of the many potential confounders and biases. The ‘gold-standard’ approach involves following large matched cohorts of vaccinated and unvaccinated individuals over the course of a season and directly measuring the protective effect of the vaccine. In practice, such cohort studies are rarely used as they are both time-consuming and expensive. Instead, a range of alternative techniques has been developed to estimate VE more rapidly and with fewer resources [Reference Farrington1Reference Skowronski4].

Commonly reported techniques include the screening method and the case-control method [Reference Farrington1, Reference Valenciano2]. The screening method compares vaccination levels in influenza cases that are seen by the health services with estimated vaccination levels in the population as a whole, whereas the case-control method compares vaccination levels in ILI cases that are laboratory confirmed as influenza with ILI cases that test negative for influenza. These methods were used to estimate the efficacy of the H1N1v (2009) vaccine during the 2009–2010 season to be 70% in England and Scotland [Reference Hardelid5], and around 90% in Germany [Reference Wichmann6] and Canada [Reference Skowronski7], and have been applied elsewhere in previous influenza seasons [Reference Kelly3, Reference Skowronski4, Reference Savulescu8]. The first estimates of the efficacy of the 2010 seasonal influenza vaccine, obtained using the case-control method, were recently published [Reference Pebody9], suggesting an efficacy of around 63% in people vaccinated in both the 2009–2010 and 2010–2011 seasons.

Although widely used, these methods suffer from the fact that they require cases to present themselves for detection, generally to general practitioners (GPs), and to be identified as ILI cases and tested for influenza. It is highly unlikely that individuals with ILI who are tested for influenza by their GP are a random sample of individuals with ILI [Reference Brooks-Pollock10]. In particular, such efficacy estimates depend on assumptions about the relative rates of treatment-seeking and case-detection in vaccinated and unvaccinated individuals. There are a number of possible ways in which these assumptions may not be met: incidence of influenza-negative ILI may differ between vaccinated and unvaccinated groups; there may be differences in the presentation of symptoms between vaccinated and unvaccinated individuals, leading to differences in the probability of cases being identified for testing by GPs; there may be differences in viral shedding levels or duration of infectiousness leading to differences in test sensitivity. These reasons provide motivation for seeking additional methods of estimating VE that do not rely on individuals presenting themselves for treatment.

Here, we present a novel method for monitoring VE in real-time using an internet-based cohort [Reference Tilston11, Reference Friesma12]. In contrast to other methods, internet-based surveillance does not rely on individuals seeking medical attention [Reference Brooks-Pollock10]. During a normal influenza season, internet surveillance can complement traditional methods by providing rapid preliminary estimates of VE and may prove vital during a pandemic when traditional surveillance methods are not practical [Reference Tilston11, Reference Friesma12].


The internet-based UK flusurvey ( [Reference Tilston11] was launched for the 2010–2011 season on 1 December 2010. Participation was voluntary, and members of the public were encouraged to participate through outreach activities and the media. Participants completed an intake questionnaire on registration, recording their age, gender, risk group status (taking regular medication for diabetes, asthma, other lung, heart or kidney disorders, being immunocompromised or pregnant), whether and when they were vaccinated, public transport use, and contact with groups of patients, the elderly, and children. Each week, participants were prompted by email to complete a survey documenting any symptoms experienced, onset date, use of health services, and whether the episode resulted in absenteeism. Further details can be found in Tilston et al. [Reference Tilston11]. Participants were able to update their reported vaccination status throughout the season, and were periodically reminded of this in the weekly email.

We analysed all reports collected up to 31 January 2011 from all participants who had completed the symptom questionnaire at least once. The study period, 1 December 2010 to 31 January 2011, includes the time during which GP-reported levels of ILI peaked and were above baseline levels (Fig. 1).

Fig. 1. Vaccine uptake in flusurvey participants. Cumulative 2010 seasonal influenza vaccine uptake among flusurvey participants across three key groups: participants aged ⩾65 years (red line, circles); participants aged <65 years in risk groups (blue line, squares) and healthcare workers (purple line, diamonds). Bars represent 95% confidence intervals. Also shown (grey line) is ILI incidence (weekly cases per 100 000), as reported by GP-based surveillance [18]. Date is shown on the horizontal axis. The launch date of flusurvey for the 2010–2011 season is indicated with a cross (×).

Episodes of illness reported through flusurvey were classified as ILI if the recorded symptoms matched the Health Protection Agency (HPA) ILI case definition of a fever plus two or more of: cough, sore throat, runny nose, joint pain, headache, vomiting or diarrhoea. Episodes of acute respiratory illness that did not match the ILI case definition but included symptoms of sneezing, cough, runny/blocked nose or sore throat were diagnosed as milder respiratory illness (MRI).

An unadjusted estimate of VE was estimated from the ratio of attack rates in vaccinated and unvaccinated participants as 1 – RR, where RR is the relative risk of reporting illness in the vaccinated and unvaccinated groups. As this crude estimate of VE is not adjusted for any covariates, such as age or belonging to a risk group, multivariate regression models were used to model the probability of ever reporting ILI and, for those individuals who did not report ILI, ever reporting MRI during the study period. Age group, membership of a risk group, contact patterns and vaccine status were used as predictor variables. For the purposes of this study, participants were considered as vaccinated during the 2010–2011 season if they reported having received a vaccine at any stage during the season. Because of the timing of the study, the majority of those vaccinated had received their vaccine before the study began (Fig. 1).

VE is defined as the relative reduction in risk between the vaccinated and unvaccinated groups. Because ILI is relatively common, Poisson regression with robust standard errors was used to estimate directly the RR of reporting ILI given each of the predictor variables [Reference McNutt13] (the common approach of using the odds ratio (OR) to approximate RR is only appropriate when measuring rare events [Reference McNutt13]). Estimates generated using log-binomial regression [Reference McNutt13, Reference Zou14] and adjusting ORs using the method of Zhang & Yu [Reference Zhang and Yu15] gave very similar results (not shown). Analyses were conducted using R version 2.10 [Reference Ihaka and Gentleman16] and Stata v. 11 [17].


In all, 653 individuals registered for the flusurvey, of whom six were excluded from the analysis because of unknown vaccine status. The symptom questionnaire was completed a total of 3102 times, with a median of five reports per participant.

Table 1 shows the characteristics of the 647 individuals included in the analysis. Sixty-two percent of participants were female and, as might be expected from an internet-based survey, children and people aged ⩾65 years were underrepresented, with 78% of participants aged between 25 and 64 years. Influenza-like-illness (ILI) was reported at least once by 185 (27%) of the sample; MRI was reported on at least one occasion by 390 (60%) individuals, and by 309 (67%) of those who did not report ILI. Thirty-four percent (220) reported receiving an H1N1v-containing vaccine either in 2009–2010 (as the ‘swine flu’ vaccine) or in 2010–2011 (as the seasonal flu vaccine). Vaccine uptake during the 2010–2011 influenza season was 70% (95% CI 57–82) in participants aged ⩾65 years, with the majority vaccinated before November 2010 (Fig. 1). Participants aged <65 years with an underlying health condition reported a slower and slightly lower vaccine uptake of 60% (95% CI 49–70). Thirty percent (95% CI 18–47) of healthcare workers (defined as those who reported having regular contact with patients) reported receiving the 2010–2011 vaccine, with an increase in uptake coinciding with the rise in cases in December 2010 [18].

Table 1. Multivariate regression analyses of factors related to risk of acquiring ILI or MRI

ILI, Influenza-like-illness; MRI, mild respiratory illness; RR, relative risk; CI, confidence interval.

ILI defined as fever plus two or more of runny or blocked nose, sore throat, cough, headache, muscle pain, vomiting, diarrhoea.

MRI defined as ‘not ILI’ plus at least one of runny or blocked nose, sore throat, cough, sneezing. Participants reporting ILI were excluded from this analysis, so the relevant denominator is given by (N-NILI).

§ Defined as taking regular medication for diabetes, asthma, other lung, heart or kidney disorders, being immunocompromised or pregnant.

Contact with more than 10 during the course of a typical day.

* P<0·05, ** P<0·01, +P<0·1.

Participants aged >25 years were significantly less likely (P<0·05) to report ILI than younger participants (Table 1). Participants receiving the H1N1v vaccine either in 2010–2011 alone or both in 2010–2011 and 2009–2010 were significantly less likely (P<0·01) to report ILI than unvaccinated participants. Other factors were not statistically significant, although there was a trend for an increased risk of ILI in individuals who were in a risk group, or had regular contact with groups of children. In those individuals who did not report ILI, the risk of MRI was not associated with any factor except vaccine status: those reporting H1N1v vaccination in 2010–2011 and 2009–2010 were significantly less likely (P=0·029) to report MRI.

ILI was reported by 14% (26/181) of individuals vaccinated against H1N1v influenza in the 2010–2011 season and 34% (159/466) of unvaccinated individuals, giving a crude estimate of VE of 58% (95% CI 39–71). Adjusting for other factors, and considering vaccination in 2010–2011 only, VE against ILI was estimated to be 52% (95% CI 27–68). Additionally considering vaccination in 2009–2010, VE against ILI was estimated to be 46% (95% CI 9–68) for individuals who received the 2010 seasonal vaccine only and 59% (95% CI 27–77) for individuals who received both the 2010 seasonal vaccine and the 2009 pandemic vaccine. In those individuals who did not report ILI, VE against MRI was estimated to be 25% (95% CI 3–42) for those receiving H1N1v-containing vaccines in both the 2009–2010 and 2010–2011 seasons. There was no significant protection against illness during the 2010–2011 season afforded to individuals who just received the 2009 pandemic vaccine, although the number of such participants was small.

Vaccination was associated with reduced absenteeism among participants. In participants aged between 25 and 64 years, 4·1% of vaccinated participants and 11·6% of unvaccinated participants reported taking time off work due to symptoms (P<0·001). Of the participants who took time off work, vaccinated participants took an average of 3·4 days off and unvaccinated participants took 5·3 days off (P<0·001).

The healthcare-seeking behaviour of participants with symptoms did not vary by vaccine status after accounting for age and risk groups (RR 0·99, P>0·5).


The analysis presented here demonstrates the effectiveness of H1N1v vaccination at reducing reported ILI and at reducing absenteeism. We found VE at reducing ILI to be 59% in those receiving H1N1v vaccination in both 2009–2010 and 2010–2011. The results are broadly consistent with a randomized control trial that found that vaccination protected against ILI, physician visits, and laboratory-confirmed influenza in healthy adults when the vaccine and circulating viruses were well matched [Reference Bridges19] and with estimates of influenza vaccine efficacy made using data from previous seasons [Reference Skowronski4Reference Savulescu8]. Our results also agree with the first published study looking at the effectiveness of the 2010 seasonal vaccine in the UK [Reference Pebody9], which reports an effectiveness of 46% for individuals vaccinated in 2010–2011 only and 63% in individuals vaccinated in both the 2009–2010 and 2010–2011 seasons, and with studies conducted elsewhere in Europe [Reference Kissling20, Reference Savulescu21] (we note that both our estimates and those in other studies [Reference Pebody9, Reference Kissling20, Reference Savulescu21] have fairly wide confidence intervals).

The finding that individuals vaccinated with both the 2009 pandemic influenza vaccine and the 2010 seasonal influenza vaccine may have had slightly elevated effectiveness against ILI and were protected against milder illness is consistent with results recently published elsewhere [Reference Pebody9], and suggests that receipt of the pandemic vaccine in 2009–2010 may have primed individuals leading to a better response to the seasonal vaccine in that year. Further studies are needed to confirm or refute this.

There are several limitations to the approach taken here, many of which are discussed elsewhere [Reference Tilston11]. The participants were a self-selecting sample with potential biases, and in particular the youngest and oldest age groups were underrepresented. ILI was determined using a symptom-based case definition, and therefore influenza was not laboratory confirmed; however, ILI as determined here has been shown to provide a good match to patterns of influenza incidence [Reference Brooks-Pollock10Reference Friesma12] and the difference in estimated effectiveness of vaccination against ILI compared to effectiveness against MRI suggests that ILI is capturing true influenza infections. In those individuals who did not report ILI, H1N1v vaccination offered some protection against MRI, suggesting that some MRI cases were actually mild doses of influenza. The approach here relies on self-reported symptoms and vaccination status, which, owing to the anonymous nature of the survey, cannot be verified. Furthermore, although participants were reminded to keep their vaccination status updated, we may not have recorded all vaccinations that took place after the study began; however, because the study started before the majority of ILI cases but after the majority of vaccinations had taken place in the UK (Fig. 1) [18], it is unlikely that many vaccinations went unrecorded. In the analysis presented here we have treated individuals as vaccinated if they received an H1N1v influenza vaccine at any point during the 2010–2011 season, hence ‘non-effectiveness’ of vaccination could be the result of infection taking place before vaccination; however, because of the timing of the 2010–2011 UK influenza season (incidence peaking in late December/early January) [18] almost all vaccinated individuals had received their vaccination before being exposed to influenza, and in our study there were only two individuals who reported ILI prior to their date of vaccination.

There are advantages to our approach. First, the internet-based system collects ILI incidence data almost in real time, and thus allows us to make extremely rapid estimates of VE; second, the approach follows vaccinated and unvaccinated individuals through the influenza season rather than relying on people with ILI accessing healthcare services, and is therefore free from potential biases introduced by GP-based estimates of vaccine efficacy; third, our community-based approach is inexpensive and can be scaled up to include more participants at no extra cost. Thus, while our study is relatively small, involving only about 650 participants, it is hoped that substantially larger sample sizes will be obtained in future years as the UK flusurvey becomes more established.

In common with methods that rely on people visiting their GP, there is no guarantee that our sample is representative of the population. However, among the participants in this study the level of vaccine uptake across various groups closely matches provisional data from the HPA (according to provisional data contained in the HPA weekly national influenza report [18], 72·8% of people in England aged ⩾65 years had received the 2010 seasonal influenza vaccine vs. 70% in our sample; in people aged <65 years in a risk group the comparable figures are 50·3% and 60%, respectively; and in healthcare workers provisional data show that 34·2% were vaccinated by the end of January 2011 vs. 30% in our sample). Thus, although there may well be other biases in our cohort, it resembles the UK population in terms of vaccination behaviour. Furthermore, there is no reason to expect that vaccinated and unvaccinated participants who exhibit flu-like symptoms would complete the survey in different ways. That traditional methods of estimating vaccine efficacy and the method presented here are obliged to make use of potentially biased samples is an unfortunate reality; we believe that the method described here, although in need of further testing over future seasons, is a valuable addition to existing techniques.

Internet-based monitoring of influenza in the community allows real-time surveillance of ILI, rapid assessment of VE, and an understanding of the wider impact of illness, which can be used to inform healthcare providers and the public of the utility of influenza vaccination in both resource-rich and resource-poor settings.


The authors thank Steve Whitbread at LSHTM for help in setting up the flusurvey website, the Epiwork Work Package 5 team, and Helen Green (HPA) and Sara Thomas (LSHTM) for useful discussions. We also thank all flusurvey participants. The study was funded by Epiwork as part of the EU Seventh Framework Programme (Project Reference: 231807).


The partner of W.J.E. is employed by GlaxoSmithKline.


1.Farrington, CP. Estimation of vaccine effectiveness using the screening method. International Journal of Epidemiology 1993; 22: 742746.CrossRefGoogle Scholar
2.Valenciano, M, et al. Study designs for timely estimation of influenza vaccine effectiveness using European sentinel practitioner networks. Vaccine 2010; 28: 73817388.CrossRefGoogle Scholar
3.Kelly, H, et al. Estimation of influenza vaccine effectiveness from routine surveillance data. PLoS One 2009; 4: e5079.CrossRefGoogle ScholarPubMed
4.Skowronski, DM, et al. Estimating vaccine effectiveness against laboratory-confirmed influenza using a sentinel physician network: results from the 2005–2006 season of dual A and B vaccine mismatch in Canada. Vaccine 2007; 25: 28422851.CrossRefGoogle Scholar
5.Hardelid, P, et al. Effectiveness of pandemic and seasonal influenza vaccine in preventing pandemic influenza A(H1N1)2009 infection in England and Scotland 2009–2010. Eurosurveillance 2011; 16: pii=19763.Google ScholarPubMed
6.Wichmann, O, et al. Pandemic influenza A(H1N1) 2009 breakthrough infections and estimates of vaccine effectiveness in Germany 2009–2010. Eurosurveillance 2010; 15: pii=19561.Google ScholarPubMed
7.Skowronski, DM, et al. Effectiveness of AS03 adjuvanted pandemic H1N1 vaccine: case-control evaluation based on sentinel surveillance system in Canada, autumn 2009. British Medical Journal 2011; 342: c7297.CrossRefGoogle Scholar
8.Savulescu, C, et al. Estimating the influenza vaccine effectiveness in elderly on a yearly basis using the Spanish influenza surveillance network – pilot case-control studies using different control groups, 2008–2009 season, Spain. Vaccine 2010; 28: 29032907.CrossRefGoogle ScholarPubMed
9.Pebody, R, et al. Effectiveness of seasonal 2010/2011 and pandemic influenza A(H1N1)2009 vaccines in preventing influenza infection in the United Kingdom: mid-season analysis 2010/11. Eurosurveillance 2011; 16: pii=19791.Google Scholar
10.Brooks-Pollock, E, et al. Using an online survey of healthcare-seeking behaviour to estimate the magnitude and severity of the 2009 H1N1v influenza epidemic in England. BMC Infectious Diseases 2011; 11: 68.CrossRefGoogle ScholarPubMed
11.Tilston, N, et al. Internet-based surveillance of Influenza-like-illness in the UK during the 2009 H1N1 influenza pandemic. BMC Public Health 2010; 10: 650.CrossRefGoogle ScholarPubMed
12.Friesma, I, et al. Internet-based monitoring of influenza-like illness in the general population: experience of five influenza seasons in the Netherlands. Vaccine 2009; 27: 63536357.CrossRefGoogle Scholar
13.McNutt, LA, et al. Estimating the relative risk in cohort studies and clinical trials of common outcomes. American Journal of Epidemiology 2003; 157: 940943CrossRefGoogle ScholarPubMed
14.Zou, G. A modified Poisson regression approach to prospective studies with binary data. American Journal of Epidemiology 2004; 159: 702706.CrossRefGoogle ScholarPubMed
15.Zhang, J, Yu, KF. What's the relative risk? A method of correcting the odds ratio in cohort studies of common outcomes. Journal of the American Medical Association 1998; 280: 16901691.CrossRefGoogle ScholarPubMed
16.Ihaka, R, Gentleman, R. R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics 1996; 5: 299314.Google Scholar
17.StataCorp. Stata statistical software: release 11. College Station, TX: StataCorp LP, 2009.Google Scholar
18.HPA. HPA weekly national influenza report ( Accessed 8 March 2011.Google Scholar
19.Bridges, CB, et al. Effectiveness and cost-benefit of influenza vaccination of healthy working adults. Journal of the American Medical Association 2000; 284: 16551663.CrossRefGoogle ScholarPubMed
20.Kissling, E, et al. Early estimates of seasonal influenza vaccine effectiveness in Europe, 2010/11: I-MOVE, a multicentre case-control study. Eurosurveillance 2011; 16: pii=19818.Google ScholarPubMed
21.Savulescu, C, et al. Effectiveness of the 2010/11 seasonal trivalent influenza vaccine in Spain: preliminary results of a case-control study. Eurosurveillance 2011; 16: pii=19820.Google Scholar

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 72
Total number of PDF views: 92 *
View data table for this chart

* Views captured on Cambridge Core between September 2016 - 25th February 2021. This data will be updated every 24 hours.


Send article to Kindle

To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Rapid assessment of influenza vaccine effectiveness: analysis of an internet-based cohort
Available formats

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Rapid assessment of influenza vaccine effectiveness: analysis of an internet-based cohort
Available formats

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Rapid assessment of influenza vaccine effectiveness: analysis of an internet-based cohort
Available formats

Reply to: Submit a response

Your details

Conflicting interests

Do you have any conflicting interests? *