Skip to main content

Performances of statistical methods for the detection of seasonal influenza epidemics using a consensus-based gold standard

  • C. SOUTY (a1), R. JREICH (a1), Y. LE STRAT (a2), C. PELAT (a2), P. Y. BOËLLE (a1) (a3), C. GUERRISI (a1), S. MASSE (a1) (a4), T. BLANCHON (a1), T. HANSLIK (a1) (a5) (a6) and C. TURBELIN (a1)...

Influenza epidemics are monitored using influenza-like illness (ILI) data reported by health-care professionals. Timely detection of the onset of epidemics is often performed by applying a statistical method on weekly ILI incidence estimates with a large range of methods used worldwide. However, performance evaluation and comparison of these algorithms is hindered by: (1) the absence of a gold standard regarding influenza epidemic periods and (2) the absence of consensual evaluation criteria. As of now, performance evaluations metrics are based only on sensitivity, specificity and timeliness of detection, since definitions are not clear for time-repeated measurements such as weekly epidemic detection. We aimed to evaluate several epidemic detection methods by comparing their alerts to a gold standard determined by international expert consensus. We introduced new performance metrics that meet important objective of influenza surveillance in temperate countries: to detect accurately the start of the single epidemic period each year. Evaluations are presented using ILI incidence in France between 1995 and 2011. We found that the two performance metrics defined allowed discrimination between epidemic detection methods. In the context of performance detection evaluation, other metrics used commonly than the standard could better achieve the needs of real-time influenza surveillance.

Corresponding author
*Author for correspondence: C. Souty, IPLESP UMRS 1136 INSERM UPMC, Faculté de médecine Pierre et Marie Curie, Paris 6, 27 rue Chaligny, 75571 Paris Cedex 12, France. (Email:
Hide All
1.Influenza ( Accessed 4 October 2016.
2. Thompson, WW, et al. Mortality associated with influenza and respiratory syncytial virus in the United States. Journal of the American Medical Association 2003; 289: 179186.
3. Monto, AS. Epidemiology of influenza. Vaccine 2008; 26(Suppl. 4): D45D48.
4. Ortiz, JR, et al. Strategy to enhance influenza surveillance worldwide. Emerging Infectious Diseases 2009; 15: 12711278.
5. Carrat, F, et al. Evaluation of clinical case definitions of influenza: detailed investigation of patients during the 1995–1996 epidemic in France. Clinical Infectious Diseases 1999; 28: 283290.
6. Cowling, BJ, et al. Methods for monitoring influenza surveillance data. International Journal of Epidemiology 2006; 35: 13141321.
7. Serfling, RE. Methods for current statistical analysis of excess pneumonia-influenza deaths. Public Health Reports 1963; 78: 494506.
8. Wang, X, et al. Using an adjusted Serfling regression model to improve the early warning at the arrival of peak timing of influenza in Beijing. PLoS ONE 2015; 10: e0119923.
9. Le Strat, Y, Carrat, F. Monitoring epidemiologic surveillance data using hidden Markov models. Statistics in Medicine 1999; 18: 34633478.
10. Vega, T, et al. Influenza surveillance in Europe: establishing epidemic thresholds by the moving epidemic method. Influenza and Other Respiratory Viruses 2013; 7: 546558.
11. Unkel, S, et al. Statistical methods for the prospective detection of infectious disease outbreaks: a review. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2012; 175: 4982.
12. Closas, P, Coma, E, Méndez, L. Sequential detection of influenza epidemics by the Kolmogorov–Smirnov test. BMC Medical Informatics and Decision Making 2012; 12: 112.
13. Choi, BY, et al. Comparison of various statistical methods for detecting disease outbreaks. Computational Statistics 2010; 25: 603617.
14. Kleinman, KP, Abrams, AM. Assessing surveillance using sensitivity, specificity and timeliness. Statistical Methods in Medical Research 2006; 15: 445464.
15. Debin, M, et al. Determination of French influenza outbreaks periods between 1985 and 2011 through a web-based Delphi method. BMC Medical Informatics and Decision Making 2013; 13: 138.
16. Flahault, A, et al. Virtual surveillance of communicable diseases: a 20-year experience in France. Statistical Methods in Medical Research 2006; 15: 413421.
17. Sentinelles Network Database ( Accessed 4 September 2017.
18. Turbelin, C, et al. Age distribution of influenza like illness cases during post-pandemic A(H3N2): comparison with the twelve previous seasons, in France. PLoS ONE 2013; 8: e65919.
19. Souty, C, et al. Improving disease incidence estimates in primary care surveillance systems. Population Health Metrics 2014; 12: 19.
20. Pelat, C, et al. Online detection and quantification of epidemics. BMC Medical Informatics and Decision Making 2007; 7: 29.
21. Costagliola, D, et al. A routine tool for detection and assessment of epidemics of influenza-like syndromes in France. American Journal of Public Health 1991; 81: 9799.
22. Huber, PJ. Robust estimation of a location parameter. Annals of Mathematical Statistics 1964; 35: 73101.
23. Fox, J. An R and S-Plus Companion to Applied Regression. CA: Sage, 2002.
24. Viboud, C, et al. Influenza epidemics in the United States, France, and Australia, 1972–1997. Emerging Infectious Diseases 2004; 10: 3239.
25. Costagliola, D. When is the epidemic warning cut-off point exceeded? European Journal of Epidemiology 1994; 10: 475476.
26. Tsui, F-C, et al. Value of ICD-9-Coded chief complaints for detection of epidemics. Journal of the American Medical Informatics Association 2002; 9: s41s47.
27. Martinez-Beneito, MA, et al. Bayesian Markov switching models for the early detection of influenza epidemics. Statistics in Medicine 2008; 27: 44554468.
28. Spreco, A, Timpka, T. Algorithms for detecting and predicting influenza outbreaks: metanarrative review of prospective evaluations. BMJ Open 2016; 6: e010683.
29. Fleming, DM, Cross, KW. Respiratory syncytial virus or influenza? Lancet (London) 1993; 342: 15071510.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Epidemiology & Infection
  • ISSN: 0950-2688
  • EISSN: 1469-4409
  • URL: /core/journals/epidemiology-and-infection
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed