Hostname: page-component-76d6cb85b7-92wsb Total loading time: 0 Render date: 2026-07-13T20:33:56.663Z Has data issue: false hasContentIssue false
  • English
  • Français

INTERRUPTED TIME SERIES DESIGNS IN HEALTH TECHNOLOGY ASSESSMENT: LESSONS FROM TWO SYSTEMATIC REVIEWS OF BEHAVIOR CHANGE STRATEGIES

Published online by Cambridge University Press:  21 April 2004

Craig R. Ramsay ,
Lloyd Matowe ,
Roberto Grilli ,
Jeremy M. Grimshaw  and
Ruth E. Thomas
Craig R. Ramsay
Affiliation:
University of Aberdeen
Lloyd Matowe
Affiliation:
Kuwait University
Roberto Grilli
Affiliation:
Agenzia Sanitaria Regionale
Jeremy M. Grimshaw
Affiliation:
University of Ottawa
Ruth E. Thomas
Affiliation:
University of Aberdeen
Get access Rights & Permissions [Opens in a new window]

Abstract

Objectives: In an interrupted time series (ITS) design, data are collected at multiple instances over time before and after an intervention to detect whether the intervention has an effect significantly greater than the underlying secular trend. We critically reviewed the methodological quality of ITS designs using studies included in two systematic reviews (a review of mass media interventions and a review of guideline dissemination and implementation strategies).

Methods: Quality criteria were developed, and data were abstracted from each study. If the primary study analyzed the ITS design inappropriately, we reanalyzed the results by using time series regression.

Results: Twenty mass media studies and thirty-eight guideline studies were included. A total of 66% of ITS studies did not rule out the threat that another event could have occurred at the point of intervention. Thirty-three studies were reanalyzed, of which eight had significant preintervention trends. All of the studies were considered “effective” in the original report, but approximately half of the reanalyzed studies showed no statistically significant differences.

Conclusions: We demonstrated that ITS designs are often analyzed inappropriately, underpowered, and poorly reported in implementation research. We have illustrated a framework for appraising ITS designs, and more widespread adoption of this framework would strengthen reviews that use ITS designs.

Keywords

Professional practiceCross-sectional studiesRegression analysisTime factors

Information

Type
GENERAL ESSAYS
Information
International Journal of Technology Assessment in Health Care , Volume 19 , Issue 4 , December 2003 , pp. 613 - 623
Copyright
© 2004 Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Avorn J, Soumerai SB, Taylor W et al. 1988 Reduction of incorrect antibiotic dosing through a structured educational order form. Arch Intern Med. 148: 1720- 1724.Google Scholar
Bareford D, Hayling A. 1990 Inappropriate use of laboratory services: Long term combined approach to modify request patterns. BMJ. 301: 1305- 1307.Google Scholar
Barnett GO, Winickoff R, Dorsey JL, Morgan MM, Lurie RS. 1978 Quality assurance through automated monitoring and concurrent feedback usinga computer-based medical information system. Med Care. 16: 962- 970.Google Scholar
Berbatis CG, Maher MJ, Plumridge RJ, Stoelwinder JU, Zubrick SR. 1982 Impact of a drug bulletin on prescribing oral analgesics in a teaching hospital. Am J Hosp Pharm. 39: 98- 100.Google Scholar
Bero L, Freemantle N, Grilli R et al. 1997 Checklist for data abstraction in EPOC reviews. Aberdeen: EPOC(mimeo)
Bero L, Grilli R, Grimshaw JM et al. 1998 Closing the gap between research and practice: An overview of systematic reviews of interventions to promote implementation findings by health care professionals. BMJ. 317: 465- 468.Google Scholar
Bero L, Grilli R, Grimshaw JM et al. 2002 The Cochrane Effective Practice and Organisation of care Group (EPOC) Module. In: The Cochrane Library. Oxford: Update Software Issue 1.
Blohm M, Herlitz J, Hartford M et al. 1992 Consequences of a media campaign focusing on delay in acute myocardial infarction. Am J Cardiol. 69: 411- 413.Google Scholar
Bonerandi JJ, Grob JJ, Cnudde N, Enel P, Gouvernet J. 1992 Campaign of early detection of melanoma in the Provence-Alpes-Cote-d'Azur area 1989. Lessons of an experience. Ann Dermatol Venereol. 119: 105- 109.Google Scholar
Box EP, Jenkins GM. 1976 Time series analysis. London: Holden-Day
Brasca A, Pigliacampo J, Pollastri E et al. 1987 An awareness campaign on the diagnosis and prevention of colon and rectal cancer. Planning and evaluation of its results. Acta Gastroenterol Latinoam. 17: 325- 329.Google Scholar
Brook RH, Williams KN. 1976 Effect of medical care review on the use of injections: A study of the New Mexico Experimental Medical Care Review Organization. Ann Intern Med. 85: 509- 515.Google Scholar
Brooks RJ. 1996 Implementing guidelines for eye care of diabetic patients: Results from an HMO intervention study. Am J Managed Care. 2: 365- 369.Google Scholar
Brufsky JW, Ross-Degnan D, Calabrese D, Gao X, Soumerai SB. 1998 Shifting physician prescribing to a preferred histamine-2-receptor antagonist. Effects of a multifactorial intervention in a mixed-model health maintenance organization. Med. Care 36: 321- 332.Google Scholar
Campbell DT, Stanley JC. 1966 Experimental and quasi-experimental designs for research. Chicago: Rand McNally
Clarke JA, Adams JE. 1990 The application of clinical guidelines for skull radiography in the accident and emergency department: Theory and practice. Clin Radiol. 41: 152-5.Google Scholar
Cochrane AL. 1989 Effectiveness and efficiency: Random reflections on health services. London: British Medical Journal for Nuffield Provincial Health Trust
Cook TD, Campbell DT. 1979 Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin Company
Crosbie 1993 J. Interrupted time-series analysis with brief single-subject data. J Consult Clin Psychol. 61: 966- 974.
Crosbie J. 1995 Interrupted time-series analysis with short series; why it is problematic; how it can be improved. In: Gottman JM, ed. The analysis of change, New Jersey: Lawrence Erlbaum Associates
Del Mar CB, Green AC, Battistutta D. 1997 Do public media campaigns designed to increase skin cancer awareness result in increased skin excision rates? Aust N Z J Public Health. 21: 751- 754.Google Scholar
Dempsey CL. 1995 Nursing home-acquired pneumonia: Outcomes from a clinical process improvement program. Pharmacotherapy. 15: 33S- 38S.Google Scholar
1989 Department of Health and the Welsh Office. General practice in the National Health Service: A new contract. London: HMSO
Domenighetti G, Luraschi P, Casabianca A et al. 1988 Effect of information campaign by the mass media on hysterectomy rates. Lancet. 2: 1470- 1473.Google Scholar
Draper N, Smith H. 1981 Applied regression analysis. New York: Wiley
Elam K, Taylor V, Ciol MA, Franklin GM, Deyo RA. 1997 Impact of a worker's compensation practice guideline on lumbar spine fusion in Washington State. Med Care. 35: 417- 424.Google Scholar
Eppler E, Eisenberg MS, Schaeffer S, Meischke H, Larson MP. 1994 911 and emergency department use for chest pain: Results of a media campaign. Ann Emerg Med. 24: 202- 208.Google Scholar
Everitt DE, Soumerai SB, Avorn J, Klapholz H, Wessels M. 1990 Changing surgical antimicrobial prophylaxis practices through education targeted at senior department leaders. Infect Control Hosp Epidemiol. 11: 578- 583.Google Scholar
Fowkes FG, Evans RC, Williams LA et al. 1984 Implementation of guidelines for the use of skull radiographs in patients with head injuries. Lancet. 2: 795- 796.Google Scholar
Fowkes FG, Hall R, Jones JH et al. 1986 Trial of strategy for reducing the use of laboratory tests. BMJ Clin Res Ed. 292: 883- 885.Google Scholar
Fraser GL, Wennberg DE, Dickens JD Jr, Lambrew CT. 1996 Changing physician behavior in ordering digoxin assays. Ann Pharmacother 30: 449- 454.Google Scholar
Gama R, Nightingale PG, Ratcliffe JG. 1992 Effect of educational feedback on clinicians' requesting of cardiac enzymes. Ann Clin Biochem. 29: 224- 225.Google Scholar
Gortmaker SL, Bickford AF, Mathewson HO, Dumbaugh K Tirrell PC. 1988 A successful experiment to reduce unnecessary laboratory use in a community hospital. Med Care. 26: 631- 642.Google Scholar
Gottman JM. 1981 Time series analysis: A comprehensive introduction for social scientists. Cambridge, England: Cambridge University Press
Grilli R, Ramsay CR, Minozzi S. 2002 Mass media interventions: Effects on health services utilisation. In: The Cochrane Library. Oxford: Update Software Issue 1.
Grimshaw JM, Thomas RE, MacLennan G et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess (in press).
Gurwitz JH, Noonan JP, Soumerai SB. 1992 Reducing the use of H2-receptor antagonists in the long-term-care setting. J Am Geriatr Soc. 40: 359- 364.Google Scholar
Hammond KW, Snowden M, Risse SC, Adkins TG, O'Brien JJ. 1995 An effective computer-based tardive dyskinesia monitoring system. Am J Med Qual. 10: 133- 137.Google Scholar
Harris DW, Karandikar DS, Spencer MG et al. 1979 Returned-medicines campaign in Birmingham, 1977. Lancet. 1: 599- 601.Google Scholar
Healsmith MF, Graham-Brown RA, Osborne JE, London SP, Fletcher A. 1993 Further experience of public education for the early diagnosis of malignant melanoma in Leicestershire. Clin Exp Dermatol. 18: 396- 400.Google Scholar
Herd RM, Cooper EJ, Hunter JA et al. 1995 Cutaneous malignant melanoma. Publicity screening clinics and survival – the Edinburgh experience 1982-90. Br J Dermatol. 132: 563- 570.Google Scholar
Joshi UY, Cameron SO, Sommerville JM, Sommerville RG. 1988 HIV testing in Glasgow Genito-Urinary Medicine Clinics 1985-1987. Scott Med J. 33: 294- 295.Google Scholar
Kong KM, Johnson WF, Anderson PO, Sato LA, Simonian RI. 1987 Effect of a pharmacy-based cost-containment program on cimetidine costs and use. Am J Hosp Pharm. 44: 2068- 2073.Google Scholar
Legorreta AP, Hasan MM, Peters AL, Pelletier KR, Leung KM. 1997 An intervention for enhancing compliance with screening recommendations for diabetic retinopathy. A bicoastal experience. Diabetes Care 20: 520- 523.Google Scholar
Lomas J, Anderson GM, Domnick-Pierre K et al. 1989 Do practice guidelines guide practice? The effect of a consensus statement on the practice of physicians. N Engl J Med. 321: 1306- 1311.Google Scholar
Lowe JB, Balanda KP, Del Mar CB, Purdie D, Hilsdon AM. 1994 General practitioner and patient response during a public education program to encourage skin examinations. Med J Aust. 161: 195- 198.Google Scholar
Macdonald H, Roder D. 1985 The planning, implementation and evaluation of an immunization promotion campaign in South Australia. Hygie. 4: 13- 17.Google Scholar
Maclure M, Dormuth C, Naumann T et al. 1998 Influences of educational interventions and adverse news about calcium-channel blockers on first-line prescribing of antihypertensive drugs to elderly people in British Columbia. Lancet. 352: 943- 948.Google Scholar
Matowe L, Ramsay CR, Grimshaw JM, Gilbert FI, Macleod MJ, Needham G. 2002 Effects of mailed dissemination of the Royal College of Radiologists' guidelines on general practitioner referrals for radiography: A time series analysis. Clin Radiol. 57: 575- 578.Google Scholar
Morrison JC, Sumrall DD, Chevalier SP et al. 1993 The effect of provider education on blood utilization practices. Am J Obstet Gynecol 169: 1240- 1245.Google Scholar
Nattinger AB, Hoffmann RG, Howell-Pelz A, Goodwin JS. 1998 Effect of Nancy Reagan's mastectomy on choice of surgery for breast cancer by US women. JAMA. 279: 762- 766.Google Scholar
Novich M, Gillis L, Tauber AI. 1985 The laboratory test justified. An effective means to reduce routine laboratory testing. Am J Clin Pathol. 84: 756- 759.Google Scholar
Paunio M, Virtanen M, Peltola H et al. 1991 Increase of vaccination coverage by mass media and individual approach: Intensified measles, mumps, and rubella prevention program in Finland. Am J Epidemiol. 133: 1152- 1160.Google Scholar
Pearce MJ, Begg EJ. 1997 Encouraging consensus cost effective drug therapy: Five years experience with a hospital drug utilisation review programme. N Z Med J. 110: 92- 95.Google Scholar
Pehamberger H, Binder M, Knollmayer S, Wolff K. 1993 Immediate effects of a public education campaign on prognostic features of melanoma. J Am Acad Dermatol. 29: 106- 109.Google Scholar
Poma PA. 1998 Effect of departmental policies on cesarean delivery rates: A community hospital experience. Obstet Gynecol 91: 1013- 1018.Google Scholar
Ratnaike S, Hunt D, Eilermann LJ, Hazen R, Deam D. 1993 The investigation of chest pain: Audit and intervention. Med J Aust. 159: 666- 671.Google Scholar
1993 Royal College of Radiologists Working Party. Influence of Royal College of Radiologists' guidelines on referral from general practice. BMJ. 306: 110- 111.
Santerre RE. 1996 The effect of the ACOG guideline on vaginal births after cesarean. Med Care Res Rev. 53: 315- 329.Google Scholar
1989 Secretaries of State for Social Services WNIaS. Working for patients. London: HMSO
Shelley JM, Irwig LM, Simpson JM, Makaskill P. 1991 Evaluation of a mass-media-led campaign to increase Pap smear screening. Health Educ Res. 6: 267- 277.Google Scholar
Sherman CR, Potosky AL, Weis KA, Ferguson JH. 1992 The Consensus Development Program. Detecting changes in medical practice following a consensus conference on the treatment of prostate cancer. Int J Technol Assess Health Care. 8: 683- 693.Google Scholar
Shorr RI, Fought RL, Ray WA. 1994 Changes in antipsychotic drug use in nursing homes during implementation of the OBRA-87 regulations. JAMA. 271: 358- 362.Google Scholar
Soumerai SB, Ross-Degnan D, Kahn JS. 1992 Effects of professional and media warnings about the association between aspirin use in children and Reye's syndrome. Milbank Q. 70: 155- 182.Google Scholar
Soumerai SB, Avorn J, Gortmaker S, Hawley S. 1987 Effect of government and commercial warnings on reducing prescription misuse: The case of propoxyphene. Am J Public Health. 77: 1518- 1523.Google Scholar
Stuart ME, Macuiba J, Heidrich F et al. 1997 Successful implementation of an evidence-based clinical practice guideline: Acute dysuria/urgency in adult women. HMO Pract. 11: 150- 157.Google Scholar
Studnicki J, Remmel R, Campbell R, Werner DC. 1997 The impact of legislatively imposed practice guidelines on cesarean section rates: The Florida experience. Am J Med Qual. 12: 62- 68.Google Scholar
Suwangool P, Moola-Or P, Waiwatana A et al. 1991 Effect of a selective restriction policy on antibiotic expenditure and use: An institutional model. J Med Assoc Thai. 74: 272- 275.Google Scholar
Tesoriero JM, Sorin MD. 1992 The effect of ‘Magic’ Johnson's HIV disclosure on anonymous HIV counselling and testing services in New York State. AIDS Public Policy J. 7: 216- 224.Google Scholar
Thamer M, Ray NF, Henderson SC et al. 1998 Influence of the NIH Consensus Conference on Helicobacter pylori on physician prescribing among a Medicaid population. Med Care. 36: 646- 660.Google Scholar
Turner GC, Mutton KJ. 1987 HIV testing: Changing trends. BMJ. 295: 502.Google Scholar
van Walraven C, Goel V, Chan B. 1998 Effect of population-based interventions on laboratory utilization: A time-series analysis. JAMA. 280: 2028- 2033.Google Scholar
Vincent EC, Hardin PA, Norman LA, Lester EA, Stinton SH. 1995 The effects of a computer-assisted reminder system on patient compliance with recommended health maintenance procedures. Proc Ann Symp Comput Appl Med Care. 656- 660.Google Scholar
Wong ET, McCarron MM. Shaw ST Jr. 1983 Ordering of laboratory tests in a teaching hospital. Can it be improved? JAMA. 249: 3076- 3080.Google Scholar
Zehr KJ, Dawson PB, Yang SC, Heitmiller RF. 1998 Standardized clinical care pathways for major thoracic cases reduce hospital costs. Ann Thorac Surg. 66: 914- 919.Google Scholar