Clinical practice guidelines have become a standard tool in the attempt to increase the quality of clinical care. Surprisingly, whereas the pathway from evidence generation to evidence synthesis and guideline development is highly developed and sophisticated, the pathway from evidence-based guidelines to evidence-based practice is much less developed. Reference Fretheim, Schünemann and Oxman1 Implementation methods range from simple interventions, such as dissemination of educational material, to more complex and multifaceted interventions, including tutorial and consultation sessions, use of treatment algorithms, reminder systems, audit and feedback and use of psychological theories to overcome obstacles. Reference Grimshaw, Thomas, MacLennan, Fraser, Ramsay and Vale2 Whether these implementation strategies are effective in terms of better provider performance and patient outcomes in severe mental illness has been researched in previous systematic reviews. Reference Weinmann, Koesters and Becker3–Reference Girlanda, Fiedler, Ay, Barbui and Koesters5 Although these reviews found insufficient high-quality evidence to draw firm conclusions on the effects of implementation of psychiatric guidelines, they report modest effects towards an improvement of healthcare provider performance and the clinical conditions of the patients. Reference Weinmann, Koesters and Becker3–Reference Girlanda, Fiedler, Ay, Barbui and Koesters5 In spite of these findings, guidelines are used to develop quality indicators for healthcare. These indicators usually measure adherence to guideline recommendation under the assumption that an increased adherence to an evidence-based guideline would subsequently lead to improved patient outcomes. Therefore, there is need for up-to-date evidence on guideline implementation in order to verify this assumption. Furthermore, it is relevant to understand how guidelines should be best implemented into practice. In order to shed light on these issues, the present systematic review updated previous research and summarised the evidence on the effects of guideline implementation on provider performance and patient outcomes.
Method
This review followed an a priori-defined protocol that was published on our institutional website in 2012. Reference Koesters, Girlanda, Ay, Cipriani and Barbui6 The protocol was developed following the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). Reference Liberati, Altman, Tetzlaff, Mulrow, Gøtzsche and Ioannidis7
Study inclusion criteria
Types of studies and participants
This systematic review included randomised controlled trials (RCTs), controlled clinical trials (CCTs) and before-and-after studies. We considered lack of a control condition an exclusion criterion. We included non-RCTs because guideline implementation is complex and often investigated in non-RCTs. Only including RCTs would therefore miss large parts of the evidence available. However, we only conducted the meta-analyses for the RCTs. Study participants were adults of either gender, with a primary diagnosis of schizophrenia or related psychotic disorder, unipolar and bipolar affective disorders and severe depression. As our main interest was the care of patients with severe mental illness, we only included studies with participants recruited and treated in specialist mental healthcare settings. We excluded studies in non-adult populations because of the different treatment process in children and adolescents, which may involve parents.
Types of intervention
We considered any active or passive guideline implementation strategy. For the purposes of this review, we defined guidelines as ‘systematically developed statements (or algorithms, flow charts or tables) prepared to assist decisions about appropriate healthcare for specific clinical circumstances’. Reference Barbui, Girlanda, Ay, Cipriani, Becker and Koesters4 An implementation strategy was defined as ‘any planned process and systematic introduction of guidelines with the aim of giving them a structural place in professional practice’. Reference Barbui, Girlanda, Ay, Cipriani, Becker and Koesters4 We classified the interventions for implementing guidelines according to a taxonomy developed by the Cochrane Effective Practice and Organisation of Care Review Group (EPOC). 8 The following comparisons were included: (a) guidelines implementation strategy v. usual care to examine the impact of guideline implementation on provider performance and patient outcomes; (b) guidelines implementation strategy A v. guidelines implementation strategy B to understand if effects can be augmented by enhanced implementation strategies.
Outcome measures
The primary outcome measure was provider performance, because clinical practice guidelines are developed to improve the performance of healthcare providers in accordance with the best available scientific evidence. The secondary outcome measures were patient outcomes, such as psychopathological symptoms, satisfaction with care, treatment adherence, attitude towards psychiatric medications, and quality of life, as defined by each of the studies.
We hypothesised that guideline implementation may have a positive impact on healthcare provider performance and that this would subsequently influence patient outcomes. Furthermore, we expected that outcomes would differ in different studies according to the characteristics and purposes of the guideline under scrutiny. Only one outcome for provider performance and one patient outcome for each study (where available) was selected for the analyses. The criterion used to select the outcomes was their strength to shed light on the impact of guideline implementation strategies in real clinical practice. Furthermore, we made the selection of the outcomes on the basis of their coherence with the other selected measures. With regard to the performance of healthcare providers we selected outcomes that clearly indicated the level of adherence to the implemented guideline (for example proportion of patients with polypharmacy). Psychopathological symptoms measured by validated scales were considered the primary indicator of the impact of the implementation process on patients (patient outcomes).
Search strategy for identification of studies
We performed a literature search of the following electronic databases: EMBASE, Ovid MEDLINE, PsycINFO, PSYNDEX, Cochrane Database of Systematic Reviews and Cochrane Central Register of Controlled Trials. The electronic search was run using the comprehensive search strategy listed in online supplement DS1 and was last updated in November 2015. Additionally, we screened the references of all included studies for further relevant literature. We did not apply language restrictions.
Data collection
We screened all titles and abstracts, and obtained full texts of potentially relevant papers. Working independently and in duplicate, two reviewers (F.G. and I.F.) read the papers and determined whether they met inclusion criteria. Two reviewers (F.G. and I.F.) carried out data extraction, using a standardised sheet. The information collected included study design, patient characteristics (such as diagnosis), provider characteristics (profession), setting of care, type of intervention (such as purpose of recommendation, guideline development process, implementation strategy) and outcome measures (process and patient outcomes). We solved disagreements by discussion. If no consensus could be reached, the final decision was taken together with a third reviewer (M.K.). For continuous outcomes we extracted the mean score at end-point (for RCTs), the mean change from baseline to end-point (for CCTs) and the mean score at baseline and end-point (for before-and-after studies), along with the standard deviation of these values and the number of participants included in the analysis. For dichotomous data we recorded the number of patients rated as responders and the total number of participants included in the study.
Study quality
Two reviewers (F.G. and I.F.) independently assessed the quality of the included studies using the Downs & Black checklist, an instrument suitable for assessing the methodological quality of randomised trials and non-randomised studies. Reference Downs and Black9 The scale is composed of 27 criteria and assesses five domains: study reporting, external validity, internal validity (bias), internal validity (confounding) and power. In the present review, the Downs & Black checklist was used in a modified version in accordance with previous reviews. Reference Samoocha, Bruinvels, Elbers, Anema, van der Beek and Allard10–Reference Chudyk, Jutai, Petrella and Speechley12 Specifically, scoring for item 27 dealing with statistical power was simplified to a choice of awarding either one point or zero, depending on whether there was sufficient power to detect a clinically important effect. Reference Samoocha, Bruinvels, Elbers, Anema, van der Beek and Allard10,Reference Chudyk, Jutai, Petrella and Speechley12 As has been done in the other reviews using the Downs & Black scale, Reference Samoocha, Bruinvels, Elbers, Anema, van der Beek and Allard10–Reference Chudyk, Jutai, Petrella and Speechley12 we grouped score ranges into the following four levels: excellent (26–28), good (20–25), fair (15–19) and poor (⩽14). The quality level of excellent can be achieved by randomised studies only. We solved disagreements in the quality assessment by discussion.
Data presentation
We graphically represented the effects of the included studies using a modification of the Harvest plot, which is a flexible method suitable for synthesising evidence about differential effects of heterogeneous and complex interventions, across different variables of interest (such as study design and participant characteristics). Reference Ogilvie, Fayter, Petticrew, Sowden, Thomas and Whitehead13,Reference Crowther, Avenell, MacLennan and Mowatt14 In our plots, effect size is represented by a bar, amended by the 95% confidence interval. The study showed a statistically significant effect if the confidence interval is not including the null.
Effect sizes and meta-analyses
Only randomised studies were included in the meta-analysis. We entered and analysed data using comprehensive meta-analysis. We calculated dichotomous outcomes on an intention-to-treat basis, taking into account the initial number of participants included in each study. For the analysis of dichotomous outcomes we used odds ratios (ORs) with 95% confidence intervals, in order to use direct estimates from cluster randomised trials accounting for clustering wherever possible. We analysed continuous data using standardised mean differences (SMDs), as this measure of treatment effect allows combining scores from different scales. If end-point data were unavailable, we analysed change score data. Where intention-to-treat (ITT) data were available we preferred these over ‘per-protocol analysis’. Effect sizes were converted to SMDs for all outcomes in the harvest plot. In contrast to the protocol, we combined effect sizes for provider performance and patient outcomes across diagnoses, because of the small number of studies and the studies with mixed populations.
We investigated statistical heterogeneity by visual inspection of the forest plots. This was supplemented by the I 2 statistic, which provides an estimate of percentage of variability because of heterogeneity rather than chance alone. When the I 2 estimate was greater than or equal to 50%, we interpreted this as indicating the presence of high levels of heterogeneity. Reference Higgins, Julian, Thompson, Deeks and Altman15
Results
Characteristics of included studies
The search yielded a total of 1750 records potentially relevant for this review (online Fig. DS1). Of these, we excluded 1668 records based on the review of titles or abstracts. We retrieved the remaining 82 articles in full text and assessed them for inclusion. Of these, we excluded 63 studies for the reasons reported in online Fig. DS1. The remaining 19 studies Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16–Reference Yoshino34 met the inclusion criteria and were included in the present review. The meta-analysis included six RCTs providing data on the impact of guideline implementation on process outcomes Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Bauer, Biswas and Kilbourne19,Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 and three RCTs on patient outcomes. Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Osborn, Nazareth, Wright and King26 The main study characteristics are presented in online Table DS1.
We included eight RCTs Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Bauer, Biswas and Kilbourne19,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 four CCTs Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32,Reference Yoshino34 and seven studies with a before-and-after design. Reference Baker, Lovell and Harris17,Reference Chong, Ravichandran, Poon, Soo and Verma20,Reference Chuang and Crismon21,Reference Mistler, Mellman and Drake24,Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28,Reference Steinacher, Mausolff and Gusy29,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 Nine studies enrolled patients with a diagnosis of schizophrenia or related psychotic disorders, Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Chong, Ravichandran, Poon, Soo and Verma20–Reference Ince, Tai and Haddock23,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Steinacher, Mausolff and Gusy29,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 whereas five studies included participants with unipolar and bipolar affective disorders. Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Bauer, Biswas and Kilbourne19,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32,Reference Yoshino34 One study Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28 included two independent samples of patients diagnosed with schizophrenia or depression. Two studies Reference Mistler, Mellman and Drake24,Reference Osborn, Nazareth, Wright and King26 recruited people with mixed diagnoses (schizophrenia or related psychotic disorders, unipolar and bipolar depression). Diagnosis was not reported in two studies. Reference Baker, Lovell and Harris17,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 The majority of studies Reference Baker, Lovell and Harris17–Reference Bauer, Biswas and Kilbourne19,Reference Chuang and Crismon21,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Mistler, Mellman and Drake24,Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28,Reference Steinacher, Mausolff and Gusy29,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 were conducted in in-patient settings, six studies were performed in out-patient settings, Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Ince, Tai and Haddock23,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32,Reference Yoshino34 one study Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27 reported both in-patient and out-patient settings and two studies Reference Chong, Ravichandran, Poon, Soo and Verma20,Reference Osborn, Nazareth, Wright and King26 did not report information on the setting.
Characteristics of the implementation strategies
The implemented guidelines included clinical practice manuals and evidence-based treatment algorithms, which were developed with a structured methodology that included a consensus group of experts, analysis of the evidence base and clinical input. Fifteen studies Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Baker, Lovell and Harris17,Reference Bauer, Biswas and Kilbourne19,Reference Chuang and Crismon21–Reference Ince, Tai and Haddock23,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25–Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 assessed the impact of multifaceted implementation strategies that included two or more components. All studies implemented the guidelines at a professional level, for example by distribution of educational material, educational meetings, educational outreach visits, reminders and audit and feedback. Five studies Reference Bauer, Biswas and Kilbourne19,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32 additionally used strategies at an organisational level targeting patient needs, for example offering an education programme for patients and their families. Two studies Reference Bauer, Biswas and Kilbourne19,Reference Osborn, Nazareth, Wright and King26 used provider-orientated implementation strategies at an organisational level, such as through employment of a nurse as coordinator (online Table DS1). Details about implementation strategies were not available in four studies. Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Chong, Ravichandran, Poon, Soo and Verma20,Reference Mistler, Mellman and Drake24,Reference Yoshino34 Fifteen studies Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16–Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Mistler, Mellman and Drake24,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25, Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28–Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32–Reference Yoshino34 compared guideline implementation strategies with usual care, and four randomised studies Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 compared different guideline implementation strategies (online Table DS1). According to the Downs & Black quality check, the quality of included studies was graded as fair in eleven Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Bauer, Biswas and Kilbourne19,Reference Ince, Tai and Haddock23,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25–Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32 and poor in six studies. Reference Baker, Lovell and Harris17,Reference Chong, Ravichandran, Poon, Soo and Verma20–Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Mistler, Mellman and Drake24,Reference Yoshino34 Only two studies Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 achieved a good-quality score.
Impact of guideline implementation on provider performance
Twelve studies assessed the impact of guideline implementation on provider performance Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Baker, Lovell and Harris17,Reference Bauer, Biswas and Kilbourne19–Reference Chuang and Crismon21,Reference Mistler, Mellman and Drake24,Reference Osborn, Nazareth, Wright and King26–Reference Steinacher, Mausolff and Gusy29,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 (Fig. 1), including four studies comparing implementation strategies. Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 The quality of the majority of studies was judged to be fair.
Fig. 1 Harvest plot.
The studies are grouped according to their quality (poor, fair good) and the two rows indicate whether the outcome refers to patients or provider performance. SMD, standardised mean difference.
Guideline implementation v. treatment as usual
In comparison with usual care, a positive effect of guideline implementation was shown in six studies, Reference Baker, Lovell and Harris17,Reference Bauer, Biswas and Kilbourne19,Reference Chong, Ravichandran, Poon, Soo and Verma20,Reference Mistler, Mellman and Drake24,Reference Steinacher, Mausolff and Gusy29,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 but in only three of them Reference Bauer, Biswas and Kilbourne19,Reference Mistler, Mellman and Drake24,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 was this difference statistically significant. With the exception of one study, Reference Mistler, Mellman and Drake24 showing a large statistically significant effect size, the effect sizes were small to modest. A negative effect of guideline implementation strategies on provider performance was found in four studies. Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Chuang and Crismon21,Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28,Reference Steinacher, Mausolff and Gusy29 This negative effect was statistically significant in two studies. Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Chuang and Crismon21 One study Reference Steinacher, Mausolff and Gusy29 examined two different implementation strategies (active v. passive), but did not compare these strategies directly. The study showed a small positive effect size for the active dissemination strategy and, by contrast, a larger negative effect for the passive strategy, but both results were not statistically significant. Reference Steinacher, Mausolff and Gusy29
Two RCTs comparing guideline implementation with usual care were included in the meta-analysis. Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Bauer, Biswas and Kilbourne19 These studies (Fig. 2) did not show a statistically significant effect of guideline implementation on provider performance (OR = 1.01, 95% CI 0.37–2.79) (Fig. 2). The statistical heterogeneity of these two studies was high (I 2 = 91%).
Fig. 2 Forest plot for process outcomes.
s.e., standard error; IV, inverse variance; CI, confidence interval; TAU, treatment as usual.
Enhanced guideline implementation v. basic implementation strategy
All four studies Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 comparing different guideline strategies showed positive effect sizes, but in two of these Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 the effect was negligible. One study Reference Osborn, Nazareth, Wright and King26 showed a large effect size and a statistical significant advantage of the enhanced implementation strategy (Fig. 2). All four studies were included in the meta-analysis, which did not show a statistically significant advantage of enhanced implementation strategies (OR = 1.47, 95% CI 0.86 to 2.52) (Fig. 2).
Impact of guideline implementation on patient outcomes
Guideline implementation v. treatment as usual
The impact of guideline implementation on patient outcomes was investigated in ten studies Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Mistler, Mellman and Drake24,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28–Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32–Reference Yoshino34 (Fig. 1). Three studies Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32 presented the results of the same project separately for patients with different diagnoses. The quality of the majority of studies was judged to be fair. Two studies Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28,Reference Steinacher, Mausolff and Gusy29 showed negative effects of guideline implementation on patient outcomes. Four non-randomised studies Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32–Reference Yoshino34 showed a statistically significant effect in favour of the guideline implementation strategy (Fig. 1). With the exception of one study, Reference Steinacher, Mausolff and Gusy29 showing a large and statistically significant, but negative effect for a passive implementation strategy, effect sizes were small to modest. Two randomised studies Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22 were included in the meta-analysis of patient outcomes (Fig. 3). The meta-analysis did not reveal a statistically significant effect of guideline implementation on patient outcomes (OR = 1.46, 95% CI 0.91–2.35). There was no statistical between-study heterogeneity (I 2 = 0%) (Fig. 3).
Fig. 3 Forest plot for patient outcomes.
s.e., standard error; IV, inverse variance; CI, confidence interval; TAU, treatment as usual.
Enhanced guideline implementation v. basic implementation strategy
One RCT comparing two implementation strategies assessed patient outcomes. Reference Osborn, Nazareth, Wright and King26 The effect reported in this study was not statistically significant (OR = 1.62, 95% CI 0.83–3.15), but numerically in favour of the enhanced guideline strategy (Fig. 3).
Discussion
Main findings
The present systematic review revealed that the pathway from evidence-based guidelines to evidence-based practice is still a neglected research area and that convincing evidence for beneficial effects of guideline implementation is scant. This is consistent with results from previous studies in other fields of medicine, Reference Karbach, Schubert, Hagemeister, Ernstmann, Pfaff and Höpp35–Reference Hagemeister, Schneider, Barabas, Schadt, Wassmer and Mager37 which showed that guidelines are still not adequately implemented. Only a minority of the studies included in our review showed a positive statistically significant effect of guideline implementation on provider performance or patient outcomes, and often these studies employed a non-randomised design. In contrast to previous reviews, our data, including the meta-analyses of randomised trials, do not suggest an impact of guideline implementation on provider performance. In other words, efforts to implement guidelines did not modify healthcare provider performance in terms of better adherence to clinical practice guidelines. Despite this finding, the studies showed a more or less consistent positive effect of guideline implementation on patient outcomes, i.e. clinical condition, remission rate and satisfaction with care. Although the meta-analysis did not reveal a statistically significant effect of guideline implementation strategies compared with usual treatment or basic implementation strategies, the magnitude of the effect sizes and the consistency of the effects across studies seem to speak in favour of an effect on patient outcomes, which may not have been revealed as a result of low power.
Interpretation of our findings
The surprising result of a missing effect of guideline implementation on guideline adherence may, in part, be explained by differences between the studies included in meta-analyses and by the methodological difficulties of evaluating complex interventions. One reasonable explanation may be that guideline adherence is already high in treatment-as-usual patient groups, and thus may not be sensitive for change. However, this raises the issue of why a majority of studies have suggested positive effects on patient outcomes. Modest clinical improvement of patients may reflect a Hawthorne effect. However, the development of guidelines based on RCTs is criticised for poor external validity, as a result of highly selective inclusion criteria and selective trial settings. Reference Rothwell38 Furthermore, in a small survey of clinicians in the USA Perlis et al Reference Perlis39 found that the most frequently cited reason for not using guidelines was that they do not adequately reflect features of patients that are relevant in making treatment decisions. Thus, the effect on patient outcomes may reflect (some) thoughtful and well-founded clinical decisions of non-adherence for a particular patient. To shed light on the limited effects of guideline implementation, reasons for guideline deviations should be researched. Only one of the studies Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18 included in our review reported having documented reasons for non-adherence with the guideline in the intervention group, but these results were not included in the analyses. Clinical practice guidelines neither are intended to overrule practitioners' experience, nor are they unequivocal reflections of the scientific evidence. Our review reveals a clear gap of knowledge on how guidelines may influence the process of clinical care. Nevertheless, if quality of care is understood as actions increasing the likelihood that best patient outcomes are reached, our results challenge the use of guideline adherence as a quality indicator.
In the light of the low level of evidence in general, we were not able to determine which implementation strategies perform best. Both studies that were showing a substantial and significant effect on provider performance implemented the guideline at an organisational level by introducing a nurse as coordinator. Reference Bauer, Biswas and Kilbourne19,Reference Osborn, Nazareth, Wright and King26 However, our results do not clearly indicate that multifaceted implementation strategies are superior to simple strategies. This is consistent with findings from a systematic review of guideline implementation strategies in other fields of medicine that failed to report either superiority of multifaceted interventions over other types of implementation or any relationship of number of intervention components and implementation effect. Reference Grimshaw, Thomas, MacLennan, Fraser, Ramsay and Vale2 Furthermore, evidence on the cost-effectiveness of such organisational changes is currently not available.
Limitations
The present systematic review has some limitations. First, the included studies were substantially heterogeneous in terms of the focus of the guideline, target of the intervention, population, implementation strategies and control group definition. Consequently, it may be difficult to assume that an implementation strategy that proved to be successful in a specific local setting would be similarly successful in a different setting. A second concern is the selection of only one outcome for provider performance and one patient outcome for the purposes of this review (although most studies reported several outcomes). This selection process was based on the background logic of focusing on primary outcomes, but we acknowledge that the included studies analysed a wide range of secondary outcomes that generated clinically interesting insights. The search strategy may have missed some studies as publications may not have used common keywords or may have used keywords that our search failed to capture. Furthermore, the definition of guideline that we employed (systematically developed statements or algorithms, flow charts, tables to assist decisions about appropriate healthcare for specific clinical circumstances) Reference Barbui, Girlanda, Ay, Cipriani, Becker and Koesters4 left some subjectivity in deciding whether a publication or document could be considered a guideline.
Implications
In conclusion, mental healthcare professionals are faced with little safe ground regarding the best use of available guidelines, and further efforts should be made to understand the effects of guidelines on clinical practice. Also, developing guidelines and having them available (and performing studies on their implementation) could change routine practice in domains not related or not coinciding with provider-guideline adherence. If having guidelines available or being involved in studies on guideline implementation were related to improvements in clinician (or team) motivation, qualification or behaviour, patient outcomes might improve without enhancing guideline adherence. This possibility highlights the conundrum of how clinical effects come about and what makes clinicians and interventions effective in everyday practice.
Funding
This research was supported by a grant no. from the German Federal Ministry of Education and Research. We thank Professor Steinacher for providing additional data.
eLetters
No eLetters have been published for this article.