Background
The relevance of the drug and alcohol problem in terms of frequency and health and social impact is quite pronounced. Substance use disorders are associated with a wide range of serious health, social and economic complications. The life expectancy of alcohol and drug users is often significantly lower than that of the general population (Wahren et al. Reference Wahren, Brandt and Allebeck1997; Price et al. Reference Price, Risk, Murray, Virgo and Spitznagel2001; Sørensen et al. Reference Sørensen, Jepsen, Haastrup and Juel2005), with a great impact on the mortality of young adults (Bargagli et al. Reference Bargagli, Hickman, Davoli, Perucci, Schifano, Buster, Brugal and Vicente2006). Different interventions are offered for the prevention and treatment of substance use and dependence. The choice is often guided by common sense, intuition, experience, beliefs or ideology and not always by evidence. Clinicians and policy makers need accessible, up-to-date, objective evidence regarding the effectiveness of different interventions.
The Cochrane approach puts much emphasis on the rigor of method, independence and transparency, and historically relied on reviewers' interest in specific topics and individual commitment. As soon as the potential use of Cochrane reviews was recognized, particularly in UK, Canada and Australia, substantial amount of investment has been made in trying to produce reviews that were not only of good quality but were also relevant for different stakeholders (clinicians, consumers, policy makers, etc.). While there is a wide range of literature showing that the quality of Cochrane systematic reviews is consistently better that non-Cochrane reviews (Jadad et al. Reference Jadad, Moher, Browman, Booker, Sigouin, Fuentes and Stevens2000; Olsen et al. Reference Olsen, Middleton, Ezzo, Gøtzsche, Hadhazy, Herxheimer, Kleijnen and McIntosh2001; Moja et al. Reference Moja, Telaro, D'Amico, Moschetti, Coe and Liberati2005; Delaney et al. Reference Delaney, Bagshaw, Ferland, Laupland, Manns and Doig2007; Moher et al. Reference Moher, Tetzlaff, Tricco, Sampson and Altman2007; Jørgensen et al. Reference Jørgensen, Maric, Tendal, Faurschou and Gøtzsche2008; Tricco et al. Reference Tricco, Pham, Brehaut, Tetroe, Cappelli, Hopewell, Lavis, Berlin and Moher2009), there is less evidence about the relevance and ability of Cochrane reviews to meet the needs of the different stakeholders. Actually some papers have been recently published raising issues of potential limits of Cochrane reviews (Mandel et al. Reference Mandel, Littner, Mimouni and Lubetzky2006; Lang et al. Reference Lang, Edwards and Fleiszer2007; Pagliaro et al. Reference Pagliaro, Bruzzi and Bobbio2010).
The aim of this paper is to review the experience of the Cochrane Drugs and Alcohol Group (CDAG) in terms of production of evidence to guide policy and practice. For this purpose, we analyzed the contribution of the systematic reviews published by the CDAG as of 31 December 2010 in terms of: scope of the group, coverage of topics, body of primary researched reviewed and eventually results of reviews in terms of sufficient evidence to inform practice and research.
What is in the Cochrane Library
The CDAG, as part of the Cochrane Collaboration, aims to produce, update and disseminate systematic reviews of trials on the prevention, treatment and rehabilitation of problematic drug and alcohol use. CDAG was founded in 1998 and has an editorial base in Rome (Davoli & Ferri, Reference Davoli and Ferri2000; further information is available at http://www.cdag.cochrane.org).
Cochrane reviews are produced internationally, a total of 299 authors have published with the CDAG: 185 from the European Union, 29 from Australia, 33 from Asia, 29 from North America, 10 from South America, 8 from South Africa and 5 from the Middle East. The systematic reviews published by CDAG are based primarily on randomized controlled trials (RCTs) and controlled clinical trials that describe an active intervention (including prevention, treatment and rehabilitation) aimed at reducing the potential for harm or the actual harm directly related to the use of different dependence-producing substances; the inclusion of other study designs, however, is considered in limited circumstances (Amato et al. Reference Amato, Davoli, Vecchi, Ali, Farrell, Faggiano, Foxcroft, Ling, Minozzi and Chengzheng2011).
By December 2010, the CDAG published 55 reviews covering pharmacological and psychosocial treatments of opioid (21 reviews), alcohol (12 reviews), cocaine and other psychostimulant (10 reviews), poly drug (4 reviews) and cannabis, benzodiazepine, inhalants and metaqualone (1 review each) abuse or dependence. The effectiveness of preventive interventions across different substances was considered in four reviews (see Table 1).
Studies included and excluded in the Cochrane drugs and alcohol systematic reviews and country of origin of included studies (January 2011)
aOther = cannabis, benzodiazepine mono dependence, metaqualone, inhalants, one review each.
bTen studies in common between two reviews (four studies between opiate and polydrug; four studies between opiate and alcohol; one study between opiate and cocaine; one study between alcohol and cocaine).
One of the main features of the Cochrane Systematic Reviews is the comprehensiveness of the search, which in fact entails an explicit transparent search strategy to find both published and unpublished trials (Higgins & Green, Reference Higgins and Green2008); for this purpose, CDAG created and maintains a specialized register of trials on the evaluation of treatment effectiveness. When the group was created, in 1998, there was no previous knowledge of the body of primary research available to conduct systematic reviews. Actually, the body of primary research on the effectiveness of interventions for alcohol and drug addiction proved to be quite substantial. As of January 2011, the register contained 7150 references to the studies.
The number of trials considered for inclusion in the 55 published reviews represent about one-third of the overall number of potentially available trials. Moreover, out of a total of 2114 trials considered for inclusion, only 744 studies were included (35%), with a total of 241 802 participants (see Table 1). Although the inclusion criteria can vary between reviews and be attributed, in part, to the sensitivity of the search strategy, the proportion of studies that satisfy the criteria is low overall.
Despite the considerable number of trials carried out on the treatment of addiction, our findings seem to confirm that only a few of them contribute to the cumulative knowledge on the effectiveness of interventions (Chalmers, Reference Chalmers1998) (see Table 1).
In the reviews published by the group, only 10% were not RCTs and were included in 3 out of the 55 reviews. Two of these reviews considered maintenance treatments for opioid dependence, and the reasons for inclusion were that they considered long-term outcomes, such as mortality, difficult to be properly analyzed in RCTs due to power limitations. The other non-RCT studies were included in one review on preventive interventions in which non-RCTs often represent the only available source of evidence.
Challenges for the Cochrane reviews
The extent to which a Cochrane review can draw conclusions about the effects of an intervention depends on whether the data and results from the included studies are valid. Quality of evidence is part of the judgment provided by Cochrane reviews: for each study included in a Cochrane review, quality is judged in relation to risk of bias for sequence generation, allocation concealment and blinding; eventually, when results of a Cochrane review come from studies with high risks of bias, the quality of the produced evidence will be described as low.
However, systematic reviews should evaluate and take into account not only the internal validity (i.e., the extent to which systematic errors or bias are avoided) of each trial included but also the applicability and generalizability or external validity (i.e., whether the results of a trial can be reasonably applied to a definable group of patients in a particular setting in routine practice) (Dekkers et al. Reference Dekkers, von Elm, Algra, Romijn and Vandenbroucke2009).
The main threat to external validity comes from the clinical setting, and the social and cultural context in which the studies were conducted, and this is particularly true in the field of addiction, where these contexts can actively affect the overall treatment outcome.
Primary studies included in these reviews, as shown in Table 1, were conducted in North America (64%), Europe (25%), Australia/New Zealand (5%), Asia (3%), the Middle East (1%), South America (1%) and South Africa (1%). The distribution, however, was heterogeneous across substances of abuse; for example, studies conducted in North America varied, respectively, from 93% and 92% for polydrug abuse and psychostimulants, respectively, to 42% and 57% for alcohol and opioid dependence, respectively.
Finally, systematic reviews could be improved in the future by improving the applicability of the results in clinical practice. A recent article (Ahmad et al. Reference Ahmad, Boutron, Dechartres, Durieux and Ravaud2010) assessed the methods and reporting of information on the applicability of trial results in systematic reviews and found that the applicability is poorly reported or taken into account. In order to ameliorate the applicability of results, in the future, authors must identify which applicability items are important (according to the type of treatment evaluated) and should be collected and reported.
Are results of Cochrane reviews useful to inform clinical practice?
In order to measure the extent to which the published Cochrane reviews in the field of drug and alcohol do provide evidence for each intervention evaluated in the systematic reviews, we considered the ‘Implication for practice’ section of the Cochrane review and, following the classification suggested by NHS and UK Cochrane Centre (http://www.library.nhs.uk/qipp/), we classified the interventions as sufficient evidence to: implement the intervention into clinical practice (Do it), to avoid the intervention into clinical practice (Do not do it) and insufficient evidence (Only for research).
Based on these criteria, for the interventions considered in the published reviews, 25 (31%) were classified as to do, 9 (11%) as do not do it, 25 (52%) to do only in research and for 5 (6%) the judgment was impossible because the reviews included no studies or only one study.
These proportions varied according to the type of substance of abuse studied; for example, the interventions that were judged as to do were 26% for alcohol, 32% for opioids, 12% for psychostimulants, 33% for polydrugs and for prevention (see Fig. 1). These results should be considered cautiously, remembering that they referred only to interventions, comparisons and outcomes considered in the studies included in the reviews, and we are aware that the evidence presented is not thorough and definitive. Furthermore, the assessment of the methodological quality of included studies showed relevant weaknesses in the information available to judge their quality, which is not considered in this summary of available evidence.
Evidence of effectiveness of interventions considered in the CDAG reviews. Cochrane Library 2.2011.
Conclusions
Systematic reviews published by the CDAG cover a wide range of substances and interventions, and provide evidence on effectiveness of several interventions.
Providing evidence of effectiveness, however, is only the first step to reduce the gap between practice and policy. Even though Cochrane reviews in the field of addiction were already used to inform clinical guidelines (Prodigy, 2005; Australian National Drug Strategy, 2006; NICE, 2006), it is only after the recent development of the GRADE method (Guyatt et al. Reference Guyatt, Oxman, Kunz, Vist, Falck-Ytter and Schünemann2008) that global guidelines based on Cochrane reviews have been published (WHO, 2009). The main features of the GRADE method that facilitated the development of global guidelines for the treatment of addiction is the explicit identification of critical outcomes and the separation between judgment on quality of evidence and strength of recommendation. A method that provides a transparent and pragmatic way of combining evidence to value judgment and preference to formulate clinical recommendations is of particular use in the field of drug addiction where several cultural and social constraints are involved, and production of evidence-based guidelines have always been problematic.
However, if Cochrane reviews are developed ‘to provide the type of information that is needed by physicians to make clinical decisions’ there is still some progress to be made at least in the way they are presented. Most reviews are too lengthy containing many sections, most of which describing the adherence of the review to the standardized methodology of the Collaboration, which can make their consultation time-consuming with respect to the clinical content of interest for clinicians; a more user-friendly format of Cochrane reviews should be probably developed in the near future to overcome this limitation.
The other side of the coin that should be considered is the potential role of Cochrane reviews to inform the research agenda; there are no empirical data describing the extent to which this occurs, at least in the field of addiction. Cochrane reviews, in fact, do also underline areas of uncertainty, and eventually the high proportion of trials excluded in the reviews highlight a major issue of relevance of primary research in this field. Eventually, Cochrane systematic reviews should also be considered as a tool to inform the research agenda in terms of setting priorities, identifying areas of uncertainty and promoting multicenter high-quality studies addressing questions that contribute to progressing knowledge on the effectiveness of treatment and, consequently, meet the needs of patients, their care givers and policy makers.
Eventually, there are several questions to which our reviews are not providing answers; this may frustrate the confidence of physicians in the usefulness of Cochrane reviews to answer relevant clinical questions, in these cases, when evidence from randomized trials is lacking or insufficient, the opportunity of evaluating the evidence coming from non-randomized studies should be further explored.
Role of Funding Sources
Funding for this study was provided by the Department of Epidemiology, Regional Health Authority Lazio.
Contributors
Marina Davoli and Laura Amato wrote the manuscript.
Conflict of Interest
Both authors declare that they have no conflict of interest pertaining to the manuscript.
