Limitations of CONSORT guidelines for social and psychological interventions

Researchers and journal editors in the social and behavioural sciences are generally aware of CONSORT but often object that it is not fully appropriate for social and psychological interventions. ^{Reference Perry, Weisburd and Hewitt5,Reference Stinson, McGrath and Yamada8} As a result, uptake of CONSORT guidelines in these disciplines is low. ^{Reference Grant, Mayo-Wilson, Melendez-Torres and Montgomery15} Although some criticisms are as a result of inaccurate perceptions about common features of RCTs across disciplines, many relate to real limitations for social and psychological interventions. ^{Reference Mayo-Wilson16} For example, CONSORT is most relevant to RCTs in medical disciplines; it was developed by biostatisticians and medical researchers with minimal input from experts in other disciplines. Journal editors, as well as social and behavioural science researchers, believe there is a need to include appropriate stakeholders in developing a new, targeted guideline to improve uptake in their disciplines. ^{Reference Torgerson, Torgerson, Birks and Porthouse7} The CONSORT Group has produced extensions of the original CONSORT statement relevant to social and psychological interventions, such as additional checklists for cluster, ^{Reference Campbell, Elbourne and Altman17} non-pharmacological, ^{Reference Boutron, Moher, Altman, Schulz and Ravaud18} pragmatic ^{Reference Zwarenstein, Treweek, Gagnier, Altman, Tunis and Haynes19} and quality-of-life RCTs. ^{Reference Calvert, Blazeby, Revicki, Moher and Brundage20} These extensions provide important insights, but complex social and psychological interventions, for example, include multiple, interacting components at several levels, with various outcomes require use of several extensions at once, creating a barrier to guideline uptake; increasing intervention complexity also gives rise to new issues that are not included in existing guidelines. Therefore, simply disseminating CONSORT guidelines as they stand is insufficient, as this would not address the need for editors and authors to ‘buy-in’ to this process. To improve uptake in these disciplines, CONSORT guidelines need to be extended to specifically address the important features of social and psychological interventions.

Limitations of existing social and psychological reporting guidelines

Social and behavioural scientists have developed other reporting guidelines, including the Workgroup for Intervention Development and Evaluation Research (WIDER) Recommendations for behavioural change interventions, ^{Reference Abraham21} the American Educational Research Association's (AERA) Standards for Reporting Research, ²² the REPOSE guidelines for primary research in education ^{Reference Newman and Elbourne23} and the Journal Article Reporting Standards (JARS) of the American Psychological Association (APA). ²⁴ Although they address issues not covered by the CONSORT statement and its extensions, these guidelines (except for JARS ²⁴ ) do not provide specific guidance for RCTs. Moreover, compared with the CONSORT statement and its official extensions, guidelines in the social and behavioural sciences have not consistently followed optimal techniques for guideline development and dissemination that are recommended by international leaders in the advancement of reporting guidelines, ^{Reference Moher, Schulz, Simera and Altman11} such as the use of systematic literature reviews and formal consensus methods to select reporting standards. ^{Reference Grant, Mayo-Wilson, Melendez-Torres and Montgomery15} Researchers in public health, psychology, education, social work and criminology have noted that these guidelines could be more ‘user-friendly’, and dissemination could benefit from up-to-date knowledge transfer techniques. ^{Reference Naleppa and Cagle6,Reference Torgerson, Torgerson, Birks and Porthouse7,Reference Stinson, McGrath and Yamada8,Reference Abraham21}

For example, JARS - a notable and valuable guideline for empirical psychological research - is endorsed by few journals outside of the APA, whereas CONSORT is endorsed by hundreds of journals internationally. According to ISI Web of Knowledge and Google Scholar citations, JARS is cited approximately a dozen times annually, whereas CONSORT guidelines are cited hundreds of times per year. Moreover, the APA commissioned a select group of APA journal editors and reviewers to develop JARS, and the group based most of their work on existent CONSORT guidelines. By comparison, official CONSORT extensions have been developed using rigorous consensus methods, have involved various international stakeholders in guideline development and dissemination, and update content on the most recent scientific literature. Nonetheless, no current CONSORT guideline adequately addresses the unique features of social and psychological interventions. This new CONSORT extension will incorporate lessons from previous extensions, reporting guidelines and the research literature to aid the critical appraisal, replication and uptake of this research (see online supplement).

Poor reporting of current CONSORT items

Some aspects of internal validity, although included in CONSORT, remain poorly reported - even in the least complex social and psychological intervention studies. Reports of RCTs should describe procedures for minimising selection bias, but reports often omit information about random sequence generation and allocation concealment ^{Reference Ladd, McCrady, Manuel and Campbell28} and psychological journals report methods of sequence generation less frequently than medical journals. ^{Reference Stinson, McGrath and Yamada8} A review of educational reports found no studies that adequately reported allocation concealment ^{Reference Torgerson, Torgerson, Birks and Porthouse7} and reports in criminology often lack information about randomisation procedures. ^{Reference Perry, Weisburd and Hewitt5} In addition, RCTs of social and psychological interventions may use non-traditional randomisation techniques, such as stepped-wedge or natural allocation, ²⁹ which need to be thoroughly described. Also, reports of social and psychological intervention trials often fail to include details about trial registration, protocols and adverse events, ^{Reference Ladd, McCrady, Manuel and Campbell28} which may include important negative consequences at individual, familial and community levels.

Possible amendments to CONSORT items

Other aspects of CONSORT may require greater emphasis or modification for RCTs of social and psychological interventions. In developing this CONSORT extension, we expect to identify new items and to adapt existing items that relate to the internal validity. These may include items discussed during the development of previous CONSORT extensions or other guidelines, as well as items suggested by participants in this project. For example, it may not be possible to mask participants and providers of interventions, but masking of outcome assessors is often possible but rarely reported, and few studies explain whether masking was maintained or how lack of masking was handled. ^{Reference Ladd, McCrady, Manuel and Campbell28} In social and psychological intervention studies, outcome measures are often subjective, variables may relate to latent constructs, and information may come from multiple sources (for example participants, providers). Although an issue in other areas of research, the influence on RCT results of the quality of subjective outcome measures in social and psychological intervention research has long been highlighted given their prevalence in research in these disciplines. ^{Reference Marshall, Lockwood, Bradley, Adams, Joy and Fenton30} Descriptions of the validity, reliability and psychometric properties of such measures are therefore particularly useful for social and psychological intervention trials, especially when they are not widely available or discussed in the research literature. ^{Reference Campbell, Elbourne and Altman17} Moreover, multiple measures may be analysed in several ways, so authors need to transparently report which procedures were performed and to explain their rationale.

Details of the intervention and comparator

It is useful for authors to explain the key components of interventions, how those components could be delivered, and how they relate to the outcomes selected. At present, authors can follow current standards for reporting interventions without providing adequate details about complex interventions. ^{Reference Shepperd, Lewin, Straus, Clarke, Eccles and Fitzpatrick32} Many reports neither contain sufficient information about the interventions tested nor reference treatment manuals. ^{Reference Glasziou, Meats, Heneghan and Shepperd33} Providing logic models - as described in the Medical Research Council Framework for Complex Interventions ^{Reference Craig, Dieppe, Macintyre, Mitchie, Nazareth and Petticrew34} - or presenting theories of change can help elucidate links in causal chains that can be tested, identify important mediators and moderators and facilitate syntheses in reviews. Moreover, interventions are rarely implemented exactly as designed, and complex interventions may be designed to be implemented with some flexibility, in order to accommodate differences across participants, ^{Reference Hawe, Shiell and Riley3} so it is important to report how interventions were actually delivered by providers and actually received by participants. Particularly for social and psychological interventions, the integrity of implementing the intended functions and processes of the intervention are essential to understand. ^{Reference Hawe, Shiell and Riley3} As RCTs of a particular intervention can yield different relative effects depending on the nature of the control groups, information about delivery and uptake should be provided for all trial arms.

Participant characteristics

Reports should describe recruitment processes and representativeness of samples. Participants in RCTs of social and psychological interventions are often recruited outside of routine practice settings via processes that differ from routine services. ²² An intervention that works for one group of people may not work for people living in different cultures or physical spaces, or it may not work for people with slightly different problems and comorbidities. Enrolling in an RCT can be a complex process that affects the measured and unmeasured characteristics of participants, and recruitment may differ from how users normally access interventions. Well-described RCT reports will include the characteristics of all participants (volunteers, those who enrolled and those who completed) in sufficient detail for readers to assess the comparability of the study sample to populations and in everyday services. ^22,24

Contextual influences

Given that these interventions often occur in social environments, reports should describe factors of the RCT context that are believed to support, attenuate or frustrate observed effects. ^{Reference Moore35} Interventions may differ across groups of different social or socioeconomic positions, and equity considerations should be addressed explicitly. ^{Reference Welch, Petticrew, Tugwell, Moher, O'Neill and Waters36} Several aspects of setting and implementation may be important to consider, such as administrative support, staff training and supervision, organisational resources, the wider service system and concurrent political or social events. ^{Reference Shepperd, Lewin, Straus, Clarke, Eccles and Fitzpatrick32} Reporting process evaluations may help understand mechanisms and outcomes.

Guideline developers

The project is being led by an international collaboration of researchers, methodologists, guideline developers, funders, service providers, journal editors and consumer advocacy groups. We will be recruiting participants in a manner similar to other reporting guideline initiatives - identifying stakeholders through literature reviews, the project's International Advisory Group and stakeholder-initiated interest in the project. ^{Reference Schulz, Altman and Moher10} We hope to recruit stakeholders with expertise from all related disciplines and regions of the world, including low- and middle-income countries. Methodologists will identify items that relate to known sources of bias, and they will identify items that facilitate systematic reviews and research synthesis. Funders will consider how the guideline can aid the assessment of grant applications for RCTs and methodological innovations in intervention evaluation. Practitioners will identify information that can aid decision-making. Journal editors will identify practical steps to implement the guideline and to ensure uptake.

Consensus methods

We will use formal consensus techniques to reduce bias in group decision-making and to promote widespread guideline uptake and knowledge translation activities on project completion. ^{Reference Grant, Mayo-Wilson, Melendez-Torres and Montgomery15} As indicated by previous research and the development of existing CONSORT guidelines, these methods are the most appropriate and successful ways to synthesise expertise and research evidence for our purposes, and they are beneficial to use in combination. ^{Reference Moher, Schulz, Simera and Altman11} Following a rigorous review of existing guidelines and current reporting quality, ^{Reference Grant, Mayo-Wilson, Melendez-Torres and Montgomery15} we will conduct a Delphi process to identify a prioritised list of reporting items to consider for the extension. That is, we will invite a group of experts to answer questions about reporting items and to suggest further questions. We will circulate their feedback to the group and ask a second round of questions. The Delphi process will capture a variety of international perspectives and allow participants to share their views anonymously. Following the Delphi process, we will host a consensus meeting to review the findings and to generate a list of minimal reporting standards, mirroring the development of previous CONSORT guidelines. ^{Reference Schulz, Altman and Moher10,Reference Boutron, Moher, Altman, Schulz and Ravaud18,Reference Zwarenstein, Treweek, Gagnier, Altman, Tunis and Haynes19}

Project outputs

Together, participants in this process will create a guideline consisting of a checklist of reporting items and a flow chart for reporting social and psychological intervention RCTs. In addition, we will develop an explanation and elaboration document to explain the scientific rationale for each recommendation and to provide examples of clear reporting; a similar document was developed by the CONSORT group to help disseminate a better understanding for each included checklist item. ^{Reference Moher, Hopewell, Schultz, Montori, G⊘tzsche and Devereaux38} This document will help persuade editors, authors and funders of the importance of the guideline. It will be a useful pedagogical tool, helping students and researchers understand the methods for conducting RCTs of social and psychological interventions, and it will help authors meet the guideline requirements. ^{Reference Moher, Schulz, Simera and Altman11}

Stakeholder participation and uptake

The success of this project depends on widespread involvement and agreement among key international stakeholders in research, policy and practice. For example, previous developers have obtained guideline endorsement by journal editors who require authors and peer reviewers to use the guideline during manuscript submission and who must enforce journal article word limits. ^{Reference Moher, Schulz, Simera and Altman11} Many journal editors have already agreed to participate, and we hope other researchers and stakeholders will volunteer their time and expertise.