Response by Chalder, White & Sharpe

In his Round the Corner commentary on a Cochrane Review of exercise therapy for chronic fatigue syndrome (CFS), Mitchell made a number of criticisms of the PACE trial (Reference MitchellMitchell 2017). However, the criticisms are based on inaccurate information and are consequently misleading.

First, Mitchell suggests that a re-analysis of some of the data related to the PACE trial found the effect sizes to be smaller than those which we originally reported (Reference White, Goldsmith and JohnsonWhite 2011). This is incorrect. He has confused the effect size reported in the main trial paper (which was calculated using scores of the two primary outcomes) with results of a secondary analysis of the data. The latter reports the proportions of participants meeting various criteria for recovery (see below) (Reference White, Johnson and GoldsmithWhite 2013; Reference Wilshire, Kindlon and MattheesWilshire 2017).

Second, Mitchell implies that we only released certain results, such as objective metrics from the 6-minute walking test data, as a consequence of data release that was forced on us following an information tribunal hearing in 2016. These results were in fact published in our main results paper 5 years earlier (Reference White, Goldsmith and JohnsonWhite 2011).

Third, Mitchell states: ‘it is also alleged that the investigators (perhaps inadvertently) influenced participants’ self-reports with indiscriminate encouragement in newsletters sent out during the trial’ (Reference MitchellMitchell 2017). It has indeed been alleged, but the allegation is incorrect. As in all well-run trials, we engaged with participants by sending them regular newsletters about trial progress. As part of that, we included quotations of positive feedback about the trial and the treatments that they had received. The newsletters (which readers can review at www.wolfson.qmul.ac.uk/images/pdfs/participantsnewsletter3.pdf) did not name any treatment and included positive quotations about all four treatments being evaluated in the trial. We also measured participants’ expectations of their allocated treatment after they had been informed of it and, as reported in the main paper, most participants considered adaptive pacing therapy (APT) and graded exercise therapy (GET) to be most likely to help them, whereas the trial found that cognitive–behavioural therapy (CBT) and GET were most effective (Reference White, Goldsmith and JohnsonWhite 2011).

Fourth, Mitchell says, ‘It is also alleged that the investigators switched their own scoring methods mid-trial’ (Reference MitchellMitchell 2017). As is common practice in most trials, and as we agreed to do in our original protocol (Reference White, Sharpe and ChalderWhite 2007), the outline analysis plan was reviewed and a detailed analysis plan was written and subsequently published (Reference Walwyn, Potts and McCroneWalwyn 2013). This was approved by two independent oversight committees before any outcome data were analysed. The detailed plan used the same primary outcomes. The change Mitchell is referring to was in the scoring method of one of the primary outcome measures. A binary scoring method (0, 0, 1, 1) was changed to a Likert scoring method (0, 1, 2, 3) in order to provide a more accurate measure of efficacy. This change and the reason for it were clearly reported in the papers (Reference White, Goldsmith and JohnsonWhite 2011; Reference Walwyn, Potts and McCroneWalwyn 2013). Re-analysing the data using the binary scoring made no difference to our conclusions that both CBT and GET are effective treatments (Reference Goldsmith, White and JohnsonGoldsmith 2016).

Fifth, Mitchell criticises us and one of our universities for not releasing more data and earlier. This criticism is misleading. We have already explained that we simply did not have participants’ consent to release their individual patient data (Reference White, Chalder and SharpeWhite 2016). This is because the public release of data, which has now occurred as a result of an information tribunal hearing, and which Mitchell promotes in providing a link, has been explicitly proscribed by our research ethics committee. We have, however, shared data with other researchers, including a Cochrane Collaboration team, who agreed to keep the data confidential.

Finally, Mitchell suggests that a re-analysis of the proportions of participants meeting criteria for recovery suggests that few participants recovered with CBT and GET (Reference Wilshire, Kindlon and MattheesWilshire 2017). We have already pointed out that our recovery (as opposed to improvement) estimates depended on assumptions (Reference White, Johnson and GoldsmithWhite 2013; Reference Sharpe, Chalder and JohnsonSharpe 2017). The Wilshire re-analysis simply makes different assumptions, using more stringent thresholds to determine recovery. That said, our recovery rates were similar to those found in previous studies (22% recovered after CBT and GET) (Reference Sharpe, Chalder and JohnsonSharpe 2017).

We agree with Mitchell that there are lessons to be learnt from the PACE trial, but they are not the lessons he suggests.

Mitchell says: ‘First and foremost, it is imperative for researchers to publish studies in the most open and transparent manner possible’. In fact almost all our papers were published with open access, and we have responded to scientific queries and criticisms appropriately and repeatedly in papers cited here, in journal correspondence, and in over 100 frequently asked questions available on the trial website (www.wolfson.qmul.ac.uk/current-projects/pace-trial). We have also shared data when ethically possible (Reference White, Chalder and SharpeWhite 2016).

Mitchell says: ‘A second lesson is that clinicians and researchers should work more closely with patients…’. In fact a patient charity and a patient were involved early on in designing the trial, and were full members of our trial steering and/or trial management committees (Reference White, Chalder and SharpeWhite 2015).

Mitchell says: ‘The third lesson is that, to promote acceptability, psychosocial treatments should be integrated into medical care’. In fact the PACE trial treatments were integrated with medical care and all participants in the PACE trial received appropriate medical care provided by CFS specialists.

We suggest that the most obvious lesson from our experience of the PACE trial is that science can sometimes provide answers that are not popular with everyone (Lancet 2011; Reference HawkesHawkes 2011; Reference WesselyWessely 2015; Reference Sharpe, Goldsmith and JohnsonSharpe 2016). However, such answers should stand or fall by independent replication, not by unreasonable criticism and demands for retraction. We note that the PACE trial replicated findings from many earlier randomised controlled trials, many of which were conducted by independent researchers in different countries (Reference Castell, Kazantzis and Moss-MorrisCastell 2011; Reference Larun, Brurberg and Odgaard-JensenLarun 2016).

Riposte by Mitchell

I thank Chalder et al for their response to my commentary (Reference MitchellMitchell 2017), which included discussion of the PACE trial. Since publication of this commentary I have been contacted by various individuals claiming I was too lenient and various others stating I was not lenient enough with my evaluation of the PACE trial. All the points raised by Chalder et al have been extensively discussed online already (Reference Wilshire, Kindlon and MattheesWilshire 2017; Reference Sharpe, Chalder and JohnsonSharpe 2017), but I will take this opportunity to look at three key points. The first is how many patients improved in the PACE trial, the second is how many recovered in the PACE trial and the third is the general point whether original suitably anonymised data should be published along with primary studies, an initiative called ‘open data’.

Regarding improvement, I indeed suggested that the effect sizes were smaller than originally reported, because in the original Lancet paper (Reference White, Goldsmith and JohnsonWhite 2011) it was reported that 59% improved following cognitive–behavioural therapy (CBT) and 61% improved following graded exercise therapy (GET) on the SF-36 physical functioning subscale and/or the Chalder Fatigue Questionnaire, but later, in 2016, Chalder et al themselves reported that only 21% improved following GET and 20% improved following CBT using a combination of these measures (Reference Goldsmith, White and JohnsonGoldsmith 2016). It may well be that this difference can be explained by changes to the definition of what constitutes improvement. My own analysis of the now public data-set suggests that the mean change in the Chalder Fatigue Questionnaire scores (from baseline to 52 weeks) was 26.9% (s.d. = 27.9%) in the CBT group v. 26.6% (s.d. = 26.9%) in the GET group. I find that 57.4% of those who received CBT had an arbitrary 30% or greater improvement in fatigue, compared with 52.6% of those who received GET. Looking at the important criterion of metres walked, my analysis suggests that the percentage change from baseline to 52 weeks in metres walked was 7.4% (s.d. = 29.4%) (i.e. a slight improvement) in the CBT group v. 29.6% (s.d. = 134.5%) (i.e. a moderate improvement) in the GET group. It appears that 22.1% of those receiving CBT had an arbitrary 30% or greater improvement in metres walked, compared with 42.7% of those receiving GET.

Regarding recovery, a simplistic definition of recovery would be how many no longer met Oxford criteria for the diagnosis of CFS (Reference Sharpe, Archard and BanatvalaSharpe 1991) at 52 weeks. Here I find that 27% of those who received GET and similarly 27% of those who received CBT no longer met Oxford criteria at 52 weeks (i.e. the two groups showed exactly the same rate of recovery). However, combining this metric with data for patient-rated change (a clinical global impression of change score at 52 weeks of much improved or greater) as well as data for those who were no longer fatigued (judging by a score of 3 or lower using binary scoring on the Chalder Fatigue Questionnaire at 52 weeks), I find that 12.5% who received GET v. 10.8% who received CBT were recovered. Overall, these results suggest to me that GET was probably more effective than CBT, but only on some measures. Ultimately, only a small proportion who received either (or indeed any) treatment were fully recovered at 52 weeks.

The last point I want to make is that my call for open data reflects a movement that has been gathering momentum for several years and that will hopefully reduce misunderstanding between researchers, journal editors, patients and the public, who often wish to comment on medical research. The AllTrials campaign launched in 2013 called for a clinical trials register together with a published summary of trial protocols (www.alltrials.net/about/). In the same year, the BMJ launched the Open Data Campaign to encourage sharing of primary research data and announced that it will no longer publish any trial where the authors do not commit to making the relevant anonymised patient-level data available on reasonable request (www.bmj.com/open-data). The International Committee of Medical Journal Editors (ICMJE) believes that there is an ethical obligation to responsibly share data generated in interventional clinical trials because trial participants have put themselves at risk, and it hopes to create an environment in which the sharing of de-identified individual participant data no later than 6 months after publication becomes the norm (Reference Taichman, Backus and BaethgeTaichman 2016). Facilities to allow researchers to upload their data are available. Some have objected, arguing for a much longer delay before publication of trial data and financial compensation from those using their data who were not involved in the trial (International Consortium of Investigators for Fairness in Trial Data Sharing 2016). The ICMJE accepts the quality standard of trial registration (ideally with data) in any registry listed in the WHO International Clinical Trials Registry Platform (ICTRP) or in ClinicalTrials.gov. It endorses these registries because they meet several criteria. They are accessible to the public at no charge, open to all prospective registrants, managed by a not-for-profit organisation, have a mechanism to ensure the validity of the registration data, and are electronically searchable. I thank Chalder et al for the opportunity to raise these points again. It is important that all clinicians and academics involved in mental health research are aware of the Open Data Campaign as well as the AllTrials campaign, because in my opinion there is no reason for mental health research to be treated any differently from any other area of medical research.