How were the papers selected?

A Cochrane information specialist (Box 2) conducted a search in March 2020 of MEDLINE, Embase and grey literature and complementary searches on numerous Chinese biomedical databases. For each selected article, the references were reviewed for any further unidentified papers, and conference proceedings were hand-searched to identify unpublished work.

Two authors independently assessed the suitability of papers, extracted the data and assessed methodological bias. Any disputes were reviewed by a third arbitrator. Methodological bias was assessed using the Cochrane Handbook for Systematic Reviews of Interventions (Higgins Reference Harris2011). Researchers used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to assess the certainty of the evidence. This is a framework to enable clinical recommendations by assessing the risk of bias, imprecision, inconsistency, indirectness and publication bias (Langendam Reference Kayser2013).

Continuous data outcomes were measured using mean differences (m.d.) with 95% confidence intervals (95% CI). Dichotomous outcomes were calculated as risk ratios (RR) with 95% confidence intervals using a random-effects model. Heterogeneity was assessed using the chi-squared statistic (χ²). Results were deemed to be substantially heterogenous if I ² was over 50% or P equalled less than 0.1 for χ².

Paper selection and results

A total of 382 papers were initially identified, of which 376 were excluded for reasons such as not analysing TRD and not being randomised. The remaining six papers covered three studies. Along with the three included trials, one ongoing trial was identified (NCT03191058), the recruitment phase of which is due to end in 2024. All three trials were single-site open-label parallel RCTs, and all were conducted by one research team at University Hospital Bonn, Germany. A total of 60 participants were recruited, with the mean age of 45.8 and 54.2 years for MST and ECT respectively; 48% of the participants were female.

All three studies classified TRD as a failure in response to two treatments. They excluded patients with a diagnosis of other psychiatric, neurological or cognitive disorders and those at high risk of harm from anaesthesia. The follow-up period for these studies was short – up to 6 weeks – and all took place during or before 2012.

All three studies delivered MST using twin coils at a pulse frequency of 100 Hz over 8–12 sessions. ECT was delivered unilaterally to all but one participant. All studies continued concomitant antidepressants and/or psychotherapy. The risk of bias was either unclear or high in all the trials, and high risk was seen in all three trials in relation to masking (‘blinding’) of participants and personal and outcome assessment; selective reporting was also noted.

Does MST work?

All studies compared MST with ECT; analysis showed no clear difference in reported symptom severity (40 participants: m.d. = 0.71, 95% CI −2.23 to 3.62, P = 0.64). In relation to cognitive function, two studies investigated multiple domains of cognitive function, but they did not report the methods they used, except the Wechsler Memory Scale. Analysis of both immediate and delayed memory outcomes showed no clear differences (20 participants: m.d. = 0.4, 95% CI −4.16 to 4.96, P = 0.86 and m.d. = 2.57, 95% CI −2.39 to 7.53, P = 0.31 respectively). Kayser et al (Reference Joshi2011) reported skewed follow-up data for symptom severity and cognitive function and baseline data that were unbalanced. Polster et al's (Reference Papakostas2015) study suffered the same problem in relation to delayed memory.

Of the review's six secondary outcomes, only three were reported in these studies. There were no clear differences between treatment groups in relation to quality of life and drop-out for any reason. One study reported two participants experiencing adverse events that led to discontinuation of treatment, both in the ECT arm of the trial. All analyses had a ‘very low’ quality of evidence.

Are there any limitations?

It is important to note that these three studies each lasted a maximum of 6 weeks. They gave a maximum treatment of 12 sessions of both ECT and MST, the frequency of which was not reported. As per the National Institute for Health and Care Excellence (NICE), ECT is usually conducted at a rate of two sessions a week (NICE 2003). Importantly, if ECT sessions occurred at an increased frequency in these studies then this might produce different results than are replicable in the community. Furthermore, the lack of follow-up beyond this means that we cannot assess medium- or long-term outcomes or adverse effects. This makes it hard to extrapolate the data safely to everyday use.

Removing selection bias is essential to ensure that the outcomes of the study are true and not due to participant differences (Catalogue of Bias Collaboration Reference Porter2017). In Polster et al's (Reference Papakostas2015) trial, the inclusion criterion for undergoing MST was the ‘absence of former ECT treatments’. This is significant as these patients were excluded from the MST treatment group but not the ECT group. Therefore, it could be proposed that if a participant was placed in the ECT group who had previously had a successful outcome from ECT, then this could increase the chance of ECT working and negate the true effect of MST. This is further confounded by the lack of background information regarding participants’ previous treatments. For example, there are no data on previous interventions, which antidepressants had been used and whether ECT had been trialled, emphasising the difficulty assessing the effectiveness of MST when we are unable to allow for these confounding factors.

It is worth noting that the review did not mention esketamine as a potential novel treatment for TRD (Swainson Reference Swainson2019), especially with growing evidence that, as an adjunct, it might improve remission rates and treatment response (Papakostas Reference Paillère Martinot2020). This is important, as esketamine may prove to have greater efficacy than MST.

One of the advantages of the three studies is their definition of TRD as a ‘failure of two different antidepressants’. This is important as the review's initial inclusion criterion of ‘no response to at least 4 weeks of one or more antidepressants’ might have generated results not in keeping with the consensus on the definition of TRD. This is that ‘at least 2 antidepressants in adequate dose, duration and compliance’ should ‘fail to produce a clinical improvement’ (Berlim Reference Bekelman, Li and Gross2007). Furthermore, one could argue (despite no documented evidence) that ECT is not usually offered in the UK after the failure of just one treatment for 4 weeks.

Moreover, these three studies were funded in part by MagVenture A/S, a manufacturer of the MST device, leading to potential sponsorship bias (Box 3), and were conducted by one research team at the University Hospital Bonn, thus limiting the generalisability of the results. One could suggest that changing the inclusion criteria for the review to include quasi-RCTs might have mitigated this risk despite the abovementioned increased risk of selection bias. This is because relevant data might have been missed by ruling out this type of study when only three studies, all with a high/unclear risk of bias, were included.