The UK's National Institute for Health and Care Excellence (NICE) is clear that medications should not be used to treat the core features of autism (NICE 2012, 2013). Some psychosocial interventions target these core features by improving social communication skills or reducing repetitive or restricted behaviours (Fuller Reference Fuller and Kaiser2020; Gosling Reference Gosling, Cartigny and Mellier2022). These approaches have been criticised by a growing ‘neurodiversity’ movement, which sees autism as a different but valid structure of key cognitive and affective processes (Pellicano Reference Pellicano and den Houting2022a). NICE guidance (NICE 2012, 2013) on the treatment of autism emphasises psychosocial and environmental adjustments to maximise function and reduce patient/carer distress and is clear that pharmacological treatment should not be offered to treat the core features of the condition.
Adults and children with an autism spectrum disorder (ASD), particularly those with intellectual disability, can present to clinical services with challenging behaviour, which can manifest as physical or verbal aggression, self-injurious behaviours or property damage (Melville Reference Melville, Cooper and Morrison2008). These behaviours can significantly impair functioning and quality of life, for both patients and carers (Butrimaviciute Reference Butrimaviciute and Grieve2014). Non-pharmacological interventions are considered first-line management strategies in the UK (NICE 2012, 2013). These include educational, behavioural and social communication strategies (Myers Reference Myers and Johnson2007). Antipsychotics may be used, however, if behavioural management fails (NICE 2012, 2013). Aripiprazole has a better side-effect profile than other antipsychotics, particularly in terms of weight gain, movement problems and sedation (Komossa Reference Komossa, Rummel-Kluge and Schmid2009), therefore representing a tempting choice for this patient group.
Summary of the Cochrane Review
The review by Hirsch & Pringsheim (Reference Hirsch and Pringsheim2016) in this month's Cochrane Corner searched for evidence on the treatment with aripiprazole of a number of core features of autism. Side-effects were also assessed. The authors included three randomised controlled trials (RCTs) (Marcus Reference Marcus, Owen and Kamen2009; Owen Reference Owen, Sikich and Marcus2009; Findling Reference Findling, Short and Leskovec2014) involving under-18s diagnosed with ASD, two of which (Marcus Reference Marcus, Owen and Kamen2009; Owen Reference Owen, Sikich and Marcus2009) contributed data (on 316 individuals) to the meta-analysis. The authors suggested that aripiprazole has a clinical benefit in improving irritability and hyperactivity, but also that it is associated with side-effects such as motor symptoms, sedation and weight gain.
Definition of the clinical question
This Cochrane Review aimed to assess the safety and efficacy of aripiprazole as a treatment for people with autism. The population investigated was patients with a clinical diagnosis of autism. The intervention was oral aripiprazole and the comparison was placebo (Box 1). The primary outcomes assessed were emotional and behavioural symptoms and extrapyramidal side-effects.
Placebo
Why include a placebo control in a clinical trial? In most clinical trials, participants in the placebo arm also show an improvement. There are two factors at play. First, there is the ‘placebo effect’ – patients will improve if they believe they are receiving an active ingredient. Second is the ‘Hawthorne effect’ – the effect of being studied and observed can also lead to improvements. This may be especially true in psychiatry, where research participants may receive much more intensive follow-up and support than in routine clinical care, and outcome measures are often subjective. To demonstrate that an intervention itself is effective, it must therefore perform better than the placebo.
The placebo effect and the Hawthorne effect will still happen even if a participant knows they are receiving a placebo – which is why unmasked trials are still worth doing.
An initial criticism lies in the definition of the clinical question. The title and aim of the review suggest that aripiprazole was being investigated for the treatment of autism. However, none of the included trials claimed to treat the core syndrome of autism, but rather the target for treatment was challenging behaviour in the context of autism. Furthermore, two of six possible efficacy outcome measures – i.e. stereotypy and inappropriate speech – would be considered by some people with autism to be benign characteristics of autism rather than problems requiring intervention (Pellicano Reference Pellicano, Fatima and Hall2022b).
Method and results
The authors looked at RCTs identified via a highly robust search strategy. Although the study question was broad, only three papers were eventually included in the review, of which only two could be included in the meta-analysis. The third paper was an RCT of discontinuation of aripiprazole in patients who had initially responded well.
Overall, the quality of evidence was classified as ‘moderate’ for all outcomes, owing to study design limitations and the small number of studies. The authors did not judge funding from pharmaceutical companies to be a matter of concern, and thus rated all three trials as having ‘low’ risk of bias. However, all three studies in the review were sponsored by the pharmaceutical company that manufactures aripiprazole, which was under patent in the USA at the time of publication. There is evidence, however, that industry-sponsored trials are more likely to yield a positive result than those funded by not-for-profit organisations (Lundh Reference Lundh, Lexchin and Mintzes2017; Xie Reference Xie and Zhou2022) (Box 2). Moreover, the collaboration with the pharmaceutical industry did not seem to follow best practice for ‘joint working’. Ideally, the study funder should have no input into the design of the trial or the write-up and publication of the results, and a declaration of this should be stated in the paper; however, there is no such declaration in any of the three papers. Moreover, multiple authors for each paper were employed by the study sponsor, but no statement of contributions was explicitly reported. Therefore, concerns remain regarding the assessment of this bias domain in the Cochrane Review.
Industry sponsorship of clinical trials
Drug manufacturers have a competing financial interest in demonstrating that their products are effective, particularly while those drugs are under patent and they have the exclusive right to manufacture them. Much literature shows evidence of this influence on the results of clinical trials. For example, trials sponsored by the pharmaceutical industry are more likely to report positive results than those sponsored by other bodies, in a way that cannot be easily accounted for in risk of bias analysis (Lundh Reference Lundh, Lexchin and Mintzes2017), and industry-sponsored analyses are more likely to conclude that interventions are cost-effective (Xie Reference Xie and Zhou2022).
The Cochrane authors presented findings on six primary treatment outcomes. The largest effect size for improvement was for the irritability and hyperactivity subscales of the Aberrant Behavior Checklist (ABC), with reductions of 6.17 points (out of a total possible score of 45 points) and 7.93 points (out of a total possible score of 48 points) respectively. All trials included in this review used the ABC irritability subscale as their main outcome measure. This scale was developed to assess adults with ‘severe mental retardation’ in US institutions (Aman Reference Aman, Singh and Stewart1985). The ABC checklist has been validated for use in under-18s in the community, but its irritability subscale had a correlation of only 0.6–0.7 with ratings of important clinical presenting problems such as aggressive behaviour or conduct problems (Kaat Reference Kaat, Lecavalier and Aman2014). It is also not clear that a reduction on this scale is a relevant outcome important to children or their parents; a post hoc review of quality of life (QoL) data does, however, suggest that this is the case (Varni Reference Varni, Handen and Corey-Lisle2012). It is unclear why QoL was not a primary outcome of interest, when it is the most patient-relevant outcome.
There was a statistically significant increase in extrapyramidal side-effects in the aripiprazole group, with a risk ratio (RR) of 1.89. For secondary outcomes, aripiprazole increased the risk of weight gain (RR = 3.78). However, results for weight gain appeared conflicting: weight gain as a continuous variable showed no difference between the groups, whereas weight gain categorised as a binary variable did. Body mass index (BMI) also did not show any significant difference. Furthermore, these results may be difficult to interpret in a population of children that would be expected to be gaining weight as they grow up. Aripiprazole also gave a much higher risk of sedation, drooling and tremors (RRs of 4.3, 9.6 and 10.3 respectively).
Aripiprazole is believed by some clinicians to have a neutral effect on weight gain, but the evidence does not support this. A meta-analysis found that aripiprazole did lead to significant weight gain in under-18s, although less than risperidone or olanzapine (Almandil Reference Almandil, Liu and Murray2013). Similar results were found in adults (Tek Reference Tek, Kucukgoncu and Guloksuz2016; Barton Reference Barton, Segger and Fischer2020). Aripiprazole is also believed to be less likely to cause extrapyramidal side-effects, but a network meta-analysis showed that aripiprazole performed no better than any antipsychotic except molindone in under-18s with psychosis (Pagsberg Reference Pagsberg, Tarp and Glintborg2017), while in adults a Cochrane Review found that aripiprazole caused fewer extrapyramidal side-effects than risperidone, but not olanzapine, clozapine, quetiapine or ziprasidone (Khanna Reference Khanna, Suo and Komossa2014).
Interestingly, the heterogeneity (Box 3) of side-effect outcomes was less than the heterogeneity for treatment effect outcomes, suggesting that the magnitude of the side-effects had been more consistently and robustly demonstrated by these studies.
Heterogeneity
This meta-analysis reports the statistical heterogeneity of the studies using the I 2 statistic, which in this case would help answer the following question: ‘If aripiprazole has the same effect in each of the populations studied, how likely is it that we would see such different results from each study?’. Broadly, I 2 of 0% means that the studies show very similar results, with any differences likely attributable to random error; I 2 of 100% suggests that the studies are so different that it is unlikely that they are all measuring the same underlying effect size.
Statistical heterogeneity can be driven by clinical heterogeneity, such as different populations, drug doses or outcome measures; or by methodological heterogeneity, such as different designs or risk of bias. A low statistical heterogeneity despite clinical and study heterogeneity can increase the confidence in the findings reported.
Discussion
In addition to the reservations noted above regarding the internal validity of this study, evidence for the efficacy of aripiprazole in autism has limited external validity (Box 4). Data from only 313 patients were included in the meta-analysis, and all the included trials were conducted by the same research group in the USA. One purpose of performing a meta-analysis is to use several trials to estimate the main effect of the treatment from a variety of studies with different effect sizes. Although it is statistically valid to combine the two trials (Marcus Reference Marcus, Owen and Kamen2009; Owen Reference Owen, Sikich and Marcus2009), they are identical in their intervention, outcome measure and recruitment population, and even share authors, meaning that the meta-analysis does not provide any additional generalisability to the results.
Internal versus external validity
Internal validity is the extent to which the study effectively measures the effect in question in the participants in the study. Good randomised controlled trial design, such as masking (‘blinding’), randomisation and well-validated outcome measures, improves the internal validity of the trial.
External validity is the extent to which the findings of the study are applicable to other groups of patients. External validity can be improved by recruiting a representative sample of patients (from multiple, international sites if possible), making the trial intervention similar to routine clinical practice.
Representativeness of participants
The paucity of appropriate studies found in the literature search meant that the population represented in the meta-analysis is more limited than the study title suggests. All three trials included under-18s only from the USA. The studies excluded participants with important comorbidities, such as Rett syndrome or fragile-X. Although individuals with attention-deficit hyperactivity disorder (ADHD) were not explicitly excluded, to participate in the study no other psychotropics, including stimulants, could be used. ADHD is a common comorbidity with autism (Stevens Reference Stevens, Peng and Barnard-Brak2016). There is some evidence that aripiprazole alone may be effective in the treatment of ADHD (Findling Reference Findling, Mankoski and Timko2008; Zeni Reference Zeni, Tramontina and Ketzer2009) and therefore the reduction in symptoms could be due to aripiprazole's action in treating ADHD. One also has to question whether this US population in particular would have access to treatment for ADHD, on cost grounds. It may be that the presenting complaint of some study participants was ADHD, but that their parents opted to enrol them in the study to treat ‘hyperactivity’ and ‘irritability’ because they were unable to afford treatment for ADHD. Study populations were disproportionately Black in all three trials, which makes it more plausible that these research trials might have served as a substitute for routine healthcare, as access to good-quality healthcare is poorer for Black children with autism in the USA (Magaña Reference Magaña, Parish and Rose2012). Further, all three studies were US-based. In the USA, there is a tendency to diagnose autism at higher rates than in the UK (Bougeard Reference Bougeard, Picarel-Blanchot and Schmid2021), and psychotropic medication – particularly antipsychotics – seem to be used more frequently (Houghton Reference Houghton, Ong and Bolognani2017). In the UK, the first-line treatment for challenging behaviour in autism would be behavioural or environmental interventions, and antipsychotics would only be considered when these had failed (NICE 2012, 2013). The Cochrane authors did not restrict their search to studies of patients who had tried and failed to respond to psychosocial or behavioural interventions, which is the context in which aripiprazole would be used in the UK.
Duration of treatment
None of the included studies extended past 16 weeks, with treatment of >12 weeks being defined as long-term treatment in the discontinuation study (Findling Reference Findling, Short and Leskovec2014). In clinical practice it is not uncommon to see patients on antipsychotics for challenging behaviour for years. In the longer-term trial (Findling Reference Findling, Short and Leskovec2014), no significant difference was noted between aripiprazole and placebo by the study end-point. The authors suggested that prescription of aripiprazole should therefore be regularly reviewed, and discontinuation considered (Findling Reference Findling, Short and Leskovec2014; Hirsch Reference Hirsch and Pringsheim2016). However, rather than ‘regular review’, it might be more appropriate to recommend discontinuation by default at 16 weeks, after which there is evidence of no benefit.
Conclusions
This Cochrane Review illustrates the pitfalls of a protocolised approach to evidence synthesis. The Cochrane methodology was followed meticulously. Every effort was made to conduct an unbiased literature search and to assess the risk of bias of the trials. Statistics were robustly computed. At the end of this process, it was suggested that aripiprazole has a role in treating ‘irritability’ in under-18s with autism, based on three trials sponsored and conducted by the manufacturer of said drug, in the same academic centre. Examining the data in detail shows that the side-effects (such as extrapyramidal side-effects, weight gain and sedation) were more consistently reported, with larger effect sizes, than the intended benefits. Rather than criticising the methods of the meta-analysis itself, it is arguable that data were pooled and meta-analysed despite the limited results retrieved. This meta-analysis demonstrates that the evidence base for the use of aripiprazole in autism is limited to a particular patient group not representative of the UK population and is potentially tainted by the involvement of a pharmaceutical company in all stages of the research. UK psychiatrists, however, are likely to continue to use aripiprazole for challenging behaviour in people with autism where other measures have failed, as there are few treatment options for this group that are supported by high-quality evidence.
Author contributions
Both A.B. and H.F. were equally involved in the preparation of the manuscript and production of the final article. The views expressed here are the views of the authors and not necessarily those of the National Health Service.
Funding
This article received no specific grant from any funding agency, commercial or not-for-profit sectors.
Declaration of interest
None.
eLetters
No eLetters have been published for this article.