Why randomise?

The choice faced by those designing studies to address ‘real world’ problems is either to describe outcomes in patients treated in completely naturalistic settings, without using randomisation (observational studies), or to try to adapt the traditional RCT to retain some of its key advantages, but emphasise external validity.

The problems inherent in unrandomised studies are demonstrated below. Brugha et al(1992) assessed the outcome of patients with depression who were prescribed antidepressants in psychiatric out-patient departments and compared this with the outcome of patients who had depression but did not receive a prescription. This observational study found that there was no difference between the two groups. Receiving a prescription for an antidepressant did not seem to make patients better. The authors suggested that this showed that in real life practice, antidepressants were less effective than would have been expected on the basis of RCTs. However, an alternative view is that the two groups being compared were different. The authors were able to compare, in terms of the severity and duration of symptoms, those who received antidepressants with those who did not, and they found no differences. However, it is unlikely that the doctor's decision to prescribe an antidepressant was completely arbitrary. There are many subtle (and not-so-subtle) factors that determine whether a patient with depression is given a prescription. The prescribers were probably picking up on subtle factors that might have played a major role in determining prognosis. If the group who received an antidepressant had a worse prognosis, this could have disguised the benefits of antidepressant treatment.

Another observational study sought to determine whether patients with depression treated in the USA, who were first seen by psychiatrists, had outcomes different from those seen by primary care physicians (Reference Simon, von Korff and RutterSimon et al, 2001). The study was set up in a health maintenance organisation and patients whose depression was first managed by primary care physicians (GPs) were compared with another group who were first treated by psychiatrists. Patients were followed up and outcomes, both in terms of process of care and clinical state, were compared between the two groups. The study found that some aspects of the process of care were very similar – both GPs and psychiatrists often failed to use adequate dosages of antidepressants. Clinical outcomes in terms of remission from symptoms of depression were also similar.

Again, interpreting this study is difficult, for similar reasons. It could be argued that, because there were no differences in outcome between patients seen by their GPs and those seen by psychiatrists, either professional would be as good as the other in managing depression (in effect, this would be to argue that GPs were better, because they would also presumably have been the cheaper option). However, it seems probable that the patients who saw a psychiatrist had more complex or severe problems. The study did, indeed, find that members of this group were more disabled by their depressive symptoms than the patients visiting GPs, even though levels of symptoms were similar. In other words, there may be real advantages to being seen by a psychiatrist but these were hidden because psychiatrists saw a group with more severe illness.

These examples (and many others) demonstrate the problem of non-randomised evaluations of health care interventions – you are never sure whether you are comparing like with like. In each case, randomisation would have overcome these difficulties. The key advantage of randomisation, which makes it such a powerful tool, is that it rules out confounding (including unknown confounders that had not even occurred to the investigators). Provided that enough patients are randomised, the two groups will be similar in most respects. The rationale for randomisation is, therefore, to balance the two groups receiving treatment as perfectly as possible. It is for this reason that the RCT has achieved its status as the gold standard for assessing treatments.

Generalisability and randomisation: getting the best of both worlds

So far, I have suggested that the traditional RCT is sometimes so far from routine clinical practice that its external validity (or generalisability) is called into question, whereas observational studies that simply assess outcomes under service conditions lack internal validity. The pragmatic RCT aims to bridge the gap between these methodologies. Although there are good reasons for many of the features of typical RCTs (Box 1) they are not necessarily synonymous with good trial design if interventions are too narrowly defined, patient selection is too restrictive or outcomes are of limited interest to clinicians and their patients. Pragmatic RCTs often make alterations to such features (Reference Roland and TorgensenRoland & Torgensen, 1998), and some of these adaptations are described below and in the case studies.

Participants

A key aim for pragmatic RCTs is to reflect the heterogeneity of patients encountered in clinical practice. They aim to keep exclusion criteria to a minimum – comorbid medical conditions are a common feature of common mental disorders in primary care – and researchers may therefore choose to include medically ill patients who might have been excluded in conventional trials. Suicidal ideas are common in depression, so the aim would be to keep patients in the trial, even if these are expressed.

Another aspect relating to the selection of participants for pragmatic RCTs is that they may not be so preoccupied with narrow diagnostic labels. Most GPs are uninterested in the arcane distinctions between depressive episodes; mixed anxiety and depression; and adjustment disorder with depressive features. Such disorders are probably treated in a similar way in clinical practice and they frequently overlap. Thus, pragmatic RCTs may ‘lump’ patients together more than traditional RCTs.

Pragmatic RCTs may also focus on specific clinical groups or presentations that include a wide range of diagnoses. There have been pragmatic RCTs on ‘high utilisers’ of primary care who are distressed (Reference Katon, von Korff and LinKaton et al, 1992); these patients form a recognisable group for GPs, but do not fit neatly into diagnostic categories. A similar approach might define the population to be studied as patients presenting with early-onset psychosis or deliberate self-harm. The presentation, not the diagnosis, is what defines these groups of patients and it is the presentation that leads to specific challenges for service providers.

Treatment and control groups

Pragmatic RCTs are often concerned with complex interventions. These may include approaches to screening to detect common mental disorders, or referring patients to different health professionals. Whereas traditional trials tend to use rigid dosing regimens for drugs, or very specific therapeutic models for psychotherapy, this tends not to be the case in routine clinical practice, and pragmatic RCTs are therefore more flexible in defining the intervention (Box 2). For example, a trial might assess the use of social workers for patients with depression but allow the nature of the social work intervention to vary according to what the social worker saw as the key issues for an individual client. The social work client might receive non-directive counselling, problem-solving or more pragmatic social care (e.g., help with housing problems).

The classic inactive comparison for traditional pharmacological studies is the placebo. This is not explicitly used in clinical practice and under some circumstances pragmatic RCTs may evaluate drug treatment without using placebos. In other words, if the choice in reality is to have an active treatment, or no treatment at all, this should be reflected in pragmatic RCTs. Similarly, classic psychotherapy trials have attempted to control for the non-specific effects of the therapist's time and attention in order to detect specific treatment effects, by giving some form of ‘placebo’ psychotherapy. In the pragmatic RCT there is a tendency to deal with each treatment as a black box – usually, the concern is not to try to understand specific active ingredients within the box.

Another approach for control groups is to compare a new treatment with ‘usual care’ (an approach used in case studies 1 and 2 below). ‘Usual care’ is a difficult term to define because it will depend heavily on the knowledge, skills and resources of the health care professionals delivering it. If there is evidence that usual care, as experienced by most patients in the health service, is in fact sub-optimal care, the trialist may attempt to provide guidance to participating doctors on what usual care should involve.

Tackling patient preference

In clinical practice, the choice of treatment is not simply determined by the doctor. It is usually a two-way process in which the doctor gives advice, but the patient is – with some notable exceptions in psychiatry – expected to exercise judgement to arrive at a preference. The ethical basis of RCTs is that there is a state of equipoise regarding which treatment is effective. If one treatment is proven to be more effective than another, there would be no ethical justification in comparing them. Patients, however, may make decisions about treatments on completely different grounds. Many patients with depression prefer counselling to antidepressants, despite rather sparse data in favour of the former and much more robust evidence in favour of the latter. Some patients would never accept randomisation in the first place because they would not countenance being allocated to a treatment of which they disapproved. Under certain circumstances, this may be a significant proportion of patients.

One way to include data from such patients, and to achieve better external validity, is to perform a patient-preference trial. In this design, all patients are asked to contribute outcome data for the trial, but before allocation they are given a choice to either be randomised or receive a specific treatment. The approach, although attractive, is not without complications. One example (described in more depth in case study 2) is a trial comparing usual care with referral to a counsellor or referral to a clinical psychologist in the management of depression in primary care (Reference Ward, King and LloydWard et al, 2000). Many patients opted for referral and very few opted for usual care. The problem caused by the patient preference arm of the trial was that patients who would have been happy to have been randomised to receive counselling or CBT opted for the patient-preference arm to prevent the risk of being allocated to usual care.

Blinding

Traditional RCTs are typically run double-blind. This means that neither the patient nor the clinician knows to which treatment the patient has been assigned. There are two main reasons for blinding: the first is to prevent the patient's knowledge that he or she has been allocated to a new treatment influence his or her assessment of any clinical improvement. The second reason is to prevent observer bias. This occurs when the researcher's knowledge of the treatment allocated influences (either consciously or unconsciously) his or her appraisal of the patient.

Pragmatic RCTs may not be able to use blinding. For example, when a comparison is being made between receiving treatment for depression from a GP or a psychiatrist, once randomised, it is impossible to blind the patient to his or her allocated group. Thus, the patient cannot be blind to the treatment received. Under such circumstances, it is theoretically possible for the patient's outcome to be assessed by a researcher who remains blind to treatment allocation, but if the researcher has face-to-face contact with the patient, it is likely that he or she will become unblinded (see case study 1 below for an example of when this was possible).

Particular attention, therefore, needs to be paid to two aspects of study design in pragmatic RCTs. First, concealment of allocation during randomisation must be carefully maintained. This refers to how predictable it is that a patient will be allocated to one or other treatment. The amount of attention researchers pay to concealment of allocation strongly indicates the overall quality of RCTs (Reference Schulz, Chalmers and HayesSchulz et al, 1995). Some randomisation techniques allow the researcher to predict accurately the treatment group for patients. For example, where randomisation takes place in ‘random permuted blocks’, batches of patients are randomised in blocks of four or six. If a researcher knows what treatment the last patient was allocated, he or she could predict the treatment to which the next patients would be allocated. Having predicted which treatment the patient will receive, the researcher may decide not to randomise the patient if a poor outcome is expected on this treatment. Everything possible must be done to ensure that allocation is concealed and there is no degree of predictability. One approach is to use an independent researcher to perform the randomisation.

The second aspect of the design, which may require special attention in unblinded studies, is the outcome measures used. In other branches of medicine, ‘hard’ outcomes such as death or cardiovascular events are often used. In psychiatry, outcomes are frequently much more subjective and include symptoms or self-reported quality of life. The problem with such outcomes is that they are particularly open to observer bias. This occurs when the knowledge that a patient has received a specific treatment influences the way the investigator assesses the outcome. For example, when rating symptoms of depression, an assessor who knows that a patient has received treatment in which he or she has considerable faith, may consciously or unconsciously ignore symptoms at follow-up.

The most commonly used outcome scales in traditional RCTs for depression are the Hamilton Rating Scale for Depression (HRSD; Reference HamiltonHamilton, 1960) and the Montgomery-Åsberg Depression Rating Scale (MADRS; Reference Montgomery and ÅsbergMontgomery & Åsberg, 1979). These rating scales are particularly subject to observer bias because they are administered after an unstructured clinical interview. Alternative approaches are structured interviews or questionnaires – these reduce the possibility of observer bias because they are presented to each participant in an identical manner.

Outcomes

Outcome measures should be chosen not just to prevent observer bias, but also to reflect the ‘real world’ concerns of patients, clinicians and policy-makers (Box 3). It is difficult to interpret a three-point reduction in the Hamilton Depression Rating Scale. It is easier to interpret binary outcomes such as recovery or remission, but the definitions of these are frequently arbitrary, such as a 50% reduction in symptoms, or a reduction in symptoms below a threshold.

In pragmatic RCTs, more emphasis is placed on functional outcomes that indicate more than symptomatic improvements. Some examples of relevant outcomes are shown in case study 3. Another consideration is the time spans over which outcomes are measured. Many traditional RCTs assess outcome over a 4- to 8-week period. Given the relapsing and remitting nature of many psychiatric disorders, this is an insufficient time period for many interventions.

The unit of randomisation

The traditional RCT almost always randomises individual patients. However, some interventions in health services research are aimed not at patients but at units within the health service (Reference Gilbody and WhittyGilbody & Whitty, 2002). For example, one approach to the problem of poor recognition and management of common mental disorder in primary care has been to educate GPs. If the intervention (education) is to be aimed at GPs or GP practices, these should be the unit of randomisation (see case study 3 below). Outcomes are then measured on clusters of patients seen by each GP or practice.

The advantage of cluster randomisation is that it allows study of the relevant level of health service. In the example of improving GP knowledge about common mental disorders, it might be feasible to randomise patients to see either a GP trained in the detection of common mental disorders or one who was not trained, but this would not reflect the clinical reality. The intervention is not given at the level of the individual patient but at the level of the doctor.

The main problem with cluster randomisation is that it is cumbersome to perform and analyse. This is mainly because patients within each cluster are not statistically independent of one another. For example, if general practices were randomised some would contain predominantly affluent patients and others predominantly impoverished ones. This intra-cluster correlation reduces the statistical power of the study and means that many clusters often have to be randomised in order to achieve a good balance.