Risk assessment and evidence-based medicine

The article by Roychowdhury & Adshead starts to place violence risk assessment in the context of medical care.1 Although this is welcome, their partial defence of risk assessment in general, and of structured professional judgement in particular, is based on some significant distortions. 
 
The first distortion is the gross overestimation of the power of risk assessment to discriminate between low-risk and high-risk people. The authors present a contingency table that they imagine shows the ‘potential’ outcomes of a violence risk assessment (Table 2). Using their tabulated data, a diagnostic odds ratio for risk assessment can be calculated to be 81, indicating that the risk of violence in the high-risk group (50%) is hugely higher than in the low-risk group (1.2%). These figures are totally unrealistic. In fact, the diagnostic odds ratio of violence risk assessment in replication studies was recently estimated by meta-analysis2 to be 3. Roychowdhury & Adshead overestimate the discriminating power of risk assessment by 27 times. Moreover, even an unrealistically powerful risk assessment with diagnostic odds of 16 is of little or no value because of failure to detect potential violence in the low-risk group and the large proportion of false positives in the high-risk group.3 
 
The second distortion relates to the underestimation of the precision of medical tests. In fact, the authors seem to have had difficulty finding any medical test with diagnostic odds that they could compare to a violence risk assessment. Instead they chose to compare two medical treatments. They argue that the high number-needed-to-treat as a result of a violence risk assessment is acceptable in psychiatry because in cardiology the number of bypass grafts needed to prevent one fatal outcome has been calculated to be 53.3 However, the meta-analysis they derived this figure from compared coronary bypass surgery to angioplasty - both of which are highly efficacious treatments for angina.3 In reality, medical tests that are used to diagnose conditions with serious implications for the patient are very accurate - biopsy is an excellent indicator of cancer and an angiogram a good indicator of coronary heart disease. 
 
Despite these limitations, I support the authors’ general idea of viewing risk assessment as a medical procedure. I would go further: surely violence risk assessment should be judged by the standards of evidence-based medicine. The real questions then become: (1) are there any rational interventions that can be justified in terms of cost and benefit that might reduce violence among high-risk patients (many of whom will not be violent) and yet should not be offered to low-risk patients (who commit as many or even the majority of acts of violence); and (2) is there evidence that shifting treatment resources from low-risk to high-risk people can, in any way, reduce overall levels of harm? 
 
The answer to both these questions is no.4,5 There is no doubt that medical diagnostic tests serve as a good basis for medical treatment and that medical and surgical treatment can save lives. It is simply disingenuous to suggest that the same can be said of violence risk assessment.


Risk assessment and evidence-based medicine
The article by Roychowdhury & Adshead starts to place violence risk assessment in the context of medical care. 1 Although this is welcome, their partial defence of risk assessment in general, and of structured professional judgement in particular, is based on some significant distortions.
The first distortion is the gross overestimation of the power of risk assessment to discriminate between low-risk and high-risk people.The authors present a contingency table that they imagine shows the 'potential' outcomes of a violence risk assessment (Table 2).Using their tabulated data, a diagnostic odds ratio for risk assessment can be calculated to be 81, indicating that the risk of violence in the high-risk group (50%) is hugely higher than in the low-risk group (1.2%).These figures are totally unrealistic.In fact, the diagnostic odds ratio of violence risk assessment in replication studies was recently estimated by meta-analysis 2 to be 3. Roychowdhury & Adshead overestimate the discriminating power of risk assessment by 27 times.Moreover, even an unrealistically powerful risk assessment with diagnostic odds of 16 is of little or no value because of failure to detect potential violence in the low-risk group and the large proportion of false positives in the high-risk group. 3he second distortion relates to the underestimation of the precision of medical tests.In fact, the authors seem to have had difficulty finding any medical test with diagnostic odds that they could compare to a violence risk assessment.Instead they chose to compare two medical treatments.They argue that the high number-needed-to-treat as a result of a violence risk assessment is acceptable in psychiatry because in cardiology the number of bypass grafts needed to prevent one fatal outcome has been calculated to be 53. 3However, the meta-analysis they derived this figure from compared coronary bypass surgery to angioplasty -both of which are highly efficacious treatments for angina. 3In reality, medical tests that are used to diagnose conditions with serious implications for the patient are very accurate -biopsy is an excellent indicator of cancer and an angiogram a good indicator of coronary heart disease.
Despite these limitations, I support the authors' general idea of viewing risk assessment as a medical procedure.I would go further: surely violence risk assessment should be judged by the standards of evidence-based medicine.The real questions then become: (1) are there any rational interventions that can be justified in terms of cost and benefit that might reduce violence among high-risk patients (many of whom will not be violent) and yet should not be offered to low-risk patients (who commit as many or even the majority of acts of violence); and ( 2) is there evidence that shifting treatment resources from low-risk to high-risk people can, in any way, reduce overall levels of harm?
The answer to both these questions is no. 4,5There is no doubt that medical diagnostic tests serve as a good basis for medical treatment and that medical and surgical treatment can save lives.It is simply disingenuous to suggest that the same can be said of violence risk assessment.
Declaration of interest: M.L. has provided expert evidence in matters relating to risk assessment. of some of the limitations of the trial, but are surprised that claims are still being made that the study demonstrates that CTOs do not achieve their principle purpose of reducing relapse and readmission. 2magine a hypothetical RCT comparing medication with placebo.The trial would be powered based on estimated effect size and its duration would be based on expected time for response.If, in this scenario, 25% of those in the placebo arm had inadvertently been given the active drug, and if the duration of the study had been only a third of that planned, it would be inconceivable that the investigators would claim a negative result proved the drug ineffective.Yet this is analogous to what has taken place with OCTET.
In OCTET, median length of compulsion in the community was 183 days in the CTO group v. 8 days in the Section 17 group.Although this seems to indicate that it was a trial of people who were largely either subject to long periods of community compulsion (CTO group) or only a few days of compulsion (Section 17 group), a more detailed examination brings this into question.Almost 25% of the Section 17 group were still subject to compulsion by the end of the study, and the mean length of compulsion in this group was 46 days.In the CTO group, only 50% were subject to compulsion by the end of the study, with a mean length under compulsion of 170 days.This has two main implications.
First, the difference in mean length of compulsion between the CTO group and the Section 17 group was only 125 days, or a little over 4 months.It is questionable whether this is sufficient time for any benefits of CTOs to become apparent, and presumably the initial intention had been to compare 12 months in each arm.
Second, in effect, a quarter of the control group were receiving the same type of intervention as the CTO group throughout the course of the study.Any possible benefit in the CTO group would have been offset by the same effects in a large number of control subjects, leading to a large reduction in the power of the study and to type 2 error.The sensitivity analysis does nothing to address this loss of power.We contend that given these problems, in conjunction with the broader issues of recruitment and selection, 3 it is not possible to claim that OCTET demonstrates CTOs to be ineffective.The OCTET trial, community treatment orders and evidence-based practice Based on the findings of the OCTET study, 1 Burns & Molodynski reject observations of consultants who reported directly observable benefits from community treatment orders (CTOs).They argue that it is not possible to 'see with one's own eyes' a probabilistic outcome that takes months to manifest itself.This is a false analogy.In a subgroup of patients, CTOs result in a striking improvement in treatment adherence: if the CTO is lifted, patients discontinue treatment; re-implement the CTO (following relapse and re-hospitalisation) and treatment adherence is achieved again.In such cases, clinicians are able to 'see' the effect of CTOs on treatment adherence and reasonably expect improved clinical outcomes in the longer term.With such a dramatic response (treatment adherence) to the intervention (CTO), it would be scientifically unnecessary, 2 and ethically unacceptable, to refer patients to a randomised controlled trial (RCT).
A number of previous reports have highlighted the potentially detrimental flaws in the methodology of the OCTET, 3,4 which could explain the apparent paradox between the naturalistic observational studies that have shown significant benefit from CTOs, 5 and the negative findings of the OCTET.
Take the scenario of a young man with chronic schizophrenia, who attends the psychiatric out-patient department escorted by his carer.He has a long history of nonadherence to treatment, as well as multiple formal admissions.The patient is known to discontinue treatment immediately after discharge from hospital, invariably leading to rapid relapse and hospitalisation.Since discharge from hospital on CTO 3 months earlier, his mental stability has been maintained and he has been accepting his fortnightly antipsychotic depot injections.His positive psychotic symptoms are minimal.He has become more sociable and has applied for a part-time college course.The psychiatrist tells the patient and his carer that he is going to lift the CTO.To his dismay, the carer asks the psychiatrist 'Have you not seen with your own eyes that the CTO works?'The psychiatrist replies, 'Yes I have, but an RCT says this could not have been possible'.Would this be evidence-based practice?