Skip to main content

Are we failing clinical trials? A case for strong aggregate outcomes

  • D. W. Joyce (a1), D. K. Tracy (a1) (a2) and S. S. Shergill (a1)

Clinical trials in psychiatry inherit methods for design and statistical analysis from evidence-based medicine. However, trials in other clinical disciplines benefit from a more specific relationship between instruments that measure disease state (e.g. biomarkers, clinical signs), the underlying pathology and diagnosis such that primary outcomes can be readily defined. Trials in psychiatry use diagnosis (i.e. a categorical label for a syndrome) as a proxy for the underlying disorder, and outcomes are defined, for example, as a percentage change in a univariate total score on some clinical instrument. We label this approach to defining outcomes weak aggregation of disease state. Univariate measures are necessary, because statistical methodology is both tractable and well-developed for scalar outcomes, but we show that weak aggregate approaches do not capture disease state sufficiently, potentially leading to loss of information about response to intervention. We demonstrate how multivariate disease state can be captured using geometric concepts of spaces defined over routine clinical instruments, and show how clinically meaningful disease states (e.g. representing different profiles of symptoms, recovery or remission) can be defined as prototypes (geometric locations) in these spaces. Then, we show how to derive univariate (scalar) measures, which capture patient's relationships to these prototypes and argue these represent strong aggregates of disease state that may be a better basis for outcome measures. We demonstrate our proposal using a large publically available dataset. We conclude by discussing the impact of strong aggregates for analyses in traditional and novel trial designs.

Corresponding author
*Address for correspondence: D. W. Joyce, Cognition Schizophrenia and Imaging Laboratory, Department of Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King's College London, De Crespigny Park, London SE5 8AF, PO63, UK. (Email:
Hide All
Andreasen, NC, Carpenter, WT, Kane, JM, Lasser, RA, Marder, SR and Weinberger, DR (2005) ‘Remission in Schizophrenia: Proposed Criteria and Rationale for Consensus’, Am J Psychiatry, 162(3), pp. 441449.
Annen, S, Roser, P and Brüne, M (2012) ‘Nonverbal behavior during clinical interviews: similarities and dissimilarities among schizophrenia, mania, and depression.’, The Journal of nervous and mental disease, 200, pp. 2632.
Barnow, S, Arens, EA, Sieswerda, S, Dinu-Biringer, R, Spitzer, C and Lang, S (2010) ‘Borderline personality disorder and psychosis: A review’, Current Psychiatry Reports, 12, pp. 186195.
Berkowitz, RL, Patel, U, Ni, Q, Parks, JJ and Docherty, JP (2012) ‘The impact of the clinical antipsychotic trials of intervention effectiveness (CATIE) on prescribing practices: An analysis of data from a large midwestern state’, Journal of Clinical Psychiatry, 73, pp. 498503.
Craddock, N, O'Donovan, MC and Owen, MJ (2009) ‘Psychosis genetics: Modeling the relationship between schizophrenia, bipolar disorder and mixed (or “schizoaffective”) psychoses’, Schizophrenia Bulletin, 35(3), pp. 482490.
Cuthbert, BN and Insel, TR (2013) ‘Toward the future of psychiatric diagnosis: the seven pillars of RDoC.’, BMC medicine, 11, p. 126.
Demjaha, A, Morgan, K, Morgan, C, Landau, S, Dean, K, Reichenberg, A, Sham, P, Fearon, P, Hutchinson, G, Jones, PB, Murray, RM and Dazzan, P (2009) ‘Combining dimensional and categorical representation of psychosis: the way forward for DSM-V and ICD-11?’, Psychological medicine, 39, pp. 19431955.
Demjaha, A, MacCabe, JH and Murray, RM (2012) ‘How genes and environmental factors determine the different neurodevelopmental trajectories of schizophrenia and bipolar disorder’, Schizophrenia Bulletin, 38, pp. 209214.
Frank, E, Prien, RF, Jarrett, RB, Keller, MB, Kupfer, DJ, Lavori, PW, Rush, AJ and Weissman, MM (1991) ‘Conceptualization and rationale for consensus definitions of terms in major depressive disorder. Remission, recovery, relapse, and recurrence.’, Archives of general psychiatry, 48(9), pp. 851–5.
Glaser, JP, Van Os, J, Thewissen, V and Myin-Germeys, I (2010) ‘Psychotic reactivity in borderline personality disorder’, Acta Psychiatrica Scandinavica, 121, pp. 125134.
Green, MF (2006) ‘Cognitive impairment and functional outcome in schizophrenia and bipolar disorder.’, The Journal of clinical psychiatry, 67, pp. 38.
Insel, T, Cuthbert, B, Garvey, M, Heinssen, R, Pine, DS, Quinn, K, Sanislow, C and Wang, P (2010) ‘Research Domain Criteria (RDoC): Toward a new classification framework for research on mental disorders’, American Journal of Psychiatry, 167, pp. 748751.
Jabben, N, Arts, B, Krabbendam, L and Van Os, J (2009) ‘Investigating the association between neurocognition and psychosis in bipolar disorder: Further evidence for the overlap with schizophrenia’, Bipolar Disorders, 11, pp. 166177.
Jablensky, A (2006) ‘Subtyping schizophrenia: implications for genetic research.’, Molecular psychiatry, 11(9), pp. 815–36.
Jakubovski, E, Carlson, JP and Bloch, MH (2015) ‘Prognostic Subgroups for Remission, Response, and Treatment Continuation in the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) Trial’, The Journal of Clinical Psychiatry, 76(11), pp. 15351545.
Joyce, DW, Kehagia, AA, Tracy, DK, Proctor, J and Shergill, SS (2017) ‘Realising stratified psychiatry using multidimensional signatures and trajectories’, Journal of Translational Medicine. BioMed Central, 15(1), p. 15.
Kay, SR, Fiszbein, A and Opler, L (1987) ‘The positive and negative syndrome scale (PANSS) for schizophrenia.’, Schizophrenia bulletin, 13, pp. 261–76.
Kendler, KS (2016) ‘The nature of psychiatric disorders.’, World psychiatry. World Psychiatric Association, 15(1), pp. 512.
Keshavan, MS, Morris, DW, Sweeney, JA, Pearlson, G, Thaker, G, Seidman, LJ, Eack, SM and Tamminga, C (2011) ‘A dimensional approach to the psychosis spectrum between bipolar disorder and schizophrenia: The Schizo-Bipolar Scale’, Schizophrenia Research, 133, pp. 250254.
Kohonen, T (1995) Self-Organizing Maps. Berlin: Springer.
Krzanowski, WJ (2000) Principles of Multivariate Analysis. A User's Perspective, Oxford Statistical Science Series No. 23.
Leucht, S, Davis, JM, Engel, RR, Kissling, W and Kane, JM (2009) ‘Definitions of response and remission in schizophrenia: recommendations for their use and their presentation’, Acta Psychiatrica Scandinavica, 119(438), pp. 714.
Lichtenstein, P, Yip, BH, Björk, C, Pawitan, Y, Cannon, TD, Sullivan, PF and Hultman, CM (2009) ‘Common genetic determinants of schizophrenia and bipolar disorder in Swedish families: a population-based study.’, Lancet, 373, pp. 234239.
Lindenmayer, JP, Grochowski, S, Hyman, RB, Powchik, P and Davidson, M (1995) ‘Five factor model of schizophrenia: replication across samples.’, Schizophrenia research. Netherlands, Amsterdam, 14(3), pp. 229–34.
Marquand, AF, Wolfers, T, Mennes, M, Buitelaar, J and Beckmann, CF (2016) ‘Beyond Lumping and Splitting: A Review of Computational Approaches for Stratifying Psychiatric Disorders’, Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 1(5), pp. 433447.
McCullagh, P and Nelder, JA (1989) Generalized Linear Models. Second Edi. Chapman & Hall/CRC.
Moore, A, Derry, S, Eccleston, C and Kalso, E (2013) ‘Expect analgesic failure; pursue analgesic success’, British Medical Journal, 346(May), pp. 26902690.
Moore, R, Derry, S, McQuay, HJ, Straube, S, Aldington, D, Wiffen, P, Bell, RF, Kalso, E and Rowbotham, MC (2010) ‘Clinical effectiveness: An approach to clinical trial design more relevant to clinical practice, acknowledging the importance of individual differences’, Pain. International Association for the Study of Pain, 149(2), pp. 173176.
Morris, SE and Cuthbert, BN (2012) ‘Research domain criteria: Cognitive systems, neural circuits, and dimensions of behavior’, Dialogues in Clinical Neuroscience, 14, pp. 2937.
Nishizono-Maher, A, Ikuta, N, Ogiso, Y, Moriya, N, Miyake, Y and Minakawa, K (1993) ‘Psychotic symptoms in depression and borderline personality disorder’, Journal of Affective Disorders, 28, pp. 279285.
Overall, JE and Gorham, DR (1962) ‘THE BRIEF PSYCHIATRIC RATING SCALE’, Psychological Reports. Ammons Scientific, 10(3), pp. 799812.
Purcell, SM, Wray, NR, Stone, JL, Visscher, PM, O'Donovan, MC, Sullivan, PF and Sklar, P (2009) ‘Common polygenic variation contributes to risk of schizophrenia and bipolar disorder.’, Nature, 460, pp. 748752.
Reininghaus, U, Böhnke, JR, Hosang, G, Farmer, A, Burns, T, McGuffin, P, Bentall, RP, Insel, TR, Adam, D, First, MB, Reed, GM, Hyman, SE, Saxena, S, Bebbington, P, Caspi, AHR, Belsky, DW, Goldman-Mellor, SJ, Harrington, H, Israel, S, Meier, MH, Carpenter, WT, Bustillo, JR, Thaker, GK, Os, J van, Krueger, RF, Green, MJ, Craddock, N, Owen, MJ, Cardno, AG, Rijsdijk, FV, Sham, PC, Murray, RM, McGuffin, P, Lichtenstein, P, Yip, BH, Bjork, C, Pawitan, Y, Cannon, TD, Sullivan, PF, Brown, AS, Os, J van, Driessens, C, Hoek, HW, Susser, ES, Clarke, MC, Harley, M, Cannon, M, Matheson, SL, Shepherd, AM, Pinchbeck, RM, Laurens, KR, Carr, VJ, Henquet, C, Krabbendam, L, Graaf, R, Have, M, Os, J van, Henquet, C, Murray, R, Linszen, D, Os, J van, Heinz, A, Deserno, L, Reininghaus, U, Kirkbride, JB, Errazuriz, A, Croudace, TJ, Morgan, C, Jackson, D, Boydell, J, Os, J van, Kapur, S, Reininghaus, U, Priebe, S, Bentall, RP, Russo, M, Levine, SZ, Demjaha, A, Forti, MD, Bonaccorso, S, Fearon, P, McGuffin, P, Farmer, A, Harvey, I, Burns, T, Creed, F, Fahy, T, Thompson, S, Tyrer, P, White, I, Cohen-Woods, S, Craig, I, Gaysina, D, Gray, J, Gunasinghe, C, Craddock, N, Spitzer, RL, Endicott, J, Robins, E, Rucker, J, Newman, S, Gray, J, Gunasinghe, C, Broadbent, M, Brittain, P, Chalmers, RP, Kass, RW, Wasserman, L, Peterson, LE, Coleman, MA, Kohavi, R, Efron, BG, Gong, G, Böhnke, JR, Croudace, TJ, Cardno, AG, Jones, LA, Murphy, KC, Murphy, KC, Asherson, P, Scott, LC, Markon, KE, Reininghaus, U, Morgan, C, Simpson, J, Dazzan, P, Morgan, K, Doody, G, Morgan, C, Reininghaus, U, Fearon, P, Hutchinson, G, Morgan, K, Dazzan, P, Reininghaus, U, Craig, T, Fisher, H, Hutchinson, G, Fearon, P, Morgan, K, Tamminga, CA, Ivleva, EI, Keshavan, MS, Pearlson, GD, Clementz, BA and Witte, B (2016) ‘Evaluation of the validity and utility of a transdiagnostic psychosis dimension encompassing schizophrenia and bipolar disorder.’, The British journal of psychiatry: the journal of mental science, 209(2), pp. 107–13.
Ripke, S, Neale, BM, Corvin, A, Walters, JTR, Farh, K-H, Holmans, Pa, Lee, P, Bulik-Sullivan, B, Collier, Da, Huang, H, Pers, TH, Agartz, I, Agerbo, E, Albus, M, Alexander, M, Amin, F, Bacanu, Sa, Begemann, M, Belliveau, Ra Jr, Bene, J, Bergen, SE, Bevilacqua, E, Bigdeli, TB, Black, DW, Bruggeman, R, Buccola, NG, Buckner, RL, Byerley, W, Cahn, W, Cai, G, Campion, D, Cantor, RM, Carr, VJ, Carrera, N, Catts, SV, Chambert, KD, Chan, RCK, Chen, RYL, Chen, EYH, Cheng, W, Cheung, EFC, Ann Chong, S, Robert Cloninger, C, Cohen, D, Cohen, N, Cormican, P, Craddock, N, Crowley, JJ, Curtis, D, Davidson, M, Davis, KL, Degenhardt, F, Del Favero, J, Demontis, D, Dikeos, D, Dinan, T, Djurovic, S, Donohoe, G, Drapeau, E, Duan, J, Dudbridge, F, Durmishi, N, Eichhammer, P, Eriksson, J, Escott-Price, V, Essioux, L, Fanous, AH, Farrell, MS, Frank, J, Franke, L, Freedman, R, Freimer, NB, Friedl, M, Friedman, JI, Fromer, M, Genovese, G, Georgieva, L, Giegling, I, Giusti-Rodríguez, P, Godard, S, Goldstein, JI, Golimbet, V, Gopal, S, Gratten, J, de Haan, L, Hammer, C, Hamshere, ML, Hansen, M, Hansen, T, Haroutunian, V, Hartmann, AM, Henskens, Fa, Herms, S, Hirschhorn, JN, Hoffmann, P, Hofman, A, Hollegaard, MV, Hougaard, DM, Ikeda, M, Joa, I, Julià, A, Kahn, RS, Kalaydjieva, L, Karachanak-Yankova, S, Karjalainen, J, Kavanagh, D, Keller, MC, Kennedy, JL, Khrunin, A, Kim, Y, Klovins, J, Knowles, Ja, Konte, B, Kucinskas, V, Ausrele Kucinskiene, Z, Kuzelova-Ptackova, H, Kähler, AK, Laurent, C, Lee Chee Keong, J, Hong Lee, S, Legge, SE, Lerer, B, Li, M, Li, T, Liang, K-Y, Lieberman, J, Limborska, S, Loughland, CM, Lubinski, J, Lönnqvist, J, Macek, M Jr, Magnusson, PKE, Maher, BS, Maier, W, Mallet, J, Marsal, S, Mattheisen, M, Mattingsdal, M, McCarley, RW, McDonald, C, McIntosh, AM, Meier, S, Meijer, CJ, Melegh, B, Melle, I, Mesholam-Gately, RI, Metspalu, A, Michie, PT, Milani, L, Milanova, V, Mokrab, Y, Morris, DW, Mors, O, Murphy, KC, Murray, RM, Myin-Germeys, I, Müller-Myhsok, B, Nelis, M, Nenadic, I, Nertney, Da, Nestadt, G, Nicodemus, KK, Nikitina-Zake, L, Nisenbaum, L, Nordin, A, O'Callaghan, E, O'Dushlaine, C, O'Neill, FA, Oh, S-Y, Olincy, A, Olsen, L, Van Os, J, Endophenotypes International Consortium P, Pantelis, C, Papadimitriou, GN, Papiol, S, Parkhomenko, E, Pato, MT, Paunio, T, Pejovic-Milovancevic, M, Perkins, DO, Pietiläinen, O, Pimm, J, Pocklington, AJ, Powell, J, Price, A, Pulver, AE, Purcell, SM, Quested, D, Rasmussen, HB, Reichenberg, A, Reimers, Ma, Richards, AL, Roffman, JL, Roussos, P, Ruderfer, DM, Salomaa, V, Sanders, AR, Schall, U, Schubert, CR, Schulze, TG, Schwab, SG, Scolnick, EM, Scott, RJ, Seidman, LJ, Shi, J, Sigurdsson, E, Silagadze, T, Silverman, JM, Sim, K, Slominsky, P, Smoller, JW, So, H-C, Spencer, CCA, Stahl, EA, Stefansson, H, Steinberg, S, Stogmann, E, Straub, RE, Strengman, E, Strohmaier, J, Scott Stroup, T, Subramaniam, M, Suvisaari, J, Svrakic, DM, Szatkiewicz, JP, Söderman, E, Thirumalai, S, Toncheva, D, Tosato, S, Veijola, J, Waddington, J, Walsh, D, Wang, D, Wang, Q, Webb, BT, Weiser, M, Wildenauer, DB, Williams, NM, Williams, S, Witt, SH, Wolen, AR, Wong, EHM, Wormley, BK, Simon Xi, H, Zai, CC, Zheng, X, Zimprich, F, Wray, NR, Stefansson, K, Visscher, PM, Trust Case-Control Consortium, W, Adolfsson, R, Andreassen, Oa, Blackwood, DHR, Bramon, E, Buxbaum, JD, Børglum, AD, Cichon, S, Darvasi, A, Domenici, E, Ehrenreich, H, Esko, T, Gejman, PV, Gill, M, Gurling, H, Hultman, CM, Iwata, N, Jablensky, AV, Jönsson, EG, Kendler, KS, Kirov, G, Knight, J, Lencz, T, Levinson, DF, Li, QS, Liu, J, Malhotra, AK, McCarroll, Sa, McQuillin, A, Moran, JL, Mortensen, PB, Mowry, BJ, Nöthen, MM, Ophoff, Ra, Owen, MJ, Palotie, A, Pato, CN, Petryshen, TL, Posthuma, D, Rietschel, M, Riley, BP, Rujescu, D, Sham, PC, Sklar, P, St Clair, D, Weinberger, DR, Wendland, JR, Werge, T, Daly, MJ, Sullivan, PF and O'Donovan, MC (2014) ‘Biological insights from 108 schizophrenia-associated genetic loci’, Nature, 511, pp. 421427.
Roweis, ST and Saul, LK (2000) ‘Nonlinear Dimensionality Reduction by Locally Linear Embedding’, Science, 290(5500), pp. 23232326.
Schork, NJ (2015) ‘Time for one-person trials’, Nature, 520(7549), pp. 609611.
Schroeder, K, Fisher, HL and Schäfer, I (2013) ‘Psychotic symptoms in patients with borderline personality disorder: prevalence and clinical management.’, Current opinion in psychiatry, 26, pp. 113–9.
Schumann, G, Binder, EB, Holte, A, de Kloet, E.R, Oedegaard, KJ, Robbins, TW, Walker-Tilley, TR, Bitter, I, Brown, VJ, Buitelaar, J, Ciccocioppo, R, Cools, R, Escera, C, Fleischhacker, W, Flor, H, Frith, CD, Heinz, A, Johnsen, E, Kirschbaum, C, Klingberg, T, Lesch, KP, Lewis, S, Maier, W, Mann, K, Martinot, JL, Meyer-Lindenberg, A, Müller, CP, Müller, WE, Nutt, DJ, Persico, A, Perugi, G, Pessiglione, M, Preuss, UW, Roiser, JP, Rossini, PM, Rybakowski, JK, Sandi, C, Stephan, KE, Undurraga, J, Vieta, E, van der Wee, N, Wykes, T, Haro, JM and Wittchen, HU (2014) ‘Stratified medicine for mental disorders’, European Neuropsychopharmacology, 24(1), pp. 550.
Strang, G (2004) Linear Algebra and Its Applications. 4th edn. WB Saunders. doi: 10.2307/2003783.
Stroup, T.S, Mcevoy, JP, Swartz, MS, Byerly, MJ, Qlick, ID, Canive, JM, Mcqee, MF, Simpson, QM, Stevens, MC and Lieberman, JA (2003) ‘The National Institute of Mental Health Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) project: schizophrenia trial design and protocol development.’, Schizophrenia bulletin, 29(11)
Stroup, TS, Mcevoy, JP, Swartz, MS, et The National Institute of Mental Health Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) project: schizophrenia trial design and protocol development. Schizophr Bull. 2003;29(1):1531.
Tenenbaum, JB, de Silva, V and Langford, JC (2000) ‘A global geometric framework for nonlinear dimensionality reduction.’, Science, 290(5500), pp. 2319–23.
Toh, WL, Thomas, N and Rossell, SL (2015) ‘Auditory verbal hallucinations in bipolar disorder (BD) and major depressive disorder (MDD): A systematic review’, Journal of Affective Disorders, 184, pp. 1828.
Walker, J, Curtis, V and Murray, RM (2002) ‘Schizophrenia and bipolar disorder: similarities in pathogenic mechanisms but differences in neurodevelopment.’, International clinical psychopharmacology, 17 Suppl 3, pp. S11S19.
Wallwork, RS, Fortgang, R, Hashimoto, R, Weinberger, DR and Dickinson, D (2012) ‘Searching for a consensus five-factor model of the Positive and Negative Syndrome Scale for schizophrenia.’, Schizophrenia research. NIH Public Access, 137(1–3), pp. 246–50.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Psychological Medicine
  • ISSN: 0033-2917
  • EISSN: 1469-8978
  • URL: /core/journals/psychological-medicine
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Type Description Title
Supplementary materials

Joyce supplementary material
Joyce supplementary material

 Word (752 KB)
752 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed