Young adults in a clinical high-risk state for psychosis (CHR) or with recent-onset depression (ROD) represent risk populations for severe mental illness (psychosis or recurrent depression).Reference Hartmann, Nelson, Spooner, Paul Amminger, Chanen and Davey1 Besides some shared symptom phenomenology, these two groups show comparable substantial and persistent deficits in functional outcomes,Reference Piskulic, Liu, Cadenhead, Cannon, Cornblatt and McGlashan2 which account for much of the immense individual and socioeconomic burden of mental disorders. Therefore, personalised prediction of functional outcomes in at-risk populations is a major target for prevention in psychiatry. Aiming at this objective, we recently reported machine learning models with a balanced accuracy (BAC) of up to 83% for the prediction of social and role functioning,Reference Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef and Dwyer3 i.e. the extent to which an individual is able to deal with social interactions and occupational and other demands of daily life respectively. However, as these models combined only data of previous functioning with structural neuroimaging data, an ongoing discussion was stimulated about selection of data included as predictors in such models.Reference Koutsouleris, Upthegrove and Wood4 In particular, conclusions for clinical practice could be biased such that costly diagnostics would be recommended without testing whether more cost-efficient clinical data have a similar predictive potential.Reference Nelson, Yung and McGorry5 The optimal strategy to develop models that are accurate and generalisable but also efficient in a clinical context remains unclear. Detailed phenotyping by clinical data alone might have essential advantages over neuroimaging data owing to cost-efficiency and the potential to inform clinical interventions.Reference Nelson, Yung and McGorry5 Still, adding more and more data to prediction models might reduce generalisability to new patients or settings.Reference Fusar-Poli, Hijazi, Stahl and Steyerberg6 As external performance of psychiatric prediction models is often low and highly heterogeneous,Reference Rosen, Betz, Schultze-Lutter, Chisholm, Haidl and Kambeitz-Ilankovic7 generalisability is crucial in the translation to clinical practice.
To determine the role of predictor parsimony and data type in prediction of psychosocial functioning, we conducted the current study, building on and reviewing our previous report of multi-modal models for prediction of functioning.Reference Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef and Dwyer3 We specifically tested (a) whether detailed clinical phenotyping improves prediction performance compared with parsimonious models based on previous functioning alone and (b) the so far unknown generalisability of prediction models for functional outcome to new patients from different healthcare settings and countries.
Method
For comparability, we used participants (CHR group: n = 114; ROD group: n = 106) from the observational multicentre Personalised Prognostic Tools for Early Psychosis Management (PRONIA) study (German Clinical Trials Register identifier DRKS00005042; for detailed sample description see supplementary Table 1, available at https://doi.org/10.1192/bjp.2021.141) and an analogous analysis approach following our study published in 2018.Reference Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef and Dwyer3 We trained different machine learning models using baseline variables for the prediction of social and role functioning after 1 year as assessed by the Global Functioning Scales: Social and RoleReference Cornblatt, Auther, Niendam, Smith, Zinberg and Bearden8 separately for CHR and ROD individuals.
The first parsimonious model (A) is a replication of a basic model based only on previous functioning measures we reported earlier.Reference Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef and Dwyer3 A second group of models (B) includes n = 176 clinical variables comprising additional aspects of previous functioning and detailed characterisation of disease severity by core psychopathology of the ROD and CHR participants (supplementary Table 2). To identify the most predictive and eliminate non-informative variables, we implemented feature selection based on greedy feature elimination (B1) and greedy feature elimination combined with a principal component analysis (PCA) (B2), consistent with our previous analysis.Reference Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef and Dwyer3 In addition, we implemented two alternative modelling strategies based on an L1-regularised support-vector machine (SVM) algorithm (B3) and based on sparse PCA (B4) that might be more appropriate for clinical data, yielding an informative, parsimonious set of the most predictive features. A third group of models (C1−4) was based on a stacked ensemble model by combining model A with B1–4 respectively. For detailed description of machine learning models, see the methods section in the supplementary material. We assessed the geographical generalisability of the models to patients in the different sites of the PRONIA study using nested leave-site-out cross-validation (LSO-CV). To test the generalisability to new patients, we applied all models to truly unseen data of additional patients (CHR: n = 97; ROD: n = 61) from the PRONIA sample of all original study sites plus three additional study sites that were not part of the training sample (supplementary Table 1). Differences in model performances were determined by Quade's test at the omnibus level, followed by post hoc comparisons with model A.
Ethics approval and consent
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. The multicentre PRONIA study was approved by all local research ethics committees. Written informed consent was obtained from all patients. For minors (defined as participants younger than 18 years), guardians also provided written informed assent.
Results
All models showed robust generalisability, i.e. similar prediction accuracy in the training and independent test sample (Table 1). The different dimensionality reduction methods (B1−4) performed equally well across different data domains and risk populations. For prediction of role functioning, clinical phenotyping (B1−4 and C1−4) yielded significant, clinically relevant improvements (gain in BAC up to 10.9% for ROD participants in the training set and up to 10.0% for ROD participants and 8.3% for CHR participants in the independent test data) compared with models based on functioning measures alone (A). For social functioning, detailed clinical data did not improve prediction in CHR participants and revealed only slight, non-significant improvement for ROD participants (gain in BAC of up to 5.8% in the training set).
Classification performances using leave-site-out cross-validation and validation in an independent sample of machine-learning predictors of global functioning social scale or global functioning role scale outcomes in individuals at clinical high-risk of psychosis and individuals with recent-onset depression
LSO-CV, leave-site-out cross-validation; sens., sensitivity; spec., specificity; BAC, balanced accuracy; PPV, positive predictive value; NPV, negative predictive value; CHR, clinical high-risk of psychosis; PCA, principal component analysis; L1, L1 regularisation; ROD, recent-onset depression.
a. Quade's test for statistical comparison of all models at omnibus level and post hoc comparisons with model A based on folds and repetitions of the outer cycle in the cross-validation process (reported classification performance measures are not weighted by varying size of folds).
Discussion
The present work demonstrates that detailed clinical phenotyping is valuable for prediction of functional outcomes, in particular for role functioning, which might be less biologically determined, influenced by a complex set of factors and more closely connected to the disease trajectory. Prognostication of social functioning, in contrast, did not substantially benefit from additional clinical data beyond previous functioning, and was generally more accurate in CHR participants than in ROD participants. These results are in line with previous findings showing that social impairment was more constant than role functioning across time.Reference Cornblatt, Auther, Niendam, Smith, Zinberg and Bearden8 Furthermore, in CHR individuals deficits in social cognition resulting in social impairment are more persistent,Reference Piskulic, Liu, Cadenhead, Cannon, Cornblatt and McGlashan2 whereas social cognition in people with depression tends to be affected by symptom severity.Reference Weightman, Air and Baune9
Independent of the quantity of included clinical predictors, all models proved robust when applied to new patients. This underlines the reliability and validity of established clinical assessments included into the multi-predictor models for clinical phenotyping. Furthermore, different feature reduction strategies seemed to be equally effective, emphasising that results are primarily determined by included information about symptoms and aspects of functioning, and not by a specific analysis approach.
Compared descriptively with our previously reported models complemented by neuroimaging dataReference Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef and Dwyer3 (average difference in BAC Δ = 0.44; supplementary Table 4), the current results lead to a re-evaluation and suggest no absolute need for an extension of clinical phenotyping by another prognostic data modality such as costlier magnetic resonance imaging (MRI) in early stages of preventive care. Clinical data, such as different aspects of previous functioning and detailed characterisation of disease severity, are commonly collected in clinical contexts, facilitating translation into clinical practice. Moreover, prediction models based on clinical data yield informative insights for efficient targeted interventions to prevent disabilities in psychosocial functioning. However, costlier assessments, such as MRI, might be valuable at later stages in the context of sequential testing, allowing for more precise predictions in participants identified to be at risk for poor outcomes via prediction models based on clinical data.Reference Schmidt, Cappucciati, Radua, Rutigliano, Rocchetti and Dell'Osso10
Given that we showed excellent generalisability of our personalised prognostic models based on easily accessible clinical data for functional outcome in CHR and ROD individuals across geographically and structurally diverse healthcare systems,Reference Koutsouleris, Upthegrove and Wood4 our study represents an important step for clinical application of prognostic models and towards improvement of targeted early intervention and personalised psychiatric care.
Supplementary material
Supplementary material is available online at https://doi.org/10.1192/bjp.2021.141.
Data availability
Supplemental findings supporting this study are available on request from the corresponding author (J.K.). The data are not publicly available owing to Institutional Review Board restrictions, since the participants did not consent to their data being publicly available.
Acknowledgements
We would like to thank the following members of the PRONIA Consortium who conceptualised and designed the PRONIA project, performed the screening, recruitment, rating, examination and follow-up of the study participants and were involved in implementing the examination protocols of the study, setting up its information technology infrastructure, and organising the flow and quality control of the data analysed in this study between the local study sites and the central study database. In the Department of Psychiatry and Psychotherapy, Ludwig-Maximilian-University, Munich, Bavaria, Germany: Dominic Dwyer, PhD, Anne Ruef, PhD, Mark Sen Dong, MSc, Anne Erkens, Eva Gussmann, MSc, Shalaila Haas, PhD, Alkomiet Hasan, MD, Claudius Hoff, MD, Ifrah Khanyaree, BSc, Aylin Melo, MSc, Susanna Muckenhuber-Sternbauer, MD, Janis Kohler, Omer Faruk Ozturk, MD, David Popovic, MD, Adrian Rangnick, BSc, Sebastian von Saldern, MD, Rachele Sanfelici, MSc, Moritz Spangemacher, Ana Tupac, MSc, Maria Fernanda Urquijo, MSc, Johanna Weiske, MSc, Julian Wenzel, MSc, and Antonia Wosgien. At the University of Cologne, North Rhineland–Westphalia, Germany: Karsten Blume, Mauro Seves, MSc, Tanja Pilgram, MSc, Thorsten Lichtenstein, MD, and Christiane Woopen, MD. In the Psychiatric University Hospital, University of Basel, Switzerland: Christina Andreou, MD, PhD, Laura Egloff, PhD, Fabienne Harrisberger, PhD, Claudia Lenz, PhD, Letizia Leanza, MSc, Amatya Mackintosh, MSc, Renata Smieskova, PhD, Erich Studerus, PhD, Anna Walter, MD, and Sonja Widmayer, MSc. At the Institute for Mental Health, University of Birmingham, UK: Katharine Chisholm, PhD, Chris Day, BSc, Sian Lowri Griffiths, PhD, Mariam Iqbal, BSc, Mirabel Pelton, MSc, Pavan Mallikarjun, MBBS, DPM, MRCPsych, PhD, Alexandra Stainton, MSc, and Ashleigh Lin, PhD. In the Department of Psychiatry, University of Turku, Finland: Alexander Denissoff, MD, Anu Ellila, RN, Tiina From, MSc, Markus Heinimaa, MD, PhD, Jarmo Hietala, MD, PhD, Tuula Ilonen, PhD, Paivi Jalo, RN, Mirka Kolkka, BM, Heikki Laurikainen, MD, Maarit Lehtinen, RN, Antti Luutonen, BA, Sinikka Luutonen, MD, PhD, Akseli Makela, BA, Janina Paju, MSc, Henri Pesonen, PhD, Reetta-Liina Armio (Saila), MD, Elina Sormunen, MD, Anna Toivonen, MSc, Lauri Tuominen, MD, PhD, Otto Turtonen, MD and Maija Walta, MD. At General Electric Global Research Inc, Munich, Germany: Ana Beatriz Solana, PhD, Manuela Abraham, MBA, Nicolas Hehn, PhD, and Timo Schirmer, PhD. The Workgroup of Paolo Brambilla, MD, PhD, University of Milan, Italy, including: in the Department of Neuroscience and Mental Health, Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico, University of Milan, Italy: Prof. Paolo Brambilla, MD, Carlo Altamura, MD, Marika Belleri, PsychD, Francesca Bottinelli, PsychD, Adele Ferro, PsychD, PhD, and Marta Re, PhD; in the Programma 2000, Niguarda Hospital, Milan: Emiliano Monzani, MD, Mauro Percudani, MD, and Maurizio Sberna, MD; in San Paolo Hospital, Milan: Armando D'Agostino, MD, and Lorenzo Del Fabro, MD; in Villa San Benedetto Menni, Albese con Cassano: Giampaolo Perna, MD, Maria Nobile MD, PhD, and Alessandra Alciati, MD. The Workgroup of Paolo Brambilla, MD, PhD, University of Udine, Udine, Italy, including: in the Department of Medical Area, University of Udine: Matteo Balestrieri, MD, Carolina Bonivento, PsychD, PhD, Giuseppe Cabras, PhD, and Franco Fabbro, MD, PhD; in the IRCCS Scientific Institute ‘E. Medea’, Polo FVG, Udine: Marco Garzitto, PsychD, PhD, and Sara Piccin, PsychD, PhD. The Workgroup of Prof. Alessandro Bertolino, University of Bari Aldo Moro, Italy, including: Prof. Alessandro Bertolino, Prof. Giuseppe Blasi, Prof. Linda A. Antonucci, Prof. Giulio Pergola, Grazia Caforio, PhD, Leonardo Faio, PhD, Tiziana Quarto, PhD, Barbara Gelao, PhD, Raffaella Romano, PhD, Ileana Andriola, MD, Andrea Falsetti, MD, Marina Barone, MD, Roberta Passatiore, MSc, Marina Sangiuliano, MD. In the Department of Psychiatry and Psychotherapy, Westfaelische Wilhelms-University Muenster, Germany: Prof. Rebekka Lencer, Marian Surman, MSc, Olga Bienek, MD, Georg Romer, MD, Udo Dannlowski, MD, PhD. In the Department of Psychiatry and Psychotherapy, Medical Faculty, Heinrich Heine University Düsseldorf, Germany: Christian Schmidt-Kraepelin, MD, Susanne Neufang, PhD, Alexandra Korda, PhD, and Henrik Rohner, MD.
Author contributions
Concept and design: S.B., P.B., C.P., E.M., R.K.R.S, S.W., S.R., N.Ko. Acquisition, statistical analysis, or interpretation of data: M.R., L.T.B., N.Ka, N.P., D.D., T.K.L., L.K.-I., F.S.-L., R.U., N.Ko and J.K. Drafting of the manuscript: M.R., L.T.B. and J.K. Critical revision of the manuscript for important intellectual content: N.Ka, N.P., D.D., T.K.L, L.K.-I., F.S.-L., A.B., S.B., P.B., C.P., R.L., E.M., R.K.R.S., R.U., S.W., S.R. and N.Ko.
Funding
PRONIA is a Collaboration Project funded by the European Union under the 7th Framework Programme (grant agreement no. 602152). M.R. is supported by a grant from the Koeln Fortune Program/Faculty of Medicine, University of Cologne. R.U. reports grants from the Medical Research Council, grants from the National Institute for Health Research, and personal fees from Sunovion, outside the submitted work. J.K. and N.Ko. received honoraria for talks presented at education meetings organised by Otsuka/Lundbeck. C.P. was supported by a National Health and Medical Research Council (NHMRC) Senior Principal Research Fellowship (1105825) and an NHMRC L3 Investigator Grant (1196508), has received honoraria for talks at educational meetings and has served on an advisory board for Lundbeck, Australia Pty Ltd.
Declaration of interest
D.D., F.S.-L., L.K.-I. and R.U. are members of the BJPsych editorial board and did not take part in the review or decision-making process of this paper.
eLetters
No eLetters have been published for this article.