1. Introduction
The prediction of psychosis from a Clinical High Risk state (CHR-P afterwards) is an ongoing topic of research and a common theme in the mainstream of clinical psychiatry [Reference Fusar-Poli1]. Nowadays, a range of tools, such as the Comprehensive Assessment of At Risk Mental States (CAARMS) [Reference Yung, Yuen, McGorry, Phillips, Kelly and Dell’Olio2] or the Scale of Prodromal Risk-Syndromes (SOPS) [Reference Miller, McGlashan, Rosen, Cadenhead, Cannon and Ventura3], can be used for prospective recognition of clinical “at-risk” states and clinical interventions [Reference Fusar-Poli, Davies and Bonoldi4], and demonstrate comparable prognostic accuracy with other prognostic tests used in Medicine [Reference Fusar-Poli, Cappucciati, Rutigliano, Schultze-Lutter, Bonoldi and Borgwardt5, Reference Fusar-Poli, Rutigliano, Stahl, Davies, De Micheli and Ramella-Cravaro6]. However, a recent meta-analysis indicates that the CHR-P designation retains only modest specificity, despite being characterised by high sensitivity [Reference Oliver, Kotlicka-Antczak, Minichino, Spada, McGuire and Fusar-Poli7]. This tool is clinically useful only for samples who have undergone substantial risk enrichment during the recruitment phases (15% risk at three years) [Reference Fusar-Poli, Schultze-Lutter, Cappucciati, Rutigliano, Bonoldi and Stahl8, Reference Fusar-Poli, Rutigliano, Stahl, Schmidt, Ramella-Cravaro and Hitesh9]. Accordingly, while its negative predictive values are quite good, the positive predictive value after two years is only estimated to be between 20% and 30% [Reference Oliver, Kotlicka-Antczak, Minichino, Spada, McGuire and Fusar-Poli7]. This difficulty in accurately improving specificity by identifying individuals who progress to developing psychosis have required the inclusion of Attenuated Psychosis Syndrome in the “conditions for further study” section of the DSM-5 (DSM-5-APS); however, it has also been included in the main text as a subcategory of “Other Specified Schizophrenia Spectrum and Other Psychotic Disorders” [Reference American Psychiatric Association10]. The DSM-5-APS construct has shown comparable prognostic accuracy to other CHR-P tools [Reference Fusar-Poli, De Micheli, Cappucciati, Rutigliano, Davies and Ramella-Cravaro11].
The CHR-P state, known to be an important step in the development of psychotic disorders, is now recommended as a target of therapeutic preventive interventions [Reference Joa, Gisselgård, Brønnick, McGlashan and Johannessen12–Reference Fusar-Poli15], although effective treatments for these patients have not yet been established [Reference Davies, Cipriani, Ioannidis, Radua, Stahl and Provenzani16]. Indeed, specialized clinical management for CHR-P individuals has been validated in clinical practice, and has officially become a key component of early intervention services worldwide [Reference NHS England17, 18]. Hence, interest is growing in improving the prognostic accuracy of tests for identifying CHR-P individuals with the highest risk of developing a psychotic disorder [Reference Fusar-Poli, Borgwardt, Bechdolf, Addington, Riecher-Rössler and Schultze-Lutter19]. Another important advancement would be to develop individualised approaches which would allow outcomes to be predicted at the individual subject level [Reference McGorry20].
One way by which the prognostic accuracy of the CHR-P construct may be improved would be by identifying specific clinical and biological predictors that account for the risk of developing psychosis [Reference Fusar-Poli, Borgwardt, Bechdolf, Addington, Riecher-Rössler and Schultze-Lutter19]. Of the established predictors, one of the most widely-replicated is the presence of attenuated psychotic symptoms [Reference Riecher-Rössler and Studerus21], which includes high levels of unusual thought content and/or suspiciousness [Reference Riecher-Rössler and Studerus21–Reference Crump, Arndt, Grivel, Horga, Corcoran and Brucato32], as well as disorganized communication [Reference Cornblatt, Carrión, Auther, McLaughlin, Olsen and John28, Reference Addington, Liu, Perkins, Carrion, Keefe and Woods31, Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen and Dingemans33–Reference Francesconi, Minichino, Carrión, Delle Chiaie, Bevilacqua and Parisi36]. Another line of research points to negative symptoms and poor psychosocial functioning as psychosis predictors [Reference Yung, Phillips, Yuen and McGorry22–Reference Addington, Liu, Buchy, Cadenhead, Cannon and Cornblatt26, Reference Cannon, Yu, Addington, Bearden, Cadenhead and Cornblatt29, Reference Carrión, Cornblatt, Burton, Tso, Auther and Adelsheim30, Reference Zhang, Li, Woodberry, Xu, Tang and Guo37–Reference Demjaha, Valmaggia, Stahl, Byrne and McGuire40]; however, most of these studies have been conducted in Western European countries, USA or Australia, and only a few initial reports have been published from Asia and Africa [Reference Zhang, Li, Woodberry, Xu, Tang and Guo37, Reference Katsura, Ohmuro, Obara, Kikuchi, Ito and Miyakoshi41, Reference Mamah, Musau, Mutiso, Owoso, Abdallah and Cottler42].
The present study advances knowledge in this area by investigating a group of CHR-P individuals recruited in Eastern Europe (Poland). Early intervention services have only started to develop in this region of Europe, and training and stuffing resources are insufficient. Therefore we wanted to test a simplified version of CAARMS, which could facilitate the assessment of individuals at risk for psychosis in clinical settings with less-established early detection and intervention services network.
Hence, the principal aim of the study was to develop an easy-to use, clinically-based risk-estimation model that could improve the ability of the clinician to predict the onset of psychosis in CHR-P individuals from this population in a shorter period of time.
2. Materials and methods
2.1. Settings and subjects
This prospective cohort study was performed in the Programme Of Recognition and Therapy (PORT), which was created to provide a medical service to support CHR-P individuals. The programme is affiliated with the Medical University of Lodz, and operates as secondary mental health care in the Lodz voivodship. The community-focused informative campaign regarding PORT, comprising outreach with local authorities, educational meetings and individual workshops for groups of students, parents and teachers at high schools in the Lodz voivodship, was carried within years 2010-2013. Starting from 2009, training and workshops for local mental-health professionals are provided at least once a year on a regular basis during local psychiatric conferences and meetings of the local section of the Polish Psychiatric Association. Referrals for the programme have been received from 1) community mental health services and child and adolescent mental health services, 2) inpatient mental health services, 3) educational institutions, 4) relatives or caregivers, 5) social services and 5) self-referrals. All referred persons complete the self-made PORT pre-screening questionnaire, based on Structured Interview for Prodromal Symptoms [Reference Miller, McGlashan, Rosen, Cadenhead, Cannon and Ventura3] available on the PORT website: www.mojport.pl, before their first visit in PORT centre. For positively-screened help-seeking individuals, this is followed by assessment with CAARMS and a physical examination, which includes a neurological section. Additionally, to more accurately evaluate functioning, an interview is conducted with parents or legal guardians (for underage individuals), or with closest relatives (for adult participants who give their consent). The comorbid diagnoses are established based on Structured Clinical Interviews for DSM-IV-Axis I and Axis II Disorders [Reference First, Spitzer, Gibbon and Williams43, Reference First, Gibbon, Spitzer, Williams and Benjamin44] performed within several days following the initial visit. More details regarding PORT and the procedures have been given previously [Reference Kotlicka-Antczak, Pawełczyk, Rabe-Jabłońska and Pawełczyk45, Reference Kotlicka-Antczak, Pawełczyk, Podgórski, Żurner, Karbownik and Pawełczyk46].
CHR-P individuals included in the study were consecutively referred to the programme in the period between March 2010 and July 2015. The follow-up period ended in May 2017. In this period, a group of 152 individuals, inhabitants of the Lodz voivodship, underwent the post-screening procedure, of whom 108 met the CHR-P criteria. Three cases dropped out from the programme at the recruitment phase, because of stigmatization problems. Finally, 105 individuals (49 males, 56 females) aged 15–32 years, who met the CHR-P criteria according to the CAARMS12/2006 (At Risk Mental State, ARMS) [Reference Yung, Yuen, McGorry, Phillips, Kelly and Dell’Olio2] were included in the analysis.
The exclusion criteria comprised the presence of a neurological disorder, evidence of intellectual disability or a diagnosis of psychotic disorder made according to ICD 10 criteria [Reference World Health Organization47], which is approved as the main classification in Poland. More detailed information regarding the PORT population (data until the end of 2014) can be found in our previous publication [Reference Kotlicka-Antczak, Pawełczyk, Podgórski, Żurner, Karbownik and Pawełczyk46]. We have previously demonstrated that the recruitment strategies adopted by our CHR-P service are -in line with that adopted worldwide by other similar early detection clinics [Reference Kotlicka-Antczak, Pawełczyk, Podgórski, Żurner, Karbownik and Pawełczyk46]. Accordingly, the level of risk enrichment, which indexes the probability of developing psychosis in the sample undergoing a CHR-P assessment before the result of the assessment are known [Reference Fusar-Poli and Schultze-Lutter48], was previously calculated for PORT sample as 14% at 36 months [Reference Fusar-Poli, Schultze-Lutter, Cappucciati, Rutigliano, Bonoldi and Stahl8]. The risk is comparable to that observed worldwide at the meta-analytical level: 15% at average 38 months follow-up [Reference Fusar-Poli, Schultze-Lutter, Cappucciati, Rutigliano, Bonoldi and Stahl8]. Hence, the sample may be considered representative of CHR-P services from different countries.
2.2. Baseline assessments
Individuals with a CHR-P state were identified with the use of the Polish version of CAARMS [Reference Yung, Yuen, McGorry, Phillips, Kelly and Dell’Olio2, Reference Jaracz, Grzechowiak, Raczkowiak, Rataj and Rybakowski49] and hence classified as presenting the following characteristics: (1) positive symptoms which do not reach the level of psychosis either in their frequency or intensity (the attenuated psychotic symptoms or APS subgroup), (2) experience of a psychotic episode which resolved spontaneously within one week during the past year (the brief limited intermittent psychotic symptoms or BLIPS subgroup) or, (3) meeting the criteria of schizotypal personality disorder or having a family history of psychosis in a first-degree relative (the genetic risk and deterioration syndrome, GRD subgroup) [Reference Yung, Yuen, McGorry, Phillips, Kelly and Dell’Olio2]. A substantial deterioration in psychosocial functioning, defined as a drop of at least 30% on the Social and Occupational Functioning Assessment Scale (SOFAS) was additionally required in all cases [Reference Yung, Yuen, McGorry, Phillips, Kelly and Dell’Olio2]. At the time of recruitment, a semi-structured questionnaire was also used to obtain demographic data and a family history of mental disorders. The level of psychosocial functioning was evaluated with the SOFAS included in the CAARMS [Reference Yung, Yuen, McGorry, Phillips, Kelly and Dell’Olio2]. The evaluation of intelligence level (Intelligence Quotient-IQ), was carried out using Wechsler Intelligence Scales (for Children or for Adults), Polish versions [Reference Wechsler50, Reference Wechsler51] as a part of the comprehensive neurocognitive assessment provided for PORT participants [Reference Kotlicka-Antczak, Pawełczyk, Rabe-Jabłońska and Pawełczyk45]. The obtained score for intelligence level was used as the basis for exclusion of intellectual disability.
To ensure a high standard of analysis, the model was generated according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis (TRIPOD) checklist [Reference Moons, Altman, Reitsma, Ioannidis, Macaskill and Steyerberg52].
The study was carried out in accordance with the latest version of the Declaration of Helsinki. The study protocol was accepted by the Medical University of Lodz Ethics Committee. Written informed consent was provided by the participants, or both participants and their parents, in cases where the participant was underage.
2.3. Treatment
The study participants received needs-based interventions, targeting their individual problems. As described previously, all CHR-P individuals received psychological interventions, most commonly including psychological support focusing on social relationships, educational/vocational and family issues or structured psychotherapy (cognitive behavioural therapy) [Reference Kotlicka-Antczak, Pawełczyk, Rabe-Jabłońska and Pawełczyk45]. In order to treat comorbid disorders, antidepressants or mood stabilizers were used. Based on the International Clinical Practice Guidelines for Early Psychosis, three clinical situations were considered as indications for treatment with antipsychotics: (1) rapid deterioration of mental state; (2) the presence of a severe suicide risk, with treatment of any depression proving ineffective; or (3) aggression or hostility were increasing and posing a risk to others [53]. The details regarding pharmacological treatment received by the study participants are presented in supplementary eTable2.
2.4. Outcome
The outcome predicted in the study was transition from a CHR-P to psychotic disorder, which was determined on the basis of the CAARMS criteria for psychosis threshold [Reference Yung, Yuen, McGorry, Phillips, Kelly and Dell’Olio2]. This was defined as: 1) maximum score (six) on at least one of CAARMS Positive Symptoms Scale (Unusual Thought Content, Non-Bizarre Ideas, Disorganized Speech, for Perceptual Abnormalities minimum score of five was required); 2) the frequency of symptoms as at least four on a six-point scale (frequency three to six times per week, more than an hour per occasion or daily - less than an hour per occasion) 3) symptoms present for longer than one week [Reference Yung, Yuen, McGorry, Phillips, Kelly and Dell’Olio2].
In order to identify a potential transition, participants were assessed with the CAARMS every three months during the follow-up period, or more often, if required. Since the follow-up period was relatively long, as given in the Results section, it was not possible for each evaluation to be carried out by the same clinician; both researchers engaged in the procedure were blinded to the results of their evaluations. Their inter-rater reliability determined during the CAARMS training was high (kappa = 0.9).
To further validate the transition and to determine its clinical category, the identification of full-blown psychosis based on CAARMS was followed by applying ICD-10 criteria [Reference World Health Organization47].
2.5. Preselection of predictors of transition
Potential clinical predictors of the onset of psychosis from a CHR-P stage were preselected on the basis of availability in clinical settings and existing clinical knowledge [Reference Steyerberg and Vergouwe54–Reference Fusar-Poli, Stahl and Steyerberg57]. This is regarded as the optimal approach to building robust clinical prediction models in psychiatry [Reference Studerus, Ramyead and Riecher-Rossler55, Reference Studerus, Ramyead and Riecher-Rossler55, Reference Wynants, Collins and Van Calster56, Reference Fusar-Poli, Stahl and Steyerberg57]. The model complexity was a priori limited by the number of events (transitions) to no more than two predictors. This was set to allow an event per variable (EPV) ratio of 10 and above, which is recommended to develop robust prognostic models [Reference Studerus, Ramyead and Riecher-Rossler55, Reference Studerus, Ramyead and Riecher-Rossler55, Reference Wynants, Collins and Van Calster56, Reference Fusar-Poli, Stahl and Steyerberg57]. The predictors were assumed to have a linear effect on the outcome, and to keep the model as simple as possible, no interactions were allowed [Reference Wynants, Collins and Van Calster56].
2.6. Data analysis
No data was missing from the dataset. The Kaplan-Meier method was utilized in survival analysis to estimate the cumulative risk function of transition to psychosis. The time-to-event model was developed using multivariate Cox proportional hazard regression with two prespecified predictors. The collinearity of these predictors was excluded with the use of Spearman’s rho. Because of the high imbalance in class representation (i.e. the numbers of converters and non-converters), model induction was performed after the application of a synthetic minority over-sampling technique [Reference Chawla, Bowyer, Hall and Kegelmeyer58]. The Shoenfeld residuals test was used to control for proportional hazard assumption. The degree of discrimination of the model, representing its ability to distinguish low-risk from high-risk individuals, was measured with the use of Harrell’s concordance index (c-index), a measure analogous to the area under the curve (AUC), but one that was tailored for censored data [Reference Uno, Cai, Pencina, D’Agostino and Wei59]. In addition, receiver-operating characteristics (ROC) curves against time [Reference Kamarudin, Cox and Kolamunnage-Dona60] were presented, with sensitivity, specificity, accuracy and AUCs for different time points [1, 2, 3, and 4 years, respectively].
The model was internally validated using bootstrap methods with 1000 samples [Reference Efron and Tibshirani61]. First, we applied the Cox proportional hazard regression model to each of these samples. In the next step, all resulting models were applied to the original dataset to estimate their performance. Average Somer's D calculated on all both bootstrap samples and original dataset and their difference was used to estimate model’s optimism. Optimism corrected c-index was then calculated based on its relationship to Somer's D (C = 0.5(D + 1)). In a further step we estimated the calibration (the agreement between the predicted risks and observed event rates) of the optimism-corrected model using calibration plots. The statistical analysis was conducted in R open source software, version 3.4.3 [Reference R Core Team62].
3. Results
The TRIPOD checklist for the study is presented in eTable 1.
3.1. Socio-demographic and clinical characteristics
The baseline demographic and clinical characteristics of the CHR-P participants are presented in Table 1. The CHR-P individuals most commonly presented APS, either as the only symptoms or in association with GRD syndrome (81 subjects in total; 77.1%). The median SOFAS score indicated severe impairment in psychosocial functioning. The detailed characteristics of the sample is additionally included in eTable2.
3.2. Transition to psychosis
The median follow-up period for the entire sample was 36 months (IQR: 10–59 months; mean time: 35.4 ± 25.0 months). The cumulative probability of developing psychosis of the CHR-P individuals and time to psychosis onset (failure function) is presented in Fig. 1. The median time from the entry to transition was five months (25th-75th percentile 3.5–10.5 months; mean time: 9.1 months; standard deviation: 8.9 months). All converters developed schizophrenia, diagnosed on the basis of ICD-10 criteria [Reference World Health Organization47].
Baseline socio-demographic and clinical characteristics of CHR-P individuals (N = 105).
APS: attenuated psychotic symptoms.
BLIPS: brief limited intermittent psychotic symptoms.
CAARMS: Comprehensive Assessment of At Risk Mental States.
CHR-P: clinical high risk for psychosis.
GRD: genetic risk and deterioration syndrome.
IQ: intelligence quotient.
SOFAS: Social and Occupational Functioning Assessment Scale.
* Full CAARMS characteristics have been included in supplementary materials (eTable2).
Twenty four (22.9%) of the tested individuals had been using antipsychotic drugs; however, this was unlikely to affect the transition to psychosis (p > 0.05, multivariate Cox regression adjusted for Unusual Thought Content and Disorganized Speech CAARMS scores).
3.3. Selection of predictors
A thorough literature search identified two candidate predictors of transition, which had been most commonly used in previous studies: speech disorganization and unusual thought content. Of 24 analysed studies, 12 indicated speech disorganization to be a clinical feature predicting transition or a feature that could significantly differentiate converters from non-converters [Reference Addington, Liu, Buchy, Cadenhead, Cannon and Cornblatt26, Reference Cornblatt, Carrión, Auther, McLaughlin, Olsen and John28, Reference Addington, Liu, Perkins, Carrion, Keefe and Woods31, Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen and Dingemans33–Reference Francesconi, Minichino, Carrión, Delle Chiaie, Bevilacqua and Parisi36, Reference Demjaha, Valmaggia, Stahl, Byrne and McGuire40–Reference Mamah, Musau, Mutiso, Owoso, Abdallah and Cottler42, Reference Tarbox, Addington, Cadenhead, Cannon, Cornblatt and Perkins63, Reference Armando, Pontillo, De Crescenzo, Mazzone, Monducci and Lo Cascio64] while 10 studies identified unusual thought content [Reference Cannon, Cadenhead, Cornblatt, Woods, Addington and Walker23–Reference Addington, Liu, Buchy, Cadenhead, Cannon and Cornblatt26, Reference Cannon, Yu, Addington, Bearden, Cadenhead and Cornblatt29–Reference Crump, Arndt, Grivel, Horga, Corcoran and Brucato32, Reference Mason, Startup, Halpin, Schall, Conrad and Carr38, Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos and Bruxner65]; three of the studies included samples larger than 400 CHR-P individuals [Reference Addington, Liu, Buchy, Cadenhead, Cannon and Cornblatt26, Reference Cannon, Yu, Addington, Bearden, Cadenhead and Cornblatt29, Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos and Bruxner65]. Both speech disorganization and unusual thought content have also been found to be independent predictors of transition [Reference Addington, Liu, Buchy, Cadenhead, Cannon and Cornblatt26, Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen and Dingemans33]; and high unusual thought content score has been described as a relevant predictor of psychosis in a consensus paper summarizing 20 years of research in the CHR-P field [Reference Fusar-Poli, Borgwardt, Bechdolf, Addington, Riecher-Rössler and Schultze-Lutter19]. Additionally, a systematic review by Schmidt et al. [Reference Schmidt, Cappucciati, Radua, Rutigliano, Rocchetti and Dell’Osso66] found unusual thought content to be included in a model presenting the highest positive predictive value (PPV), and speech disorganization in that with the second highest PPV. Importantly, collinearity of the two parameters has been excluded in the tested population (Spearman’s rho = 0.10; p = 0.31). Hence, these two parameters clearly merit consideration for inclusion in the model used in the present study.
The cumulative probability of developing psychosis of the CHR-P individuals and time to psychosis onset (failure function). The dashed lines represent 95% confidence intervals of the estimated probability. There were 84 individuals at risk followed up for at least 6 months (80.0% of the entire sample), 77- for 12 mo (73.7%), 69 - for 18 mo (65.7%) 66 - for 24 mo (62.9%), 57 - for 30 mo (54.3%), 54 - for 36 mo (51.4%), 42 - for 42 mo (40.0%), 41- for 48 mo (39.0%), 35 - for 54 mo (33.3%), 24 - for 60 mo (22.9%), 16 – for 66 mo (15.2%) and 12 - for 72 mo (11.4%).
3.4. Prediction of transition
The multivariate Cox proportional hazards model found both speech disorganization (HR = 1.69; 95% CI: 1.39–2.05) and unusual thought content (HR = 1.51; 95%CI: 1.27–1.80) to be statistically significant (p < 0.0001) predictors of conversion. As shown in supplementary material (eFig. 1 and eTable 3), the model met the proportional hazards assumption. The parameter estimates in the regression equation were 0.41 for unusual thought content and 0.52 for speech disorganization.
The developed model presented an adequate c-index of 0.79, and its predictive abilities at different time points were satisfactory (Fig. 2) [Reference Hosmer, Lemeshow and May67]. Based on the model, an algorithm was generated allowing the probability of transition from a CHR-P to psychosis to be estimated. The risk calculator is available online at: https://link.konsta.com.pl/psychosis. We recommend using Google Chrome to open it. We have estimated that recording the two key positive items in subjects undergoing a CHR-P assessment would require 10–15 minutes, while two hours are needed to administer the full CAARMS.
The results of the bootstrap internal validation procedure are shown in Fig. 3. The average c-index for the models developed on bootstrap samples was 0.78 and the mean c-index of their performance in the original dataset was 0.77. Optimism corrected c-index after 1000-sample bootstrap internal validation was 0.78. The calibration plots revealed a high degree of consistency between the observed probabilities and the model-predicted probabilities derived from 1000 bootstrap resamples. The internal validation of our model did not reveal substantial overfitting issues because a similar performance was observed after correction for optimism.
4. Discussion
The study develops and internally validates a pilot clinical prediction model to refine the prognosis of transition from a risk stage to established psychosis. The prognostic model has been developed in a new cohort from Poland, which is representing eastern European CHR-P patients. High levels of speech disorganization and unusual thoughts were chosen as key predictors, on the basis of a priori knowledge. The model showed satisfactory accuracy to predict psychosis and was internally validated, showing no major overfitting problems.
Performance of the model predicting transition from CHR-P to psychosis at multiple time points. Time-dependent receiver operating characteristics (ROC) curves and the parameters are illustrated for 1-, 2-, 3- and 4-year outcomes. AUC – area under the curve. Accuracy is the proportion of all subjects for which conditions were accurately predicted (transition and non-transition). Sensitivity is the proportion of truly predicted transitions to all factual transitions. Specificity is the proportion of truly predicted non-transitions to all factual non-transitions. The cut-off points were set a priori as 0.5.
Our study followed state-of-the-art recommendation for developing clinical prediction models in psychiatry [Reference Steyerberg and Vergouwe54–Reference Fusar-Poli, Stahl and Steyerberg57], although we have been unable to perform external validation (see below). Current methodological guidelines recommend selecting predictors on the basis of a priori knowledge [Reference Studerus, Ramyead and Riecher-Rossler55, Reference Fusar-Poli, Stahl and Steyerberg57]. As recently noted, preselection of predictors that can reflect the ability of clinicians to formulate a prognosis is essential to inform the development of robust prediction models in the field [Reference Nelson, Yung and McGorry68]. Accordingly, we rigorously selected predictors which have a rich literature and a proven position as predictors of psychotic transition from CHR-P. Disorganized communication was chosen, because it has been found to be a predictor of psychosis in CHR-P studies including children and adolescents [Reference Armando, Pontillo, De Crescenzo, Mazzone, Monducci and Lo Cascio64], adolescents [Reference Cornblatt, Carrión, Auther, McLaughlin, Olsen and John28, Reference Bearden, Wu, Caplan and Cannon34, Reference Mamah, Musau, Mutiso, Owoso, Abdallah and Cottler42]· or older samples [Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen and Dingemans33, Reference DeVylder, Muchomba, Gill, Ben-David, Walder and Malaspina35]. In addition, severity of unusual thoughts was also chosen, because it has been recognized as a risk factor for transition of CHR-P into psychosis in both CAARMS [Reference Thompson, Nelson and Yung25, Reference Perkins, Jeffries, Cornblatt, Woods, Addington and Bearden27, Reference Velthorst, Nieman, Becker, van de Fliert, Dingemans and Klaassen39, Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos and Bruxner65] or SIPS-based studies [Reference Cannon, Cadenhead, Cornblatt, Woods, Addington and Walker23, Reference Cannon, Yu, Addington, Bearden, Cadenhead and Cornblatt29]. In particular, both speech disorganization and unusual thought content have been found to significantly differentiate converters to psychosis from non-converters or to act as predictors of transition in studies based on populations from different world regions [Reference Addington, Liu, Buchy, Cadenhead, Cannon and Cornblatt26, Reference Addington, Liu, Perkins, Carrion, Keefe and Woods31, Reference Katsura, Ohmuro, Obara, Kikuchi, Ito and Miyakoshi41]. This specific selection was also adopted for the study for pragmatic reasons: an easy-to-perform short assessment of two positive symptoms is a routine procedure in clinical practice.
The pre-specified mode was applied to the first Polish CHR-P cohort. There is converging evidence that the actual level of psychosis that is observed in CHR-P individuals is mostly determined by the way these individuals have been recruited, and therefore by their risk enrichment before they undergo CHR-P assessment [Reference Fusar-Poli, Schultze-Lutter, Cappucciati, Rutigliano, Bonoldi and Stahl8, Reference Fusar-Poli, Rutigliano, Stahl, Schmidt, Ramella-Cravaro and Hitesh9, Reference Fusar-Poli, Schultze-Lutter and Addington69–Reference Fusar-Poli71]. As mentioned in the Methods section, our sample is representative of CHR-P services from different countries because the observed risk enrichment is very similar to the pooled meta-analytical mean [Reference Fusar-Poli, Schultze-Lutter, Cappucciati, Rutigliano, Bonoldi and Stahl8]. This suggests that our results are theoretically generalisable.
The model showed some promising clinical value, with its overall accuracy (Harrell’s c-index of 0.79) being found to be adequate [Reference Hosmer, Lemeshow and May67], confirming that these two clinical characteristics may be potentially used to estimate the risk of developing psychotic disorder in CHR-P individuals at the Pan-European level. Importantly, because our sample was rather small with rare events, it was validated internally to correct for potential overfitting problems. A small difference was observed in the optimism corrected c-index (0.78) and original c-index (0.79), suggesting that it demonstrates resilience to overfitting, further confirming its potential utility. It is however important to note that the broad generalisability of our findings need to be estimated in future external replication studies. External replication studies in samples undergoing CHR-P assessment are particularly needed given the high variance in risk enrichment which is associated with the recruitment phases for these individuals [Reference Fusar-Poli, Schultze-Lutter, Cappucciati, Rutigliano, Bonoldi and Stahl8].
Calibration plots of the bootstrap validation of the model for specific time points showing the relationship between the original model and the average of bootstrap-based models.
The grey diagonal line shows the ideal prediction;The dashed line shows the predicted and observed values in the original model; The thin black line shows the change in calibration following the bootstrap-based correction for optimism;
The calibration plots show both the accuracy of prediction and the resilience of the modelling method to overfitting.
The median time to transition in the study was found to be five months, which is in line with meta-analytical median time to psychosis observed CHR-P populations [Reference Kempton, Bonoldi, Valmaggia, McGuire and Fusar-Poli72]. To further illustrate the speed of psychosis progression for the entire cohort, the failure function was reported in Fig. 1. As shown in the Results Section (3.2), treatment with antipsychotics had no influence on the outcome. This may seem surprising in the light of previous findings which have offered promising results for implementing antipsychotic treatment in CHR-P individuals [Reference van der Gaag, Smit, Bechdolf, French, Linszen and Yung73]. Also, according to the guidelines of European Psychiatric Association, such treatment is recommended if psychological interventions prove to be ineffective [Reference Schmidt, Schultze-Lutter, Schimmelmann, Maric, Salokangas and Riecher-Rössler74]. However, recent network meta-analyses show that no treatment for CHR-P individuals to date appears superior to other forms of therapy in preventing transition to psychosis or in reducing attenuated psychotic- or negative symptoms [Reference Davies, Cipriani, Ioannidis, Radua, Stahl and Provenzani16, Reference Devoe, Farris, Townes and Addington75, Reference Devoe, Peterson and Addington76].
The model presented in this study is based on two clinical predictors that are routinely collected as part of the CHR-P assessment. The overall prognostic accuracy of the optimism-corrected model (Harrell’s c-index = 0.78) was comparable to that of the full CAARMS assessment [Reference Oliver, Kotlicka-Antczak, Minichino, Spada, McGuire and Fusar-Poli7], with no apparent predictive gain. However, there are two potential benefits that may be associated with our clinical prediction model. First, our model allows the transition to psychosis to be predicted at the individual subject level, either when employed as a complement to full CAARMS assessment or as an independent tool. This is not currently possible using available prognostic tools, which can only indicate the risk of conversion at a group level. Second, since our model employs only two predictors, it may offer some practical advantages in that it is shorter than the administration the full CAARMS. In fact, we estimate that only 10–15 minutes are typically needed to assess the two predictors that this model is based on, compared to the two hours needed to administer the full CAARMS. Although brief versions of the CAARMS (still including four positive symptom items) already exist [77, Reference Yung, Phillips, Simmons, Ward, Thompson and French78], their prognostic accuracy has not been validated; in contrast, the current study employs state-of-the-art measures of prognostic accuracy. In addition, while the brief version of the CAARMS does not allow any individualised prediction of outcomes, our clinical prediction tool, including only two essential predictors, can potentially be used to formulate predictions at the individual subject level.
We are aware that the clinical presentation of the CHR-P state is much more complex and there is undoubtedly an unequivocal need to evaluate a broad range of symptoms, problems and levels of functioning in each individual with CHR-P. However, this is not always possible, especially in low resourced early detection services. Conversely, our revised prognostic tool requires substantially less time for its administration, compared to the standard prognostic tools in this area. This may be a significant consideration when applying this tool in European countries like Poland, which may have more limited early intervention services.
The proposed risk estimation model has, however, several limitations. Firstly, it is based on a small sample size and a small number of events, which limits the generalisability of the model [Reference Fusar-Poli, Stahl and Steyerberg57]. This is particularity concerning given the lack of proper external validation. Although we have performed an internal validation, our clinical prediction model should be considered cautiously and cannot be recommended for implementation in clinical routine before it is fully validated externally in independent samples. The online version of the tool is therefore provided to help independent researchers running this clinical prediction model in their samples and to measure its actual reproducibility.
The assumed recently recommended methodological approach limited the possibility to fully explore our dataset: the necessity to keep the EPV of minimum 10 a priori reduced the number of variables included in the analysis to two. This can, however, offer some pragmatic advantages for potential implementation in clinical routine: the presence of a high level of unusual thoughts or speech disorganization can be easily recorded as part of the clinical routine of CHR-P services and further facilitate the external validation of our model.
Another limitation is that our sample was not large enough to allow the differential impact of different CHR-P subgroups on the final level of risk to psychosis to be tested [Reference Fusar-Poli, Cappucciati, Rutigliano, Schultze-Lutter, Bonoldi and Borgwardt5, Reference Fusar-Poli, Rutigliano, Stahl, Davies, De Micheli and Ramella-Cravaro6, Reference Fusar-Poli, Cappucciati, Borgwardt, Woods, Addington and Nelson79–Reference Fusar-Poli, Cappucciati, De Micheli, Rutigliano, Bonoldi and Tognin81]. Further studies analysing other clinical and biological features predicting transition from a CHR-P to psychosis may be needed to improve the prognostic accuracy of our model.
5. Conclusions
This study develops and internally validates a clinically-based risk estimation tool which is based on levels of speech disorganization and of unusual thoughts. The tool is simple and requires less clinical resources compared to the original CHR-P assessment. Future external replication studies in independent samples are required before generalisability of this model outside Polish samples can be demonstrated.
Declaration of interests
None.
Acknowledgement
The study was financed by the Polish Science National Centre as part of a larger project (grant No NN 402 1793 34).
We would like to thank all the psychiatrists referring participants to the study.
Appendix A. Supplementary data
Supplementary material related to this article can be found, in the online version, at doi:https://doi.org/10.1016/j.eurpsy.2019.02.007.
Comments
No Comments have been published for this article.