Hostname: page-component-78c5997874-dh8gc Total loading time: 0 Render date: 2024-10-30T00:31:31.694Z Has data issue: false hasContentIssue false

Clinical and psychological factors associated with resilience in patients with schizophrenia: data from the Italian network for research on psychoses using machine learning

Published online by Cambridge University Press:  11 October 2022

Linda A. Antonucci
Affiliation:
Department of Basic Medical Science, Neuroscience and Sense Organs, University of Bari Aldo Moro, Bari, Italy
Giulio Pergola
Affiliation:
Department of Basic Medical Science, Neuroscience and Sense Organs, University of Bari Aldo Moro, Bari, Italy
Antonio Rampino
Affiliation:
Department of Basic Medical Science, Neuroscience and Sense Organs, University of Bari Aldo Moro, Bari, Italy
Paola Rocca
Affiliation:
Department of Neuroscience, Section of Psychiatry, University of Turin, Turin, Italy
Alessandro Rossi
Affiliation:
Section of Psychiatry, Department of Biotechnological and Applied Clinical Sciences, University of L'Aquila, L'Aquila, Italy
Mario Amore
Affiliation:
Section of Psychiatry, Department of Neurosciences, Rehabilitation, Ophthalmology, Genetics and Maternal and Child Health, University of Genoa, Genoa, Italy
Eugenio Aguglia
Affiliation:
Department of Clinical and Molecular Biomedicine, Psychiatry Unit, University of Catania, Catania, Italy
Antonello Bellomo
Affiliation:
Psychiatry Unit, Department of Medical Sciences, University of Foggia, Foggia, Italy
Valeria Bianchini
Affiliation:
Unit of Psychiatry, Department of Life, Health and Environmental Sciences, University of L'Aquila, L'Aquila, Italy
Claudio Brasso
Affiliation:
Department of Neuroscience, Section of Psychiatry, University of Turin, Turin, Italy
Paola Bucci
Affiliation:
Department of Psychiatry, University of Campania “Luigi Vanvitelli”, Naples, Italy
Bernardo Carpiniello
Affiliation:
Section of Psychiatry, Department of Public Health, Clinical and Molecular Medicine, University of Cagliari, Cagliari, Italy
Liliana Dell'Osso
Affiliation:
Section of Psychiatry, Department of Clinical and Experimental Medicine, University of Pisa, Pisa, Italy
Fabio di Fabio
Affiliation:
Department of Neurology and Psychiatry, “La Sapienza” University of Rome, Rome, Italy
Massimo di Giannantonio
Affiliation:
Department of Neuroscience and Imaging, “G. D'Annunzio” University, Chieti, Italy
Andrea Fagiolini
Affiliation:
Department of Molecular Medicine and Clinical Department of Mental Health, University of Siena, Siena, Italy
Giulia Maria Giordano
Affiliation:
Department of Psychiatry, University of Campania “Luigi Vanvitelli”, Naples, Italy
Matteo Marcatilli
Affiliation:
Department of Psychiatry, University of Milan, Milan, Italy
Carlo Marchesi
Affiliation:
Department of Neuroscience, Psychiatry Unit, University of Parma, Parma, Italy
Paolo Meneguzzo
Affiliation:
Psychiatric Clinic, Department of Neurosciences, University of Padua, Padua, Italy
Palmiero Monteleone
Affiliation:
Department of Medicine, Surgery and Dentistry “Scuola Medica Salernitana” Section of Neuroscience, University of Salerno, Salerno, Italy
Maurizio Pompili
Affiliation:
Department of Neurosciences, Mental Health and Sensory Organs, S. Andrea Hospital, “La Sapienza” University of Rome, Rome, Italy
Rodolfo Rossi
Affiliation:
Section of Psychiatry, Department of Biotechnological and Applied Clinical Sciences, University of L'Aquila, L'Aquila, Italy
Alberto Siracusano
Affiliation:
Department of Systems Medicine, Psychiatry and Clinical Psychology Unit, “Tor Vergata” University of Rome, Rome, Italy
Antonio Vita
Affiliation:
Psychiatric Unit, School of Medicine, University of Brescia, Brescia, Italy Department of Mental Health, Spedali Civili Hospital, Brescia, Italy
Patrizia Zeppegno
Affiliation:
Department of Translational Medicine, Psychiatric Unit, University of Eastern Piedmont, Novara, Italy
Silvana Galderisi
Affiliation:
Department of Psychiatry, University of Campania “Luigi Vanvitelli”, Naples, Italy
Alessandro Bertolino*
Affiliation:
Department of Basic Medical Science, Neuroscience and Sense Organs, University of Bari Aldo Moro, Bari, Italy
Mario Maj
Affiliation:
Department of Psychiatry, University of Campania “Luigi Vanvitelli”, Naples, Italy
Italian Network for Research on Psychoses
Affiliation:
Department of Basic Medical Science, Neuroscience and Sense Organs, University of Bari Aldo Moro, Bari, Italy
*
Author for correspondence: Alessandro Bertolino, E-mail: alessandro.bertolino@uniba.it
Rights & Permissions [Opens in a new window]

Abstract

Background

Resilience is defined as the ability to modify thoughts to cope with stressful events. Patients with schizophrenia (SCZ) having higher resilience (HR) levels show less severe symptoms and better real-life functioning. However, the clinical factors contributing to determine resilience levels in patients remain unclear. Thus, based on psychological, historical, clinical and environmental variables, we built a supervised machine learning algorithm to classify patients with HR or lower resilience (LR).

Methods

SCZ from the Italian Network for Research on Psychoses (N = 598 in the Discovery sample, N = 298 in the Validation sample) underwent historical, clinical, psychological, environmental and resilience assessments. A Support Vector Machine algorithm (based on 85 variables extracted from the above-mentioned assessments) was built in the Discovery sample, and replicated in the Validation sample, to classify between HR and LR patients, within a nested, Leave-Site-Out Cross-Validation framework. We then investigated whether algorithm decision scores were associated with the cognitive and clinical characteristics of patients.

Results

The algorithm classified patients as HR or LR with a Balanced Accuracy of 74.5% (p < 0.0001) in the Discovery sample, and 80.2% in the Validation sample. Higher self-esteem, larger social network and use of adaptive coping strategies were the variables most frequently chosen by the algorithm to generate decisions. Correlations between algorithm decision scores, socio-cognitive abilities, and symptom severity were significant (pFDR < 0.05).

Conclusions

We identified an accurate, meaningful and generalizable clinical-psychological signature associated with resilience in SCZ. This study delivers relevant information regarding psychological and clinical factors that non-pharmacological interventions could target in schizophrenia.

Type
Original Article
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press

Introduction

Schizophrenia (SCZ) is a severe psychiatric disorder characterized by heterogeneous clinical manifestations and often associated with real-life functional deficits (Antonucci et al., Reference Antonucci, Fazio, Pergola, Blasi, Stolfa and Di Palo2022a; Crespo-Facorro et al., Reference Crespo-Facorro, Such, Nylander, Madera, Resemann, Worthington and Newton2021). These functional impairments may have serious consequences on social and occupational adjustment, on patients' ability to perform everyday tasks (Millier et al., Reference Millier, Schmidt, Angermeyer, Chauhan, Murthy, Toumi and Cadi-Soussi2014), and on indirect costs of the disease, which are three times larger than the direct ones (Mayoral-van Son et al., Reference Mayoral-van Son, Juncal-Ruiz, Ortiz-Garcia de la Foz, Cantarero-Prieto, Blazquez-Fernandez, Paz-Zulueta and Crespo-Facorro2019). The high personal and economic burden associated with SCZ urges researchers to detect new targets for an easier and more effective management, both from pharmacological and non-pharmacological perspectives.

So far, SCZ research has largely focused on a hypothesis-driven approach for the identification of factors associated with an increase in the risk of developing the disorder (Arango et al., Reference Arango, Dragioti, Solmi, Cortese, Domschke, Murray and Fusar-Poli2021; Tandon, Nasrallah, & Keshavan, Reference Tandon, Nasrallah and Keshavan2009). This approach has been helpful in many respects, but it does not fully capitalize on the resources available to the individual to deal with the disorder. A paradigm focused on the discovery of factors increasing the likelihood of recovery might be more suitable to foster better clinical management of patients (Fusar-Poli et al., Reference Fusar-Poli, Correll, Arango, Berk, Patel and Ioannidis2021; Kobau et al., Reference Kobau, Seligman, Peterson, Diener, Zack, Chapman and Thompson2011). Resilience has been defined as the dynamic ability to adapt, maintain (or change) thoughts and behaviors, and negotiate with the environment in the presence of adverse life events, in order to effectively cope with them. It has been operationalized as ‘all the positive resources that may be activated in the context of stress to prevent the development of negative mental health outcomes’ (Morote, Hjemdal, Martinez Uribe, & Corveleyn, Reference Morote, Hjemdal, Martinez Uribe and Corveleyn2017). Earlier studies have demonstrated that resilience abilities are plastic over the entire life course (Feder, Fred-Torres, Southwick, & Charney, Reference Feder, Fred-Torres, Southwick and Charney2019). Indeed, stressors which the individual can be exposed to across the lifespan, as well as environmental resources, personal attributes, and neurodevelopmental changes, may either facilitate or challenge the individual capacity of adaptation, negotiation, and management of stress. These variations make resilience a dynamic process over time and between individuals (Kalisch et al., Reference Kalisch, Basten, Boks, Bonanno, Brummelman, Chmitorz and Kleim2017; Rutten et al., Reference Rutten, Hammels, Geschwind, Menne-Lothmann, Pishva, Schruers and Wichers2013), rather than a static outcome (e.g. not developing a mental disorder despite the presence of risk factors).

Nevertheless, the importance of resilience as a process for mental health is testified by its association with well-being, quality of life, global functioning, and lifelong health in the general population (Windle, Bennett, & Noyes, Reference Windle, Bennett and Noyes2011). In the absence of disease, higher resilience (HR) abilities seem to be negatively associated with indices of mental illness (i.e. depression, anxiety, and negative emotions), and positively associated with indices of well-being (i.e. perceived quality of life, psychological well-being, and positive emotions) (Shapero et al., Reference Shapero, Farabaugh, Terechina, DeCross, Cheung, Fava and Holt2019; Skrove, Romundstad, & Indredavik, Reference Skrove, Romundstad and Indredavik2013; Smith, Reference Smith2009; Ungar & Theron, Reference Ungar and Theron2020; Windle et al., Reference Windle, Bennett and Noyes2011).

However, resilience abilities seem to be key also in contributing to disease course and levels of symptomatology in severe psychiatric conditions, such as SCZ (Choi et al., Reference Choi, Cha, Jang, Park, Kim, Lee and Lee2015; Haeffel & Grigorenko, Reference Haeffel and Grigorenko2007; Hjemdal, Vogel, Solem, Hagen, & Stiles, Reference Hjemdal, Vogel, Solem, Hagen and Stiles2011). Indeed, studies revealed that higher levels of resilience are associated with lower positive and negative symptom severity in chronic SCZ patients (Bozikas & Parlapani, Reference Bozikas and Parlapani2016), with a higher chance of full recovery (Torgalsboen, Reference Torgalsboen2012), and with higher service engagement (Tait, Birchwood, & Trower, Reference Tait, Birchwood and Trower2004). Notably, resilience has been associated also with disability and personal functioning (Torgalsboen, Reference Torgalsboen2012). More specifically, a recent study from the Italian Network for Research on Psychoses highlighted that resilience levels moderated the relationship of real-life functioning with avolition and internalized stigma (Galderisi et al., Reference Galderisi, Rossi, Rocca, Bertolino, Mucci and Bucci2014, Reference Galderisi, Rucci, Mucci, Rossi, Rocca and Bertolino2020). Furthermore, the crucial role of resilience abilities in SCZ onset and course is supported by previous literature showing that individuals at clinical high risk for psychosis, who subsequently converted into frank disease, had lower resilience (LR) levels than those who did not convert (Kim et al., Reference Kim, Song, Park, Lee, Lee, Lee and Kwon2013). Notably, cognitive performance, whose deficits are characteristic of SCZ patients at the level of multiple and highly variable neuropsychological domains (Antonucci et al., Reference Antonucci, Pergola, Pigoni, Dwyer, Kambeitz-Ilankovic, Penzel and Bertolino2020b),1 has been previously positively associated with resilience levels in non-clinical populations (Simeon et al., Reference Simeon, Yehuda, Cunill, Knutelska, Putnam and Smith2007).

This body of evidence suggests that (i) there is variability between patients in residual resilience abilities – associated with the clinical characteristics of the disease –, and (ii) resilience abilities are worth being fostered in clinical populations. However, factors that may potentially foster resilience abilities, especially in the presence of disease conditions, are still poorly investigated. Accurate identification of resilience predictors even in severe mental conditions such as SCZ is crucial as it would potentially allow more accurate targeting of protective factors within non-pharmacological screening and intervention programs. In this respect, resilience abilities have been reported to result from a complex interplay between psychological (e.g. personality traits, self-esteem) (Bajaj, Gupta, & Pande, Reference Bajaj, Gupta and Pande2016; Iimura & Taku, Reference Iimura and Taku2018), environmental (e.g. stressful life events, characteristics of social systems) (Chen & Kuo, Reference Chen and Kuo2020; Seery, Holman, & Silver, Reference Seery, Holman and Silver2010; Southwick et al., Reference Southwick, Sippel, Krystal, Charney, Mayes and Pietrzak2016), and historical/clinical factors (e.g. symptom severity, socio-demographic characteristics) (Bozikas & Parlapani, Reference Bozikas and Parlapani2016; Mao & Agyapong, Reference Mao and Agyapong2021). However, these studies show only limited consistency in identifying resilience-associated characteristics in health and disease. This is mainly due to small samples, absence of validation tests, differences in terms of resilience assessment (Windle et al., Reference Windle, Bennett and Noyes2011), and the use of univariate statistics. A promising way to overcome these shortcomings would be to investigate the discriminative power of a wide variety of variables in determining resilience abilities in SCZ via machine learning. Indeed, machine learning allows quantifying sensitivity, specificity and generalizability of a given set of variables at the single-subject level (Antonucci et al., Reference Antonucci, Penzel, Pergola, Kambeitz-Ilankovic, Dwyer, Kambeitz and Koutsouleris2020a, Reference Antonucci, Pergola, Pigoni, Dwyer, Kambeitz-Ilankovic, Penzel and Bertolino2020b, Reference Antonucci, Raio, Pergola, Gelao, Papalino, Rampino and Bertolino2021), rather than just characterizing group differences.

Therefore, the aims of this study were (i) to investigate predictors of resilience to be potentially fostered even in chronic conditions like SCZ, using supervised machine learning at the single-subject level in a relatively large sample (Dwyer, Falkai, & Koutsouleris, Reference Dwyer, Falkai and Koutsouleris2018), and (ii) to use algorithmic information to characterize the clinical and psychological profile of patients with resilience abilities. With these aims, we generated a Support Vector Machine algorithm to classify patients with HR from those with LR based on various psychological, historical, clinical, and environmental variables. Furthermore, we investigated the potential clinical relevance of the decision scores derived from the algorithm by testing their correlation with the cognitive and clinical characteristics of patients. We hypothesized that patients with SCZ could be differentiated based on their resilience abilities within a machine learning framework, and that the differentiation performance of the machine learning algorithm would be associated with both cognition and symptom severity.

Materials and methods

Sample determination and related statistical analyses

This study was conducted as part of the activities of the Italian Network for Research on Psychoses (Galderisi et al., Reference Galderisi, Rossi, Rocca, Bertolino, Mucci and Bucci2014). Within this consortium, 896 patients with SCZ were recruited at 26 centers. The full list of study inclusion and exclusion criteria is reported in online Supplementary Information S1. Approval of the study protocol was obtained from the Local Ethics Committees of the participating centers. The dataset was randomly split, with a 2:1 ratio, into two samples: Discovery and Validation. Notably, in this random partition procedure, sites were entirely assigned, without any partitioning, either to the Discovery or to the Validation sample (i.e. all patients recruited at a given center were assigned either to the Discovery or to the Validation sample). After this procedure, our Discovery sample was composed of 598 patients recruited at 17 centers, and our Validation sample of 298 patients recruited at 9 different centers.

Each case was defined as having higher or LR, based on the total score obtained at the Resilience Scale for Adults (RSA), a self-reported questionnaire (Bonfiglio, Renati, Hjemdal, & Friborg, Reference Bonfiglio, Renati, Hjemdal and Friborg2016) (see online Supplementary Information S2). In the RSA, resilience is operationalized as ‘all the positive resources that may be activated in the context of stress to prevent the development of negative mental health outcomes’ (Morote et al., Reference Morote, Hjemdal, Martinez Uribe and Corveleyn2017). The positive resources to which such operationalization refers to, pertain to the following processes: (1) individual positive dispositional attributes, (2) family support and coherence, and (3) external support systems outside the family (Hjemdal et al., Reference Hjemdal, Vogel, Solem, Hagen and Stiles2011). Therefore, individuals with a higher RSA total score will be more likely to dynamically activate individual and contextual resources at their own advantage to cope with stressful situations.

The median of the total RSA score was calculated in the Discovery sample, and this value was used as a cut-off also in the Validation sample. Then, individuals in both the Discovery and Validation samples having a total RSA value higher than the Discovery median were assigned to the HR group, while those with a score lower than the Discovery median were assigned to the LR group. This was done in order to generate a supervised machine learning algorithm (see section 2.3) aimed at classifying higher v. lower observed (i.e. real) resilience levels, instead of algorithm-derived or algorithm-predicted resilience levels. This latter option, indeed, would be more typical of unsupervised – rather than supervised - machine learning strategies (e.g. clustering, partial least squares (Dwyer et al., Reference Dwyer, Falkai and Koutsouleris2018).

Sample size divided per recruiting center is reported in Table 1, while demographic and clinical characteristics of HR and LR patients are reported in Table 2. The Chi-square test (χ2) served to evaluate the potential effects of the center and pharmacological treatment on resilience levels. Two-sample t tests were employed to investigate potential demographic and clinical differences between HR and LR individuals within, respectively, Discovery and Validation cohorts. For all tests, the significant level was set at α = 0.05.

Table 1. Sample size of both Discovery (panel A) and Validation (panel B) cohorts, divided across centers

Abbreviations: LR, lower resilience; HR, higher resilience; N, numerosity, χ2, chi-square test.

Table 2. Demographic and clinical characteristics of the Discovery (panel A) and Validation (panel B) samples.

Abbreviations: F/M, female/male; LR, lower resilience; HR, higher resilience; s.d., standard deviation; N, numerosity; RSA, Resilience Scale for Adults; χ2, chi-square test.

Assessments

Besides RSA, patients were administered the MATRICS Consensus Cognitive Battery (MCCB) (Keefe et al., Reference Keefe, Fox, Harvey, Cucchiaro, Siu and Loebel2011) to assess cognitive abilities, and the Positive and Negative Syndrome Scale (PANSS) (Peralta & Cuesta, Reference Peralta and Cuesta1994), to assess symptom severity (see online Supplementary Information S3). Furthermore, patients underwent the following observer-rated and self-rated clinical, historical, environmental and psychological assessments, fully described in online Supplementary Information S3:

Total scores and subscale scores from these assessments (total number of variables = 85, online Supplementary Table S1) fed our machine learning algorithm, together with anamnestic information.

Machine learning pipeline

Discovery analyses

The overall machine learning strategy was carried out through the NeuroMiner software, version 1.0 (https://github.com/neurominer-git?tab=repositories). We generated an algorithm based on the 85 variables extracted from the afore-mentioned assessments (online Supplementary Table S1) able to classify patients as HR or LR. With this aim, we implemented a mixed inner k-fold/outer leave-site-out (Antonucci et al., Reference Antonucci, Penzel, Sanfelici, Pigoni, Kambeitz-Ilankovic, Dwyer and Consortium2022b) cross-validation (CV) strategy (see online Supplementary Information, section 4) (Antonucci et al., Reference Antonucci, Penzel, Pergola, Kambeitz-Ilankovic, Dwyer, Kambeitz and Koutsouleris2020a, Reference Antonucci, Pergola, Pigoni, Dwyer, Kambeitz-Ilankovic, Penzel and Bertolino2020b, Reference Antonucci, Taurino, Laera, Taurisano, Losole, Lutricuso and Puntillo2020c, Reference Antonucci, Raio, Pergola, Gelao, Papalino, Rampino and Bertolino2021; Koutsouleris et al., Reference Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef, Dwyer and Consortium2018, Reference Koutsouleris, Dwyer, Degenhardt, Maj, Urquijo-Castro, Sanfelici and Consortium2021a, Reference Koutsouleris, Worthington, Dwyer, Kambeitz-Ilankovic, Sanfelici, Fusar-Poli and Cannon2021b). This type of nested CV prevents information leaking between individuals used for training and testing the models (Ruschhaupt, Huber, Poustka, & Mansmann, Reference Ruschhaupt, Huber, Poustka and Mansmann2004). Indeed, we split the data first into training and test sets on an outer (CV2) cycle, and then we split the resulting training folds again into an inner (CV1), training and test data cycle (Antonucci et al., Reference Antonucci, Pergola, Pigoni, Dwyer, Kambeitz-Ilankovic, Penzel and Bertolino2020b; Varma & Simon, Reference Varma and Simon2006). Therefore, nested CV induces a strict separation between training and test data. Specifically, parameter optimization is performed within the inner (CV1) cycle, and generalization error estimation is performed only from the outer (CV2) cycle. This was done in order to identify models that contributed mostly to the classification pattern separating HR and LR (previously identified based on the median RSA total score, see section 2.1) at the inner CV level, in our Discovery sample. To enforce an unbiased estimation of classification generalizability, these models were then applied to the test data at the outer CV level in each Discovery recruiting center separately (e.g. leave-site-out). This choice intended to protect from the center effects on sample size distributions we have found (see Results 3.1 and Table 1), thus guaranteeing geographical generalization to centers unseen (e.g. left-out) during training. The outer CV level included individuals whose data were not used for training the classification algorithm (Ruschhaupt et al., Reference Ruschhaupt, Huber, Poustka and Mansmann2004). We obtained a Support Vector Machine (Noble, Reference Noble2006) ensemble model based on the 85 variables reported in online Supplementary Table S1. Model performance was measured using sensitivity, specificity, balanced accuracy (BAC), positive predictive value, negative predictive value, and Area under the curve (AUC) based on the class membership probability scores generated through ensemble-based majority voting in the repeated nested CV framework (see online Supplementary Information, section 4). We also assigned statistical significance to the observed classification performance of our model through permutation analysis (see online Supplementary, Information S4). Furthermore, to understand the importance of the input features for generating decisions (i.e. HR or LR patient?), we computed for each feature the probability of being selected for classification purposes within the inner CV loop [i.e. Cross-Validation Ratio (Antonucci et al., Reference Antonucci, Penzel, Pergola, Kambeitz-Ilankovic, Dwyer, Kambeitz and Koutsouleris2020a, Reference Antonucci, Pergola, Pigoni, Dwyer, Kambeitz-Ilankovic, Penzel and Bertolino2020b, Reference Antonucci, Raio, Pergola, Gelao, Papalino, Rampino and Bertolino2021)]. A detailed description of our machine learning pipeline is reported in online Supplementary Information S4.

Out-of-sample validation analyses

To investigate whether the machine learning model classifying patients as HR or LR had good generalizability, we performed out-of-sample validation analysis. We applied the SVM ensemble decision model obtained from the Discovery analysis without any in-between re-training steps to our Validation cohort, with totally unseen centers and patients, as already done in previous publications (Antonucci et al., Reference Antonucci, Penzel, Pergola, Kambeitz-Ilankovic, Dwyer, Kambeitz and Koutsouleris2020a, Reference Antonucci, Pergola, Pigoni, Dwyer, Kambeitz-Ilankovic, Penzel and Bertolino2020b, Reference Antonucci, Raio, Pergola, Gelao, Papalino, Rampino and Bertolino2021).

Sanity checks

Further, we performed some sanity checks to exclude that the accuracy of our algorithm could have been affected by potential confounds. More specifically, we investigated: (i) whether our machine learning algorithm was associated with the type of pharmacological treatment received by patients (i.e. typical antipsychotics, atypical antipsychotics, both, none), and (ii) whether despite the robust leave-site-out CV implemented, our algorithm could carry any latent recruiting center effect, thus showing poor stability. A full description of the methods of these sanity checks is reported in online Supplementary Information S5.

Correlation analysis

To investigate the clinical relevance of our algorithm besides resilience, we investigated the association between single-subject decision scores of our model and, respectively:

A more positive single-subject decision score suggests that a given individual is highly prototypical of the LR class according to the algorithm, while a more negative decision score suggests that a given individual is highly prototypical of the HR class.

We conducted Pearson's r correlations first in the Discovery sample. All significant p values were <0.05, False Discovery Rate (FDR) corrected (Benjamini & Hockberg, Reference Benjamini and Hockberg1995). FDR-corrected correlations found in the Discovery sample were also performed in the Validation sample. For the validation sample, the significance level was set at α = 0.05.

Results

Demographic and clinical differences

The χ2 test revealed significant inhomogeneity in the distributions of LR and HR individuals across centers in both cohorts (Discovery p = 0.004, Validation p = 0.001). In the Discovery cohort, HR and LR groups did not differ significantly in terms of demographic and clinical characteristics (all p > 0.05, Table 2). In the Validation cohort, HR individuals were significantly younger than LR (p = 0.039) and had significantly shorter illness duration (p = 0.007), probably because of their age. The distribution of pharmacological treatment in use was also significantly different across LR and HR individuals, but only in the Validation sample (p = 0.011).

Machine learning results: discovery sample

The machine learning algorithm based on the scores extracted from the assessments listed in online Supplementary Table S1 correctly classified patients as HR or LR with a cross-validated leave-site-out Balanced Accuracy (BAC) of 74.5% (sensitivity: 76.6%; specificity: 72.4%; AUC: 0.82) and was significant at p < 0.001. Detailed metrics are reported in Table 3.

Table 3. Validated classification performance of the Support Vector Machine algorithm trained within a repeated nested cross-validation framework on the assessments listed in section 1.2

Abbreviations: HR, higher resilience; LR, lower resilience; Neg, negative; Pos, positive; Sens, sensitivity; Spec, specificity; BAC, balanced accuracy; DOR, diagnostic odds ratio.

We observed that features with the highest Cross-Validation Ratio (i.e. those that were selected the most for classification purposes within the CV framework) included self-esteem total score (assessed with SERS), perceived social affective support (assessed with the SNQ), age at first relapse and use of the positive reframing coping strategy in stressful situations (assessed with COPE) (Fig. 1). Thus, these features were those most contributing to the accuracy of our HR v. LR algorithm, such that higher self-esteem, higher perceived social affective support, younger age at first relapse and more frequent use of positive reframing coping strategies were found more often in patients that the algorithm classified as having HR levels.

Fig. 1. Probability of each feature for being selected in the machine learning Cross-Validation framework. Scores closer to 1 represent a higher probability of being selected for decision by the Support Vector Machine algorithm. Abbreviations: CDSS, Calgary Depression Scale for Adults; SHRS, Saint Hans Rating Scale; SNQ, Social Network Questionnaire; SERS, Self Esteem Rating Scale.

Results of our sanity checks (section 2.4, fully reported in online Supplementary Information S5) revealed that our algorithm was not significantly associated with any type of pharmacological treatment and showed very good stability despite reshuffling of participants and centers.

Machine learning results: validation sample

Results from the out-of-sample validation analysis classifying patients as HR or LR and centers from the Validation sample based on the model built in Discovery patients (Results section 3.2) revealed that such model correctly classified patients as HR or LR also when applied to unseen patients and centers (BAC: 80.2%; sensitivity: 80.0%; specificity: 80.32%; AUC: 0.87, Table 3).

Correlation analysis

To investigate the potential clinical relevance of our algorithm, we tested whether its ability to classify patients with HR or LR correctly is correlated with cognitive and clinical characteristics of patients.

Cognitive abilities

Pearson's r analysis revealed significant negative correlations in the Discovery sample between SVM-related decision scores and the speed of processing domain score (r = −0.10, pFDR = 0.04), the reasoning/problem-solving domain score (r = −0.10, pFDR = 0.04), the global composite score (i.e, cognition + social cognition, r = −0.11, pFDR = 0.04), and the composite neurocognitive score (i.e. all MCCB scores minus the social cognition composite score, r = −0.11, pFDR = 0.04) of the MCCB (see section 2.5). The correlation between decision scores and MCCB global composite score was replicated in the Validation sample (r = −0.12, p = 0.04). The full list of correlation results for cognitive abilities is reported in Tables 4A and 4B.

Table 4. Correlation analysis between Support Vector Machine-based decision scores and cognitive abilities assessed through age, gender and education-corrected Z scores extracted from the MATRICS Consensus Cognitive Battery

Abbreviations: MCCB, MATRICS Consensus Cognitive Battery; MSCEIT, Mayer Salovey Caruso Emotional Intelligence Test; TASIT, The Awareness of Social Inference Test; FEIT, Face Emotion Recognition Task.

A more positive decision score suggests that a given individual is highly prototypical of the lower resilience class, while a more negative decision score suggests that a given individual is highly prototypical of the higher resilience class.

Notably, a more positive decision score suggests that a given individual is highly prototypical of the LR class, while a more negative decision score suggests that a given individual is highly prototypical of the HR class, according to the algorithm. Therefore, this negative correlation suggests that the more a subject is classified within the LR class, the lower is the MCCB global score.

Symptom severity

Pearson's r correlation analysis between SVM-related decision scores and PANSS scores (see section 2.5) revealed significant positive correlations in the discovery sample for all scores assessed (i.e. total, disorganization/cognitive, positive, negative, and general psychopathology, all pFDR < 0.001). Associations between decision scores and PANSS total, disorganization/cognitive and general psychopathology domain scores were replicated in the Validation sample (all p > 0.022). The full list of correlation results for symptom severity is reported in Tables 5A and 5B. These correlations suggest that the more a subject is classified within the LR class, the more severe are the symptoms.

Table 5. Correlation analysis between Support Vector Machine-based decision scores and symptom severity assessed by PANSS

Abbreviations: PANSS, Positive and Negative Syndrome Scale.

A more positive decision score suggests that a given individual is highly prototypical of the lower resilience class, while a more negative decision score suggests that a given individual is highly prototypical of the higher resilience class.

Discussion

In order to identify which clinical factors are associated with resilience at the single-subject level, we built a supervised machine learning algorithm to classify patients with HR from those with LR based on several psychological, historical, clinical and environmental variables. To do so, we pooled clinical/historical, psychological and environmental data collected at 26 independent Italian centers, into a robust machine learning framework, to derive rules of classification of patients with HR v. LR abilities.

Our results support the notion that HR and LR patients can be classified within a machine learning framework based on a parsimonious clinical-psychological signature. This signature proved accurate, stable, generalizable, and clinically relevant, given its association with the clinical and cognitive profile of patients. The good accuracy and sensitivity/specificity balance of the algorithm support our hypothesis regarding the possibility to reliably identify resilience predictors in patients with SCZ. The reliable identification of such predictors has important and intriguing implications for clinical practice, given evidence highlighting the tight connection between resilience and recovery in patients with SCZ), and the existence of associations between resilience levels and, respectively, psychosocial functioning (Galderisi et al., Reference Galderisi, Rucci, Mucci, Rossi, Rocca and Bertolino2020) and life skills (Zizolfi et al., Reference Zizolfi, Poloni, Caselli, Ielmini, Lucca, Diurni and Callegari2019) in patients. Therefore, it is possible to hypothesize that non-pharmacological interventions may successfully target residual resilience abilities in patients even in chronic conditions. Future studies are warranted to validate this hypothesis.

Our findings also suggest that, within our clinical-psychological signature, variability in resilience is successfully explained by few specific clinical/historical, psychological and environmental factors. More specifically, the factors that played a most prominent role in the decisions of our algorithm (i.e. those most frequently selected within our nested, leave-site-out CV framework), were self-esteem levels, perceived social affective support, age at first relapse, and the use of positive reframing as a stable coping strategy.

Self-esteem refers to the personal evaluation that anybody can make about her/his worth, which can span between very positive or very negative (Haug et al., Reference Haug, Melle, Andreassen, Raballo, Bratlien, Oie and Moller2012). Previous studies have shown that lower self-esteem levels are associated with depressive symptoms in both patients with full-blown psychosis (Karatzias, Gumley, Power, & O'Grady, Reference Karatzias, Gumley, Power and O'Grady2007; Smith et al., Reference Smith, Fowler, Freeman, Bebbington, Bashforth, Garety and Kuipers2006) and in individuals at clinical high risk (Bemrose, Akande, & Cullen, Reference Bemrose, Akande and Cullen2021; Benavides, Brucato, & Kimhy, Reference Benavides, Brucato and Kimhy2018). Other studies also found associations between lower self-esteem and psychotic symptoms (Jongeneel, Pot-Kolder, Counotte, van der Gaag, & Veling, Reference Jongeneel, Pot-Kolder, Counotte, van der Gaag and Veling2018; Romm et al., Reference Romm, Rossberg, Hansen, Haug, Andreassen and Melle2011).

Perceived social affective support was the second variable – and the only SNQ dimension - most frequently selected by the algorithm to generate decisions. Studies have shown that the possibility to ask for support within the social network in case of need is positively associated with resilience (Harvey & Boynton, Reference Harvey and Boynton2021; Satici, Kayis, & Akin, Reference Satici, Kayis and Akin2013). This is likely due to the fact that sharing feelings and everyday difficulties in the social network would likely reduce the repression of emotions (Magai, Consedine, Fiori, & King, Reference Magai, Consedine, Fiori and King2009). Interestingly, self-esteem plays a role in this association (Arango et al., Reference Arango, Dragioti, Solmi, Cortese, Domschke, Murray and Fusar-Poli2021; Bajaj et al., Reference Bajaj, Gupta and Pande2016). Indeed, in healthy individuals, self-esteem levels mediated the positive association between the possibility to share thoughts and experiences with others, and resilience abilities (Harvey & Boynton, Reference Harvey and Boynton2021). The fact that another frequently selected variable in our algorithm was the use of positive reframing as a coping strategy in face of difficult situations would be consistent with this view. Indeed, positive reframing reflects coping with stressors through positive re-thinking and subsequent re-interpreting of the stressful situation (Carver et al., Reference Carver, Scheier and Weintraub1989). In summary, having a larger social network would allow the individual to feel more comfortable in sharing feelings and thoughts, thus facilitating positive reframing of stressful situations and, consequently, self-esteem and sense of worth (Alparone, Pagliaro, & Rizzo, Reference Alparone, Pagliaro and Rizzo2015; Harvey & Boynton, Reference Harvey and Boynton2021; Pennebaker, Mayne, & Francis, Reference Pennebaker, Mayne and Francis1997; Satici et al., Reference Satici, Kayis and Akin2013).

Therefore, the variable selection process of our algorithm proves a good degree of internal coherence and provides intriguing views not only about psychological and environmental processes fostering resilience, but also about relationships between such processes, that could be considered when planning strategies of psychological support for patients. Future studies are warranted to statistically validate, for example via Structural Equation Modeling, the interplay between all the variables selected by the algorithm and resilience, in both healthy and clinical populations.

Notably, when the algorithm generated in the Discovery cohort was applied to unseen patients and new Network centers, it showed remarkable generalizability. Indeed, the same algorithm correctly classified LR and HR patients with high BAC also in the Validation cohort (i.e. 80.2%). Besides high generalizability, the algorithm also proved to be stable and not confounded by either pharmacological, or latent recruiting center or sampling confounding effects. Indeed, as our sanity checks showed (Section 2.4, online Supplementary Information S5), newly generated models based on random resampling of both Discovery and Validation cohorts all classified patients as HR or LR with a minimum BAC of 72.9%, and a maximum BAC of 77.5%. As validation and stability of decisions are both of pivotal importance when evaluating reliability and potential translation into clinical practice of an algorithm (Sanfelici, Dwyer, Antonucci, & Koutsouleris, Reference Sanfelici, Dwyer, Antonucci and Koutsouleris2020), we believe that our results argue in favor of the validity of the clinical-psychological signature identified.

These results could inform more effective psychological assessments in patients, by making them less burdening, and more focused on relevant targets. Indeed, SCZ patients are very heterogeneous from the point of view of psychological, clinical, and environmental risk factors which could play a role in SCZ risk pathways. Within such heterogeneity, our accurate and generalizable resilience signature informs clinical practice about the existence of a parsimonious space of resilience predictors which rely mainly on psychological attributes and social network characteristics. Therefore, such psychological and environmental evaluations may be prioritized in clinical assessment sessions to investigate to which extent there is space to promote positive personal abilities – such as resilience – which could support patients in better coping with the experience of the disease. In this framework, our findings may also inform a better definition of individualized psychological interventions aimed at enhancing resilience abilities in patients with SCZ, even in chronic conditions, through the implementation of tailored programs focusing on potentiating self-esteem, coping strategies, and social skills, especially in those who have experienced an earlier disease onset. However, due to the cross-sectional nature of our project, future studies are warranted to test such clinical implications within a longitudinal framework.

Findings from correlation analysis showed that our resilience signature carries relevant information also beyond the resilience framework, thus arguing in favor of its clinical relevance. More specifically, higher decision scores (i.e. the more a given individual was considered by the algorithm as being ‘lower resilience-like’) were associated with worse global level of socio-cognitive abilities in both Discovery and Validation cohorts. Deficits in neurocognitive and social cognition abilities are key characteristics of SCZ and are highly debilitating for patients, given their impact on daily life activities (Antonucci et al., Reference Antonucci, Pergola, Pigoni, Dwyer, Kambeitz-Ilankovic, Penzel and Bertolino2020b; Maj et al., Reference Maj, van Os, De Hert, Gaebel, Galderisi, Green and Ventura2021). These findings support a recent hypothesis stating that cognition may influence disease course in psychosis via resilience which, in turn, would mediate functional outcomes (Shrivastava, De Sousa, & Lodha, Reference Shrivastava, De Sousa and Lodha2019). However, this hypothesis has not been statistically validated yet.

On the other hand, higher decision scores were associated with higher total, general and disorganization/cognitive PANSS scores, thus suggesting that the more the algorithm classified a patient as ‘lower resilience’, the higher the total, general and disorganization/cognitive symptom severity, as assessed by PANSS. A recent study has highlighted that resilience could act as a mediator also between symptom severity and perceived everyday disability in patients with SCZ (Chen et al., Reference Chen, Xu, Mao, Sun, Sun and Zhou2019; Millier et al., Reference Millier, Schmidt, Angermeyer, Chauhan, Murthy, Toumi and Cadi-Soussi2014). Given this finding, the association between our resilience signature and symptom severity would suggest that the variables most frequently selected by our algorithm (e.g. self-esteem, positive reframing, richness of the social network) could play a role in such mediation framework. Again, Structural Equation Modeling studies will likely clarify the role, the direction, and the strength of associations within this interplay of variables.

Limitations

Our study has some limitations. First, the cross-sectional design of this study certainly limits the clinical implications of our findings and does not allow us to elaborate a prognostic hypothesis regarding the role of the resilience-associated variables we found for early identification and intervention strategies. Therefore, the generalizability of our algorithm is limited to the cross-sectional framework. Furthermore, our sample comprised chronic SCZ patients under stable antipsychotic treatment. We are aware that this issue may represent another potential limitation for the generalizability of our findings. Therefore, replication studies involving participants in earlier stages of the disease, with shorter treatment course (Ventura et al., Reference Ventura, Subotnik, Guzik, Hellemann, Gitlin, Wood and Nuechterlein2011), could provide further information about the reliability of our results. Another potential limitation is that within the Italian Network for Research on Psychoses, only Italian patients with SCZ have been recruited. Thus, the geographic validation of our algorithm is limited to the national level. Future studies are needed to further understand the generalizability of our findings within and outside Europe. Another important limitation of this study is that resilience was assessed only through the self-report RSA scale, as it was the only resilience measure available within the NIRP consortium. A recent review has pointed out the lack of a gold-standard measure of resilience. Indeed, nineteen measures of resilience have been identified so far, and each is based on a different operationalization of the resilience construct (Windle et al., Reference Windle, Bennett and Noyes2011). Despite RSA seems to be one of the most reliable resilience instruments according to its psychometric profile, high variability between instruments has led to inconsistencies related to the investigation of factors associated with resilience and to accurate prevalence estimates. Furthermore, the RSA and the rest of the reviewed instruments do not consider the adverse events that may have elicited resilience abilities, or the outcome from which resilience should ‘protect’ from. As recently pointed out (Kalisch et al., Reference Kalisch, Basten, Boks, Bonanno, Brummelman, Chmitorz and Kleim2017), to move to a truly process-oriented definition of resilience, future studies are warranted (i) to assess the construct validity of our results, thus replicating it through the use of other resilience measures, and (ii) to investigate resilience also in light of adverse events and outcomes.

Conclusions and future directions

We delivered an accurate, generalizable, stable, and meaningful signature classifying patients with SCZ as HR or LR based on a parsimonious set of psychological, clinical and environmental variables. Our signature showed clinical relevance also beyond resilience, given its association with the clinical and cognitive profile of patients. Furthermore, the algorithm provided relevant information, albeit of cross-sectional nature, regarding psychological, clinical and environmental factors modulating residual resilience in patients with SCZ, which could thus be possibly targeted within individualized non-pharmacological interventions.

Supplementary material

The supplementary material for this article can be found at https://doi.org/10.1017/S003329172200294X

Author contributions

LAA = Writing – original draft, review & editing, Conceptualization, Investigation, Methodology; GP = Writing – review & editing, Methodology; AR1, PR, AR2, MA, EA, AB, VB, CB, PB, BC, LDO, FDF, MDG, AF, GMG, MM, CM, PM1, PM2, MP, RR, AS, AV, PZ = Data Curation, Writing – review & editing; SG, MM = Funding Acquisition, Supervision, Writing – review & editing; AB = Conceptualization, Funding Acquisition, Supervision, Writing- review & editing.

Financial support

This paper reflects only the author's views. The funding bodies had no role in study design, data collection, and analysis, decision to publish, or preparation of the manuscript.

Conflict of interest

Dr Galderisi reported receiving personal fees from Angelini Pharma, Gedeon Richter-Recordati, Innova Pharma-Recordati Group, Janssen Pharmaceutica NV, Lundbeck Italia, Millennium Pharmaceutical, and Sunovion Pharmaceuticals outside the submitted work. Dr Bertolino reported receiving grants and personal fees from Lundbeck and receiving personal fees from Janssen and from Otsuka during the conduct of the study. Dr Pompili reported receiving personal fees from Allergan, Angelini Pharma, FB Health Janssen, Lundbeck, and Otsuka Pharmaceutical. All other authors declare no biomedical financial interests and no potential conflicts of interest.

Ethical standards

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.

Appendix

Members of the Italian Network for Research on Psychoses involved in this study include: Ileana Andriola, Giuseppe Blasi, Laura De Mastro, Enrico D'Ambrosio, Francesco Massari, Alessandra Raio, Marianna Russo, Pierluigi Selvaggi, Angelantonio Tavella (University of Bari); Stefano Barlati, Giacomo Deste, Jacopo Lisoni (University of Brescia); Federica Pinna, Pasquale Paribello, Luca Marras (University of Cagliari); Giuseppe Piegari, Francesco Brando, Luigi Giuliani, Pasquale Pezzella (University of Campania ‘Luigi Vanvitelli’, Naples); Carmen Concerto, Laura Fusar Poli, Alessandro Rodolico (University of Catania); Giovanni Martinotti, Mauro Pettorruso, Stefano Sensi (University of Chieti); Mario Altamura, Livia Ficarella, Simona Biancofiore (University of Foggia); Pietro Calcagno, Valeria Placenti, Gianluca Serafini (University of Genoa); Rita Roncone, Laura Giusti, Silvia Mammarella, Ramona Di Stefano, Arianna Di Berardo, Luca Bonanni (University of L'Aquila); Carla Gramaglia, Eleonora Gambaro, Eleonora Gattoni (University of Eastern Piedmont, Novara); Angela Favaro, Elena Tenconi, Enrico Collantoni (University of Padua); Matteo Tonna, Paolo Ossola, Maria Lidia Gerra (University of Parma); Claudia Carmassi, Barbara Carpita, Ivan Mirko Cremone (University of Pisa); Giammarco Cascino, Giulio Corrivetti, Gianfranco Del Buono (Department of Mental Health, Salerno); Tommaso Accinni, Marianna Frascarelli, Antonino Buzzanca, Anna Comparelli, Roberto Brugnoli, Valentina Corigliano (‘La Sapienza’ University of Rome); Simone Bolognesi, Alessandro Cuomo, Arianna Goracci (University of Siena); Giorgio Di Lorenzo, Cinzia Niolu, Michele Ribolsi (‘Tor Vergata’ University of Rome); Silvio Bellino, Paola Bozzatello, Cristiana Montemagni (University of Turin).

Footnotes

*

The members of the Italian Network for Research on Psychoses involved in this study are listed in the Appendix.

References

Alparone, F. R., Pagliaro, S., & Rizzo, I. (2015). The words to tell their own pain: Linguistic markers of cognitive reappraisal in mediating benefits of expressive writing. Journal of Social and Clinical Psychology, 34(6), 495507. https://doi.org/10.1521/jscp.2015.34.6.495.CrossRefGoogle Scholar
Antonucci, L. A., Fazio, L., Pergola, G., Blasi, G., Stolfa, G., Di Palo, P., … Italian Network for Research on, P. (2022a). Joint structural-functional magnetic resonance imaging features are associated with diagnosis and real-world functioning in patients with schizophrenia. Schizophrenia Research, 240, 193203. doi: 10.1016/j.schres.2021.12.029CrossRefGoogle ScholarPubMed
Antonucci, L. A., Penzel, N., Pergola, G., Kambeitz-Ilankovic, L., Dwyer, D., Kambeitz, J., … Koutsouleris, N. (2020a). Multivariate classification of schizophrenia and its familial risk based on load-dependent attentional control brain functional connectivity. Neuropsychopharmacology, 45(4), 613621. doi: doi:10.1038/s41386-019-0532-3CrossRefGoogle ScholarPubMed
Antonucci, L. A., Penzel, N., Sanfelici, R., Pigoni, A., Kambeitz-Ilankovic, L., Dwyer, D., … Consortium, P. (2022b). Using combined environmental-clinical classification models to predict role functioning outcome in clinical high-risk states for psychosis and recent-onset depression. The British Journal of Psychiatry, 117. doi: 10.1192/bjp.2022.16Google ScholarPubMed
Antonucci, L. A., Pergola, G., Pigoni, A., Dwyer, D., Kambeitz-Ilankovic, L., Penzel, N., … Bertolino, A. (2020b). A pattern of cognitive deficits stratified for genetic and environmental risk reliably classifies patients with schizophrenia from healthy control subjects. Biological Psychiatry, 87(8), 697707. doi: 10.1016/j.biopsych.2019.11.007CrossRefGoogle ScholarPubMed
Antonucci, L. A., Raio, A., Pergola, G., Gelao, B., Papalino, M., Rampino, A., … Bertolino, A. (2021). Machine learning-based ability to classify psychosis and early stages of disease through parenting and attachment-related variables is associated with social cognition. BMC Psychology, 9(1), 47. doi: doi:10.1186/s40359-021-00552-3CrossRefGoogle ScholarPubMed
Antonucci, L. A., Taurino, A., Laera, D., Taurisano, P., Losole, J., Lutricuso, S., … Puntillo, F. (2020c). An ensemble of psychological and physical health indices discriminates between individuals with chronic pain and healthy controls with high reliability: A machine learning study. Pain and Therapy, 9(2), 601614. doi: 10.1007/s40122-020-00191-3CrossRefGoogle ScholarPubMed
Arango, C., Dragioti, E., Solmi, M., Cortese, S., Domschke, K., Murray, R. M., … Fusar-Poli, P. (2021). Risk and protective factors for mental disorders beyond genetics: An evidence-based atlas. World Psychiatry, 20(3), 417436. doi: 10.1002/wps.20894CrossRefGoogle ScholarPubMed
Bajaj, B., Gupta, R., & Pande, N. (2016). Self-esteem mediates the relationship between mindfulness and well-being. Personality and Individual Differences, 94, 96100. https://doi.org/10.1016/j.paid.2016.01.020.CrossRefGoogle Scholar
Bemrose, H. V., Akande, I. O., & Cullen, A. E. (2021). Self-esteem in individuals at ultra-high risk for psychosis: A systematic review and meta-analysis. Early Interventions in Psychiatry, 15(4), 775786. doi: 10.1111/eip.13034CrossRefGoogle ScholarPubMed
Benavides, C., Brucato, G., & Kimhy, D. (2018). Self-esteem and symptoms in individuals at clinical high risk for psychosis. The Journal of Nervous and Mental Disease, 206(6), 433438. doi: 10.1097/NMD.0000000000000824CrossRefGoogle ScholarPubMed
Benjamini, Y., & Hockberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological, 57(1), 289300.Google Scholar
Bonfiglio, N. S., Renati, R., Hjemdal, O., & Friborg, O. (2016). The resilience scale for adults in Italy: A validation study comparing clinical substance abusers with a nonclinical sample. Psychology of Addictive Behaviors, 30(4), 509515. doi: 10.1037/adb0000176CrossRefGoogle ScholarPubMed
Bozikas, V., & Parlapani, E. (2016). Resilience in patients with psychotic disorder. Psychiatrike = Psychiatriki, 27(1), 1316. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/27110878.Google ScholarPubMed
Carver, C. S., Scheier, M. F., & Weintraub, J. K. (1989). Assessing coping strategies: A theoretically based approach. Journal of Personality and Social Psychology, 56(2), 267283. doi: 10.1037//0022-3514.56.2.267CrossRefGoogle ScholarPubMed
Chen, H., Xu, J., Mao, Y., Sun, L., Sun, Y., & Zhou, Y. (2019). Positive coping and resilience as mediators between negative symptoms and disability Among patients with schizophrenia. Frontiers in Psychiatry, 10, 641. doi: 10.3389/fpsyt.2019.00641CrossRefGoogle ScholarPubMed
Chen, Y. L., & Kuo, P. H. (2020). Effects of perceived stress and resilience on suicidal behaviors in early adolescents. European Child and Adolescent Psychiatry, 29(6), 861870. doi: 10.1007/s00787-019-01401-wCrossRefGoogle ScholarPubMed
Choi, J. W., Cha, B., Jang, J., Park, C. S., Kim, B. J., Lee, C. S., & Lee, S. J. (2015). Resilience and impulsivity in euthymic patients with bipolar disorder. Journal of Affective Disorders, 170, 172177. doi: 10.1016/j.jad.2014.08.056CrossRefGoogle ScholarPubMed
Cohen, I. L., Schmidt-Lackner, S., Romanczyk, R., & Sudhalter, V. (2003). The PDD behavior inventory: A rating scale for assessing response to intervention in children with pervasive developmental disorder. Journal of Autism and Developmental Disorders, 33(1), 3145. doi: 10.1023/a:1022226403878CrossRefGoogle ScholarPubMed
Crespo-Facorro, B., Such, P., Nylander, A. G., Madera, J., Resemann, H. K., Worthington, E., … Newton, R. (2021). The burden of disease in early schizophrenia - a systematic literature review. Current Medical Research and Opinion, 37(1), 109121. doi: 10.1080/03007995.2020.1841618CrossRefGoogle ScholarPubMed
Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018). Machine learning approaches for clinical psychology and psychiatry. Annual Review of Clinical Psychology, 14, 91118. doi: 10.1146/annurev-clinpsy-032816-045037CrossRefGoogle ScholarPubMed
Feder, A., Fred-Torres, S., Southwick, S. M., & Charney, D. S. (2019). The biology of human resilience: Opportunities for enhancing resilience across the life span. Biological Psychiatry, 86(6), 443453. doi: 10.1016/j.biopsych.2019.07.012CrossRefGoogle ScholarPubMed
Fusar-Poli, P., Correll, C. U., Arango, C., Berk, M., Patel, V., & Ioannidis, J. P. A. (2021). Preventive psychiatry: A blueprint for improving the mental health of young people. World Psychiatry, 20(2), 200221. doi: 10.1002/wps.20869CrossRefGoogle ScholarPubMed
Galderisi, S., Mucci, A., Dollfus, S., Nordentoft, M., Falkai, P., Kaiser, S., … Gaebel, W. (2021). EPA Guidance on assessment of negative symptoms in schizophrenia. European Psychiatry, 64(1), e23. doi: 10.1192/j.eurpsy.2021.11CrossRefGoogle ScholarPubMed
Galderisi, S., Rossi, A., Rocca, P., Bertolino, A., Mucci, A., Bucci, P., … Italian Network For Research on, P. (2014). The influence of illness-related variables, personal resources and context-related factors on real-life functioning of people with schizophrenia. World Psychiatry, 13(3), 275287. doi: 10.1002/wps.20167CrossRefGoogle ScholarPubMed
Galderisi, S., Rucci, P., Mucci, A., Rossi, A., Rocca, P., Bertolino, A., … Italian Network for Research on, P. (2020). The interplay among psychopathology, personal resources, context-related factors and real-life functioning in schizophrenia: Stability in relationships after 4 years and differences in network structure between recovered and non-recovered patients. World Psychiatry, 19(1), 8191. doi: 10.1002/wps.20700CrossRefGoogle ScholarPubMed
Gerlach, J., Korsgaard, S., Clemmesen, P., Lauersen, A. M., Magelund, G., Noring, U., … Casey, D. E. (1993). The St. Hans rating scale for extrapyramidal syndromes: Reliability and validity. Acta Psychiatrica Scandinavica, 87(4), 244252. doi: 10.1111/j.1600-0447.1993.tb03366.xCrossRefGoogle ScholarPubMed
Gruber, M., Rumpold, T., Schrank, B., Sibitz, I., Otzelberger, B., Jahn, R., … Unger, A. (2018). Recover recovery style from psychosis: A psychometric evaluation of the German version of the Recovery Style Questionnaire (RSQ). Epidemiology and Psychiatric Sciences, 29, e4. doi: 10.1017/S2045796018000471CrossRefGoogle ScholarPubMed
Haeffel, G. J., & Grigorenko, E. L. (2007). Cognitive vulnerability to depression: Exploring risk and resilience. Child and Adolescent Psychiatric Clinics of North America, 16(2), 435448, x. doi: 10.1016/j.chc.2006.11.005CrossRefGoogle ScholarPubMed
Harvey, J., & Boynton, K. (2021). Self-disclosure and psychological resilience: The mediating roles of self-esteem and self-compassion. Interpersona, 15(1), 90104. https://doi.org/10.5964/ijpr.4533.CrossRefGoogle Scholar
Haug, E., Melle, I., Andreassen, O. A., Raballo, A., Bratlien, U., Oie, M., … Moller, P. (2012). The association between anomalous self-experience and suicidality in first-episode schizophrenia seems mediated by depression. Comprehensive Psychiatry, 53(5), 456460. doi: 10.1016/j.comppsych.2011.07.005CrossRefGoogle ScholarPubMed
Hjemdal, O., Vogel, P. A., Solem, S., Hagen, K., & Stiles, T. C. (2011). The relationship between resilience and levels of anxiety, depression, and obsessive-compulsive symptoms in adolescents. Clinical Psychology & Psychotherapy, 18(4), 314321. doi: 10.1002/cpp.719CrossRefGoogle ScholarPubMed
Iimura, S., & Taku, K. (2018). Gender differences in relationship between resilience and big five personality traits in Japanese adolescents. Psychological Reports, 121(5), 920931. doi: 10.1177/0033294117741654CrossRefGoogle ScholarPubMed
Jongeneel, A., Pot-Kolder, R., Counotte, J., van der Gaag, M., & Veling, W. (2018). Self-esteem moderates affective and psychotic responses to social stress in psychosis: A virtual reality study. Schizophrenia Research, 202, 8085. doi: 10.1016/j.schres.2018.06.042CrossRefGoogle ScholarPubMed
Kalisch, R. B. D., Basten, U., Boks, M. P., Bonanno, G. A., Brummelman, E., Chmitorz, A., … Kleim, B. (2017). The resilience framework as a strategy to combat stress-related disorders. Nature Human Behaviour, 1, 784790. doi: 10.1038/s41562-017-0200-8CrossRefGoogle ScholarPubMed
Karatzias, T., Gumley, A., Power, K., & O'Grady, M. (2007). Illness appraisals and self-esteem as correlates of anxiety and affective comorbid disorders in schizophrenia. Comprehensive Psychiatry, 48(4), 371375. doi: 10.1016/j.comppsych.2007.02.005CrossRefGoogle ScholarPubMed
Keefe, R. S., Fox, K. H., Harvey, P. D., Cucchiaro, J., Siu, C., & Loebel, A. (2011). Characteristics of the MATRICS consensus cognitive battery in a 29-site antipsychotic schizophrenia clinical trial. Schizophrenia Research, 125(2-3), 161168. doi: 10.1016/j.schres.2010.09.015CrossRefGoogle Scholar
Kim, K. R., Song, Y. Y., Park, J. Y., Lee, E. H., Lee, M., Lee, S. Y., … Kwon, J. S. (2013). The relationship between psychosocial functioning and resilience and negative symptoms in individuals at ultra-high risk for psychosis. Australian and New Zealand Journal of Psychiatry, 47(8), 762771. doi: 10.1177/0004867413488218CrossRefGoogle ScholarPubMed
Kobau, R., Seligman, M. E., Peterson, C., Diener, E., Zack, M. M., Chapman, D., & Thompson, W. (2011). Mental health promotion in public health: Perspectives and strategies from positive psychology. American Journal of Public Health, 101(8), e1e9. doi: 10.2105/AJPH.2010.300083CrossRefGoogle ScholarPubMed
Koutsouleris, N., Dwyer, D. B., Degenhardt, F., Maj, C., Urquijo-Castro, M. F., Sanfelici, R., … Consortium, P. (2021a). Multimodal machine learning workflows for prediction of psychosis in patients with clinical high-risk syndromes and recent-onset depression. JAMA Psychiatry, 78(2), 195209. doi: 10.1001/jamapsychiatry.2020.3604CrossRefGoogle ScholarPubMed
Koutsouleris, N., Kambeitz-Ilankovic, L., Ruhrmann, S., Rosen, M., Ruef, A., Dwyer, D. B., … Consortium, P. (2018). Prediction models of functional outcomes for individuals in the clinical high-risk state for psychosis or with recent-onset depression: A multimodal, multisite machine learning analysis. JAMA Psychiatry, 75(11), 11561172. doi: 10.1001/jamapsychiatry.2018.2165CrossRefGoogle ScholarPubMed
Koutsouleris, N., Worthington, M., Dwyer, D. B., Kambeitz-Ilankovic, L., Sanfelici, R., Fusar-Poli, P., … Cannon, T. D. (2021b). Toward generalizable and transdiagnostic tools for psychosis prediction: An independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort. Biological Psychiatry, 90(9), 632642. doi: 10.1016/j.biopsych.2021.06.023CrossRefGoogle ScholarPubMed
Lecomte, T., Corbiere, M., & Laisne, F. (2006). Investigating self-esteem in individuals with schizophrenia: Relevance of the self-esteem rating scale-short form. Psychiatry Research, 143(1), 99108. doi: 10.1016/j.psychres.2005.08.019CrossRefGoogle ScholarPubMed
Magai, C., Consedine, N. S., Fiori, K. L., & King, A. R. (2009). Sharing the good, sharing the bad: The benefits of emotional self-disclosure among middle-aged and older adults. Journal of Aging and Health, 21(2), 286313. doi: 10.1177/0898264308328980CrossRefGoogle ScholarPubMed
Maj, M., van Os, J., De Hert, M., Gaebel, W., Galderisi, S., Green, M. F., … Ventura, J. (2021). The clinical characterization of the patient with primary psychosis aimed at personalization of management. World Psychiatry, 20(1), 433. doi: 10.1002/wps.20809CrossRefGoogle ScholarPubMed
Mao, W., & Agyapong, V. I. O. (2021). The role of social determinants in mental health and resilience after disasters: Implications for public health policy and practice. Frontiers of Public Health, 9, 658528. doi: 10.3389/fpubh.2021.658528CrossRefGoogle ScholarPubMed
Mausbach, B. T., Bowie, C. R., Harvey, P. D., Twamley, E. W., Goldman, S. R., Jeste, D. V., & Patterson, T. L. (2008). Usefulness of the UCSD performance-based skills assessment (UPSA) for predicting residential independence in patients with chronic schizophrenia. Journal of Psychiatric Research, 42(4), 320327. doi: 10.1016/j.jpsychires.2006.12.008CrossRefGoogle ScholarPubMed
Mausbach, B. T., Depp, C. A., Bowie, C. R., Harvey, P. D., McGrath, J. A., Thronquist, M. H., … Patterson, T. L. (2011). Sensitivity and specificity of the UCSD Performance-based Skills Assessment (UPSA-B) for identifying functional milestones in schizophrenia. Schizophrenia Research, 132(2-3), 165170. doi: 10.1016/j.schres.2011.07.022CrossRefGoogle ScholarPubMed
Mausbach, B. T., Harvey, P. D., Pulver, A. E., Depp, C. A., Wolyniec, P. S., Thornquist, M. H., … Patterson, T. L. (2010). Relationship of the brief UCSD Performance-based Skills Assessment (UPSA-B) to multiple indicators of functioning in people with schizophrenia and bipolar disorder. Bipolar Disorders, 12(1), 4555. doi: 10.1111/j.1399-5618.2009.00787.xCrossRefGoogle ScholarPubMed
Mayoral-van Son, J., Juncal-Ruiz, M., Ortiz-Garcia de la Foz, V., Cantarero-Prieto, D., Blazquez-Fernandez, C., Paz-Zulueta, M., … Crespo-Facorro, B. (2019). Understanding the direct and indirect costs of a first episode of psychosis program: Insights from PAFIP of Cantabria, Spain, during the first year of intervention. Early Interventions in Psychiatry, 13(5), 11821190. doi: 10.1111/eip.12752CrossRefGoogle ScholarPubMed
Millier, A., Schmidt, U., Angermeyer, M. C., Chauhan, D., Murthy, V., Toumi, M., & Cadi-Soussi, N. (2014). Humanistic burden in schizophrenia: A literature review. Journal of Psychiatric Research, 54, 8593. doi: 10.1016/j.jpsychires.2014.03.021CrossRefGoogle ScholarPubMed
Morote, R., Hjemdal, O., Martinez Uribe, P., & Corveleyn, J. (2017). Psychometric properties of the Resilience Scale for Adults (RSA) and its relationship with life-stress, anxiety and depression in a Hispanic Latin-American community sample. PLoS One, 12(11), e0187954. doi: 10.1371/journal.pone.0187954CrossRefGoogle Scholar
Mucci, A., Rucci, P., Rocca, P., Bucci, P., Gibertoni, D., Merlotti, E., & Italian Network for Research on, P. (2014). The specific level of functioning scale: Construct validity, internal consistency and factor structure in a large Italian sample of people with schizophrenia living in the community. Schizophrenia Research, 159(1), 144150. doi: 10.1016/j.schres.2014.07.044CrossRefGoogle Scholar
Noble, W. S. (2006). What is a support vector machine? Nature Biotechnology, 24(12), 15651567. doi: 10.1038/nbt1206-1565CrossRefGoogle ScholarPubMed
Pennebaker, J. W., Mayne, T. J., & Francis, M. E. (1997). Linguistic predictors of adaptive bereavement. Journal of Personality and Social Psychology, 72(4), 863871. doi: 10.1037//0022-3514.72.4.863CrossRefGoogle ScholarPubMed
Peralta, V., & Cuesta, M. J. (1994). Psychometric properties of the positive and negative syndrome scale (PANSS) in schizophrenia. Psychiatry Research, 53(1), 3140. doi: 10.1016/0165-1781(94)90093-0CrossRefGoogle ScholarPubMed
Ribe, J. M., Salamero, M., Perez-Testor, C., Valero, A., & Garcia, M. C. (2015). Spanish version of the social network questionnaire in schizophrenia patients' caregivers. Schizophrenia Research, 164(1–3), 273274. doi: 10.1016/j.schres.2015.03.008CrossRefGoogle ScholarPubMed
Ritsher, J. B., Otilingam, P. G., & Grajales, M. (2003). Internalized stigma of mental illness: Psychometric properties of a new measure. Psychiatry Research, 121(1), 3149. doi: 10.1016/j.psychres.2003.08.008CrossRefGoogle ScholarPubMed
Romm, K. L., Rossberg, J. I., Hansen, C. F., Haug, E., Andreassen, O. A., & Melle, I. (2011). Self-esteem is associated with premorbid adjustment and positive psychotic symptoms in early psychosis. BMC Psychiatry, 11, 136. doi: 10.1186/1471-244X-11-136CrossRefGoogle ScholarPubMed
Ruschhaupt, M., Huber, W., Poustka, A., & Mansmann, U. (2004). A compendium to ensure computational reproducibility in high-dimensional classification tasks. Statistical Applications in Genetics and Molelcular Biology, 3, Article37, Epub 2004 Dec 19, 1-24. doi:10.2202/1544-6115.1078.Google ScholarPubMed
Rutten, B. P., Hammels, C., Geschwind, N., Menne-Lothmann, C., Pishva, E., Schruers, K., … Wichers, M. (2013). Resilience in mental health: Linking psychological and neurobiological perspectives. Acta Psychiatrica Scandinavica, 128(1), 320. doi: 10.1111/acps.12095CrossRefGoogle ScholarPubMed
Sanfelici, R., Dwyer, D. B., Antonucci, L. A., & Koutsouleris, N. (2020). Individualized diagnostic and prognostic models for patients with psychosis risk syndromes: A meta-analytic view on the state of the art. Biological Psychiatry, 88(4), 349360. doi: 10.1016/j.biopsych.2020.02.009CrossRefGoogle ScholarPubMed
Satici, S. A., Kayis, A. R., & Akin, A. (2013). Predictive role of authenticity on psychological vulnerability in Turkish university students. Psychological Reports, 112(2), 519528. doi: 10.2466/02.07.PR0.112.2.519-528CrossRefGoogle ScholarPubMed
Schennach-Wolff, R., Obermeier, M., Seemuller, F., Jager, M., Messer, T., Laux, G., … Riedel, M. (2011). Evaluating depressive symptoms and their impact on outcome in schizophrenia applying the Calgary Depression Scale. Acta Psychiatrica Scandinavica, 123(3), 228238. doi: 10.1111/j.1600-0447.2010.01608.xCrossRefGoogle ScholarPubMed
Seery, M. D., Holman, E. A., & Silver, R. C. (2010). Whatever does not kill us: Cumulative lifetime adversity, vulnerability, and resilience. Journal of Personality and Social Psychology, 99(6), 10251041. doi: 10.1037/a0021344CrossRefGoogle Scholar
Shapero, B. G., Farabaugh, A., Terechina, O., DeCross, S., Cheung, J. C., Fava, M., & Holt, D. J. (2019). Understanding the effects of emotional reactivity on depression and suicidal thoughts and behaviors: Moderating effects of childhood adversity and resilience. Journal of Affective Disorders, 245, 419427. doi: 10.1016/j.jad.2018.11.033CrossRefGoogle ScholarPubMed
Shapiro, D. I., Marenco, S., Spoor, E. H., Egan, M. F., Weinberger, D. R., & Gold, J. M. (2009). The premorbid adjustment scale as a measure of developmental compromise in patients with schizophrenia and their healthy siblings. Schizophrenia Research, 112(1-3), 136142. doi: 10.1016/j.schres.2009.04.007CrossRefGoogle ScholarPubMed
Shrivastava, A., De Sousa, A., & Lodha, P. (2019). Resilience as a psychopathological construct for psychiatric disorders. Advances in Experimental Medicine and Biology, 1192, 479489. doi: 10.1007/978-981-32-9721-0_23CrossRefGoogle ScholarPubMed
Simeon, D., Yehuda, R., Cunill, R., Knutelska, M., Putnam, F. W., & Smith, L. M. (2007). Factors associated with resilience in healthy adults. Psychoneuroendocrinology, 32(8-10), 11491152. doi: 10.1016/j.psyneuen.2007.08.005CrossRefGoogle ScholarPubMed
Skrove, M., Romundstad, P., & Indredavik, M. S. (2013). Resilience, lifestyle and symptoms of anxiety and depression in adolescence: The Young-HUNT study. Social Psychiatry and Psychiatric Epidemiology, 48(3), 407416. doi: 10.1007/s00127-012-0561-2CrossRefGoogle ScholarPubMed
Smith, B., Fowler, D. G., Freeman, D., Bebbington, P., Bashforth, H., Garety, P., … Kuipers, E. (2006). Emotion and psychosis: Links between depression, self-esteem, negative schematic beliefs and delusions and hallucinations. Schizophrenia Research, 86(1-3), 181188. doi: 10.1016/j.schres.2006.06.018CrossRefGoogle ScholarPubMed
Smith, P. R. (2009). Resilience: Resistance factor for depressive symptom. Journal of Psychiatric and Mental Health Nursing, 16(9), 829837. doi: 10.1111/j.1365-2850.2009.01463.xCrossRefGoogle ScholarPubMed
Southwick, S. M., Sippel, L., Krystal, J., Charney, D., Mayes, L., & Pietrzak, R. (2016). Why are some individuals more resilient than others: The role of social support. World Psychiatry, 15(1), 7779. doi: 10.1002/wps.20282CrossRefGoogle ScholarPubMed
Tait, L., Birchwood, M., & Trower, P. (2002). A new scale (SES) to measure engagement with community mental health services. Journal of Mental Health, 11(2), 191198. doi: 10.1080/09638230020023570-2CrossRefGoogle ScholarPubMed
Tait, L., Birchwood, M., & Trower, P. (2004). Adapting to the challenge of psychosis: Personal resilience and the use of sealing-over (avoidant) coping strategies. The British Journal of Psychiatry, 185, 410415. doi: 10.1192/bjp.185.5.410CrossRefGoogle Scholar
Tandon, R., Nasrallah, H. A., & Keshavan, M. S. (2009). Schizophrenia, “just the facts” 4. Clinical features and conceptualization. Schizophrenia Research, 110(1–3), 123. doi: 10.1016/j.schres.2009.03.005CrossRefGoogle Scholar
Torgalsboen, A. K. (2012). Sustaining full recovery in schizophrenia after 15 years: Does resilience matter? Clinical Schizophrenia & Related Psychoses, 5(4), 193200. doi: 10.3371/CSRP.5.4.3CrossRefGoogle ScholarPubMed
Ungar, M., & Theron, L. (2020). Resilience and mental health: How multisystemic processes contribute to positive outcomes. The Lancet. Psychiatry, 7(5), 441448. doi: 10.1016/S2215-0366(19)30434-1CrossRefGoogle ScholarPubMed
Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7, 91. doi: 10.1186/1471-2105-7-91CrossRefGoogle ScholarPubMed
Ventura, J., Subotnik, K. L., Guzik, L. H., Hellemann, G. S., Gitlin, M. J., Wood, R. C., & Nuechterlein, K. H. (2011). Remission and recovery during the first outpatient year of the early course of schizophrenia. Schizophrenia Research, 132(1), 1823. doi: 10.1016/j.schres.2011.06.025CrossRefGoogle ScholarPubMed
Wallwork, R. S., Fortgang, R., Hashimoto, R., Weinberger, D. R., & Dickinson, D. (2012). Searching for a consensus five-factor model of the positive and negative syndrome scale for schizophrenia. Schizophrenia Research, 137(1-3), 246250. doi: 10.1016/j.schres.2012.01.031CrossRefGoogle ScholarPubMed
Windle, G., Bennett, K. M., & Noyes, J. (2011). A methodological review of resilience measurement scales. Health and Quality of Life Outcomes, 9, 8. doi: 10.1186/1477-7525-9-8CrossRefGoogle ScholarPubMed
Zizolfi, D., Poloni, N., Caselli, I., Ielmini, M., Lucca, G., Diurni, M., … Callegari, C. (2019). Resilience and recovery style: A retrospective study on associations among personal resources, symptoms, neurocognition, quality of life and psychosocial functioning in psychotic patients. Psychology Research and Behavior Management, 12, 385395. doi: 10.2147/PRBM.S205424CrossRefGoogle Scholar
Figure 0

Table 1. Sample size of both Discovery (panel A) and Validation (panel B) cohorts, divided across centers

Figure 1

Table 2. Demographic and clinical characteristics of the Discovery (panel A) and Validation (panel B) samples.

Figure 2

Table 3. Validated classification performance of the Support Vector Machine algorithm trained within a repeated nested cross-validation framework on the assessments listed in section 1.2

Figure 3

Fig. 1. Probability of each feature for being selected in the machine learning Cross-Validation framework. Scores closer to 1 represent a higher probability of being selected for decision by the Support Vector Machine algorithm. Abbreviations: CDSS, Calgary Depression Scale for Adults; SHRS, Saint Hans Rating Scale; SNQ, Social Network Questionnaire; SERS, Self Esteem Rating Scale.

Figure 4

Table 4. Correlation analysis between Support Vector Machine-based decision scores and cognitive abilities assessed through age, gender and education-corrected Z scores extracted from the MATRICS Consensus Cognitive Battery

Figure 5

Table 5. Correlation analysis between Support Vector Machine-based decision scores and symptom severity assessed by PANSS

Supplementary material: File

Antonucci et al. supplementary material

Antonucci et al. supplementary material
Download Antonucci et al. supplementary material(File)
File 133.8 KB