Data

Data were collected within the scope of the RISEUP-PPD-COVID-19 cross-sectional study (ClinicalTrials.gov registration: NCT04595123) conducted in 13 countries (Albania, Argentina, Brazil, Bulgaria, Chile, Cyprus, Greece, Israel, Malta, Portugal, Spain, Turkey, United Kingdom) with the aim of assessing and determining the risk and protective factors for perinatal mental health problems during the COVID-19 pandemic. A detailed description of the methods is available in Motrico et al. (Reference Motrico, Bina, Dominguez-Salas, Mateus, Contreras-Garcia, Carrasco-Portino, Ajaz, Apter, Christoforou, Dikmen-Yildiz, Felice, Hancheva, Vousoura, Wilson, Buhagiar, Cadarso-Suárez, Costa, Devouche, Ganho-Ávila, Gómez-Baya and Gude2021). The sample consists of adult pregnant and postpartum women (with biological children up to 6 months of age). From these, participants who were pregnant and completed the Edinburgh Postnatal Depression Scale (EPDS, Cox, Holden, & Sagovsky, Reference Cox, Holden and Sagovsky1987), as well as data on the predictors of interest described later in the article, were included for the illustrative example (n = 5,372).

Variables

Symptoms of depression were assessed using the EPDS, a 10-item self-report scale. The total scores is calculated through the sum of the scores obtained in each item, with total scores ranging from 0 to 30. Higher scores indicate a higher severity of symptoms. Potential predictors of depression symptoms were chosen based on a literature review, and the probability that the data can be available in some healthcare facilities or easily obtained already during pregnancy (see Section “Variable selection”). Some of the variables in the database were not included in the model due to chronological constraints (see Section “Prognostic vs. diagnostic models”), since they were exclusive for women in the postpartum period (n = 16; e.g., low birth weight at birth) or because, according to the available literature, they were not considered relevant for the prediction of depression (n = 52; e.g., owning or renting the residence). The variables included sociodemographic information, including age, country of residence, country of birth (country born vs. foreign born), educational level (secondary education or lower vs. higher educational level), unemployment (yes vs. no); obstetric-related data, including gravidity (primipara vs. multipara), type of pregnancy (single vs. multiple), obstetric complications (yes vs. no); mental health related data, including history of mental health problems (yes vs. no), current treatment for mental health problems (yes vs. no); medical conditions of close relatives, including history of medical conditions of partner or child (no vs. yes, if any of the following – respiratory disease, diabetes, heart disease, lung disease, liver disease, cancer, disease compromising the immune system, mood and/or anxiety disorder), history of cancer of close relatives (i.e., partner, child, other relative) living in the same house hold (no vs. yes), history of mental disorders of close relatives (i.e., partner, child, other relative) living in the same household (no vs. yes); and healthcare and social support information, including perception of the quality of support by the primary maternity care provider (scale from 1 – not very well supported – to 3 – very well supported), and the perception of support by the social network (scale from 1 – not supported – to 7 – highly supported).

Aim of the prognostic model

The underlying objective of the prognostic model is to assess the individual liability of a woman at the beginning of pregnancy to develop symptoms of prenatal depression (PND). The outcome will be the EPDS score. The model should be constructed in a way that healthcare providers can use it to screen the general population of pregnant women, relying ideally on information that could be available in clinical records (e.g., without the need for additional contacts to the individual) or easily obtained already during pregnancy. The focus on accessible clinical data sources and valid measures at the individual level is recommended for improving prediction models (Kraus, Sampathgiri, & Mittal, Reference Kraus, Sampathgiri and Mittal2024).

Software and Data

All data cleaning, transformations and statistical modeling for this study were performed using the statistical computing environment R (R Core Team, 2024) and Python (Python Software Foundation, 2025). For modeling approaches, different add-on packages like glmnet, mboost, xgboost and InterpretML were used. The code to reproduce our analyses is available at PrognosticModel.git. Due to considerations of data privacy and confidentiality, the original data cannot be made publicly available. However, we provide an artificial dataset with similar characteristics, which can be used for reproducibility.

Statistical prediction models

When developing a clinical prediction model, the decision for a specific model type depends on, inter alia, (i) the type of the outcome variable that shall be predicted (e.g., continuous vs. binary), (ii) the available explanatory information (e.g., number of covariates; types of covariates), (iii) specifics of the data collection process (grouping arising, e.g., from multicenter studies; designed experiment vs. observational data). In any case, regression models form the most prominent class of statistical models usually employed to develop prediction models due to their interpretability and the broad software availability to estimate them. In the following, we discuss some important decisions for specific types of regression models.

Concerning the type of outcome variable, linear models for continuous, approximately Gaussian responses, logistic regression for binary outcomes and Poisson regression for count responses are more often used despite a plethora of other types of models that have been developed in statistics. Especially for continuous responses, it is quite common to apply transformations to the responses to take restrictions on the domain of the response into account (e.g., a logarithmic transformation for non-negative responses) or to make the distribution of the responses more symmetric (Fahrmeir, Kneib, Lang, & Marx, Reference Fahrmeir, Kneib, Lang and Marx2022, Ch. 3). Although an exact match with the normal distribution is not as central to a prediction model as it is to a model from which statistical conclusions on model parameters should be performed (confirmatory analyses), transformations that turn the distribution of the response into something close to a normal are often also beneficial for prediction quality. In practice, the normal distribution often serves as a useful starting point for analyzing quasi-continuous scores. In our analysis, we hence consider the EPDS score as the primary response variable and treat it as a quasi-continuous outcome.

For covariates, the classical challenge is to determine/select which of them are relevant for the prediction problem at hand (see Section “Variable selection”). Expert knowledge, for example, derived from a literature review or clinical experience, can often guide the decision of a suitable set of potential covariates, but data-driven variable selection is usually applied in an additional step. The most typical cases are stepwise selection approaches based on some model performance criterion such as Akaike’s information criterion (AIC) (Cavanaugh & Neath, Reference Cavanaugh and Neath2019), but alternatives such as LASSO regression (Zhang, Wei, Lu, & Pan, Reference Zhang, Wei, Lu and Pan2020) or boosting methods (Mayr, Binder, Gefeller, & Schmid, Reference Mayr, Binder, Gefeller and Schmid2014) have attracted increasing interest. In addition to the ideal subset of covariates, transformations of covariates may be considered in the variable selection step. The general aim is to determine a model that achieves good predictive abilities (i.e., it should include all relevant covariates for the prediction task) but is also easily applied in practice (i.e., it should contain a small set of variables that are directly available).

In our illustrative analysis, we rely on 15 candidate variables that could be available from clinical records or easily obtained already during pregnancy, mostly reflecting socioeconomic and clinical characteristics. Afterward, we will then perform data-driven variable selection via penalized regression with the LASSO. Table 2 reports the resulting coefficients for our example on PND, while Fig. 2 displays the corresponding coefficient paths and the tuning of λ. The LASSO method was able to perform an automated variable selection. The effect estimates of variables with minor importance are shrunken toward 0, which means that they are effectively excluded from the model. The tuning parameter λ controls the selection of variables and the shrinkage of effect estimates; larger values of λ lead to sparse models (strong regularization), and small values of λ lead to larger models close to a classical regression model without regularization. In our case, λ was optimized via a 10-fold cross-validation (re-fitting the model and checking for prediction performance). Of the 15 candidate variables, 9 are selected to be part of the prediction model. The direction of the effects is as expected, for example, a history of mental problems has a strong positive effect on the predicted EPDS.

Table 2.

Resulting coefficients of a penalized regression model for EPDS, fitted via the LASSO

Table 2. Long description

Starting from the top row, the left column lists variables and the right column shows their coefficients. The intercept is 13.53. Age mother has minus 0.03. Born in foreign country and Living with partner are excluded, with no coefficient shown. Low education is 0.31. Unemployment is excluded. History of mental health problems is 2.89. Primiparity is minus 0.09. Obstetric complications is 0.70. Singleton pregnancy is excluded. High support by maternity care provider is minus 1.01. High social support is minus 0.57. Health problems child is 0.84. Health problems partner and Cancer case in household are excluded. Mental disorder in household is 0.34. Variables without coefficients are shrunk toward zero and excluded from the model.

Note: Variables without entry have been shrunk toward 0, and are effectively excluded from the model (see also Fig. 2 for the corresponding coefficient paths). For a comparison to the variable importance from a ML approach, see Fig. 3 Note that country was not considered as a variable here, in order to ensure also predictions to regions not included in our data are possible.

Especially when combining data from multiple trials or multiple centers that collected the data, it can be important to control for unobserved heterogeneity in the trials/centers. As a consequence, random effects models for the different centers are typically employed in such contexts where the random effects are representatives for the unobserved heterogeneity (Tutz, Reference Tutz2026). In transnational studies, random effects can also be used to represent country-specific heterogeneity. Similarly to the distributional assumption for the response discussed earlier, the inclusion of random effects is less essential for a prediction model than for a confirmatory analysis of a clinical trial, where they are often considered a means of adequately reflecting uncertainty. However, considering unobserved heterogeneity and confounding effects can be relevant to ensure that the prediction model can be transferred across different contexts. For a more detailed look at the challenge of generalizability of prediction models, see Section “Generalizability.”

In any kind of regression model, the regression effects provide interpretable information on the contribution of the different covariates toward the prediction, which is useful both for clinicians themselves and for the communication with patients (for a recent discussion of different model types, see Heller, Reference Heller2024). In our EPDS example, one can observe in Table 2 for example a small negative effect of the age of the mother, meaning that we predict a lower EPDS score (less depressive symptoms) for older women. On the other hand, a history of mental problems is estimated to have a strong positive effect: considering everything else equal, a woman who had suffered from mental problems in the past is predicted to experience a 2.89 points higher EPDS score. For comparison, 10 years of age only reduce the predicted EPDS by 0.3. Note that the LASSO estimated model incorporates variable selection (some coefficients are set to 0) and does not directly provide confidence intervals for coefficients. Nevertheless, prediction models are usually built for prediction purposes such that quantification of uncertainty for the estimated coefficients is of lesser relevance than in other contexts.

Machine learning and algorithmic predictions

ML has emerged as a modern way of constructing prediction models that often achieve better predictive performance compared to classical statistical regression models (as discussed in the previous section) at the price of lacking interpretability (Lucasius et al., Reference Lucasius, Ali, Patel, Kundur, Szatmari, Strauss and Battaglia2025). More precisely, many ML approaches do not make strong structural assumptions about the impact of covariates on the outcome of interest, but rather “let the data speak” to determine the most suitable model specification. This includes the automatic consideration of interactions between covariates that are usually restricted in statistical models (or at least have to be included manually). Despite these different goals of statistical models and ML, there is also considerable overlap in terms of approaches that are – depending on the community – considered to belong to the one or the other field. For example, LASSO regularization is sometimes claimed to be part of ML but originated in the field of statistics (Caner, Reference Caner2021).

One particularly prominent example of ML prediction models is random forests relying on an ensemble of trees (classification or regression trees, depending on the type of the response variable, Breiman, Reference Breiman2001; Hastie, Tibshirani, & Friedman, Reference Hastie, Tibshirani and Friedman2009). Trees are based on iteratively splitting the data set based on the covariates available such that the resulting partitioning makes partitions homogeneous with respect to the response of interest while maximizing between-partition heterogeneity. Each partition is then assigned a predicted outcome. Due to the iterative splitting approach, trees are very flexible and also include interactions between covariates, but are typically quite unstable, that is, the tree might change considerably in response to even quite small changes in the data. Random forests compensate for this by considering an ensemble of trees built on random subsets of the data, which increases the stability of the approach. The individual covariate effects are then of complex form and only hardly accessible, but covariates can still be assigned a variable importance to facilitate a certain level of interpretability.

Another typically tree-based approach is gradient boosting. In contrast to random forest that grow trees independently on bootstrap samples, gradient boosting grows them sequentially on re-weighted observations. Observations that have been misclassified before receive higher weights (Freund & Schapire, Reference Freund and Schapire1996) A very powerful and fast implementation is XGBoost. Note that there are also statistical boosting algorithms, which are based on a similar principles but yield statistical models (Bühlmann & Hothorn, Reference Bühlmann and Hothorn2007; Friedman, Hastie, & Tibshirani, Reference Friedman, Hastie and Tibshirani2000).

In many applications, ML approaches achieve very good prediction accuracy and can outperform classical statistical models in this respect. However, in addition to the more challenging interpretation, they are also more difficult to turn into clinical practice since they cannot be easily represented in a formula/equation that can be evaluated in day-to-day practice. In addition to random forests, other forms of ML could also be used to construct prediction models. This includes approaches such as support vector machines (Song and Diederich Reference Song and Diederich2013) but also neural network approaches, establishing a connection to the area of AI (see Section “Challenges and opportunities in AI-driven decision support systems”).

In Fig. 3, we display the variable importance estimates from a tree-based approach – extreme gradient boosting (XGBoost – (Chen & Guestrin, Reference Chen and Guestrin2016) to estimate an algorithmic solution. In contrast to the LASSO solution in Table 2, we cannot directly interpret the model itself, in the sense of interpreting the direction and size of effects via coefficients, but we need to assess the relative importance of the variables for the prediction performance to gain some insights (feature importance). The ML model was run on the same set of candidate variables. Also here we can see a strong importance of the history of mental problems variable, followed by the age of the mother. Most of the variables that had not been selected by the LASSO are also showing a very low importance.

Challenges and opportunities in AI-driven decision support systems

AI has revolutionized various sectors, and mental healthcare is no exception. Decision Support Systems (DSS) powered by AI could, in the future, provide better prognostic and diagnostic accuracy, personalized treatment plans and improved patient outcomes. These systems are designed to help clinicians make informed decisions, thereby optimizing mental healthcare delivery.

Traditional diagnostic methods are partially based on subjective assessments, which can lead to misdiagnosis or overlooked conditions. Recent work of Bradford et al. (Reference Bradford, Meyer, Khan, Giardina and Singh2024) clearly calls for more research to identify missed opportunities in the diagnosis of mental health problems. AI-driven DSS analyzes large amounts of data from various sources, including clinical records, patient interviews and even social media activity, to identify indicative patterns of specific mental health disorders. As mentioned in Section “Machine learning and algorithmic predictions,” ML algorithms as well as big data analytics can be used to process this volume of data and detect subtle signs and symptoms that might be missed by human clinicians. For example, Natural Language Processing (NLP) algorithms can use patient speech or writing to analyze signs of depression, anxiety or other mental health conditions. Although these data-driven approaches tend to produce more accurate and timely diagnoses, ultimately leading to better patient outcomes, they lack interpretability, transparency and robustness. These are some of the challenges identified in recent studies by Higgins, Short, Chalup, and Wilson (Reference Higgins, Short, Chalup and Wilson2023) and Andrew, Rudra, Eunice, and Belfin (Reference Andrew, Rudra, Eunice and Belfin2023) that can justify the small number of studies considered in the integrated review by Higgins, Short, Chalup, and Wilson (Reference Higgins, Short, Chalup and Wilson2023).

To address these challenges, Nori, Jenkins, Koch, and Caruana (Reference Nori, Jenkins, Koch and Caruana2019) proposed InterpretML, an open source Python package designed as a unified framework for ML interpretability. It offers two approaches to interpretability: Glassbox and Blackbox forms. The Blackbox model provides explanations for any ML model, regardless of its complexity, using interpretation techniques such as LIME, SHAP and Partial Dependence Plots. The Glassbox models are ML algorithms designed to give clear interpretations of the decisions provided for models such as linear models, decision trees and generalized additive models (GAM). It uses an Explainable Boosting Machine (EBM) algorithm, an extension of the GAM that allows the automatically inclusion of pairwise interactions, providing two levels of explanations: global and local effects. Thus, InterpretML extends linear models such as LASSO regression approaches beyond linearity, opening the black-box models such as ML approaches to inspection and interpretability, where visual plots are provided to show how the different covariates affect predictions, allowing intuitive interpretation to non-technical decision-makers.

Consider our EPDS data set with the 15 variables previously presented as predictors and a dependent variable defined as 1 (depression) if a woman’s EPDS score was 13 or higher, and zero (no depression) otherwise.

Local explanations are instance-specific and provide insight into why a particular prediction was made for a specific input. In Fig. 4, we clearly see that the main factors that contribute to a no-depression classification (top) are high social support, no history of mental problems, strong support from the maternity health provider and no obstetrics complications. For a diagnosis of depression (EPDS ≥13), it is worth noting the inclusion of living with a partner and low education at the top of the list (middle). Finally, for a misdiagnosis of no depression (bottom), the most contributing factor is the lack of a history of mental problems and being older (41 years). Note also that the estimated probability of depression in this case is 0.227, high enough for the healthcare professional to consider a potential case of developing depression.

Although the potential promise of AI-driven DSS in reducing the burden and burnout of medical professionals is immense, giving time back to health professionals, Bossewitch et al., Reference Bossewitch, Brown, Gooding, Harris, Horton, Katterl, Myrick, Ubozoh and Encalada2022 alert to the possible harm of the use of algorithmic and data-driven technologies in the mental health context, which could lead to further stigmatization of patients, overdiagnosis and unnecessary medication (Thakkar, Gupta, & De Sousa, Reference Thakkar, Gupta and De Sousa2024).

Preprocessing data

A well-known saying in the community with respect to data is: “garbage in, garbage out…”. If the dataset is of low scientific quality, that is, non-representative, ambiguous, noisy, imbalanced or very sparse, this might likely translate into bad or inaccurate findings, no matter the quality or the sophistication of the model. Therefore, effective data preprocessing is important for any data analysis. The goals and characteristics of the research question and the requirements of the models will drive the preprocessing procedures. These procedures are also sometimes summarized under the term initial data analysis (Baillie, Le Cessie, Schmidt, & Lusa, Reference Baillie, Le Cessie, Schmidt and Lusa2022; Heinze et al., Reference Heinze, Baillie, Lusa, Sauerbrei, Schmidt, Harrell and Huebner2024). This might include verifying the distribution of the variables, checking for inconsistencies or suspicious patterns in responses to identify or filter out unreliable data (e.g., checking for outliers or missing responses). In addition, one of the most common preprocessing techniques involves the transformation of variables. Transformations are used for various reasons, primarily to correct skewness or to standardize the data by centering and scaling. Many predictive models perform better when the predictors are standardized on a common scale. However, a significant drawback is that the interpretability of the estimated effects can be lost because the data are no longer in their original units. For instance, predictive models based on decision trees (Section “Statistical prediction models”) are less sensitive to the characteristics of predictors. In contrast, classical linear regression (Section “Machine learning and algorithmic predictions”) heavily depends on these characteristics.

Some of the data collected in mental health surveys are on an ordinal scale, for example, perception of the quality of support by the primary maternity care provider (1 – not very well supported; 2 – well supported; 3 – very well supported), which means that the response points are ordered, but the distance between the values is without meaning. Unlike nominal data, these variables/features have a distinct rank or order, adding complexity to the preprocessing steps. Often, they need to be encoded to facilitate their inclusion in predictive models by assigning numerical values to categories based on their order. It is crucial to ensure that the assigned numerical values respect the inherent order of the categories, which means that higher values must correspond to higher ranks or levels on the ordinal scale. For example, perception of the quality of support by the primary maternity care provider might be coded as 0 – not very well supported; 1 – well supported; 2 – very well supported. Alternatively, an ordinal (monotonic) encoding can be applied, where the ordered categories are transformed into integers that reflect their ranking without assuming equal distances between them; this approach allows models that explicitly handle ordinal predictors (e.g., ordinal logistic regression or tree-based methods with monotonic constraints) to incorporate the ordering information directly. A dummy variable encoding can also be used when order is not relevant, but each category needs to be represented distinctly, by assigning a binary indicator (1 or 0) to each category.

Data eligibility is another challenge. In our PND example, data were collected online (for more information on methods, see Motrico et al., Reference Motrico, Bina, Dominguez-Salas, Mateus, Contreras-Garcia, Carrasco-Portino, Ajaz, Apter, Christoforou, Dikmen-Yildiz, Felice, Hancheva, Vousoura, Wilson, Buhagiar, Cadarso-Suárez, Costa, Devouche, Ganho-Ávila, Gómez-Baya and Gude2021); therefore, special attention was paid to making sure that all participants were, in fact, complying with the eligibility criteria and that there were no duplicates in the database. Since in online platforms there is no guidance from a researcher, it is quite often that participants misunderstand the information on eligibility criteria or answer more than once, especially when the dissemination of the study is ongoing for several months. To manage these challenges, several approaches can be implemented. For instance, in our illustrative example, we checked the questions regarding the eligibility criteria and excluded participants who did not meet the criteria. The questionnaire was anonymous, so to check for potential duplicates, we looked at the information on the date of birth of the participants and the child. In case there was a match of dates, we used other sociodemographic variables to ascertain the likelihood of being the same person. In cases of high matches, we disregarded the second entry in the questionnaire and considered only the first (Mesquita et al., Reference Mesquita, Costa, Bina, Cadarso-Suárez, Gude, Díaz-Louzao, Dikmen-Yildiz, Osorio, Mateus, Domínguez-Salas, Vousoura, Levy, Alfayumi-Zeadna, Wilson, Contreras-García, Carrasco-Portiño, Saldivia, Christoforou, Hadjigeorgiou and Motrico2023).

Missing data

Among all the obstacles to start the construction of a predictive model, missing data is likely one of the most challenging. It is of utmost importance to understand why the values are missing, because this will drive the researcher’s decision about what to do with the predictors or with the respective participants for whom some values are missing. After flagging the missing values, one is typically faced with two options. Simply remove individuals or variables with missing values or try to fill in the gaps in the data set (imputation). If the pattern/characteristics of the missing values somehow allow us to anticipate the outcome of the response, then we will be in the presence of what is commonly called an informative omission (Little & Rubin, Reference Little and Rubin2019, Ch. 13). A typical example of this situation is when, during a follow-up, some patients stop attending depression therapy sessions, thus stop reporting their depression levels. These dropouts may be due to significant improvement of symptoms, making them feel that reporting is unnecessary. This is noted in many research studies on attrition in mental health (Cuijpers et al., Reference Cuijpers, Karyotaki, Weitz, Andersson, Hollon and van Straten2014; Warden et al., Reference Warden, Trivedi, Wisniewski, Davis, Nierenberg, Gaynes, Zisook, Hollon, Balasubramani, Howland, Fava, Stewart and Rush2007). Statistically, this situation is especially challenging because researchers cannot simply exclude individuals with missing data; the absence of a measurement at a specific time point can, in itself, reveal insights about the response. To address this, researchers could either develop an imputation model to predict the missing values or use the missingness as a separate category (see, e.g., Figure 4). We refrain from going further into this subject, as this is beyond the scope of our work. For a comprehensive review and an evaluation of the methods dealing with missing data can be found elsewhere (Carpenter et al., Reference Carpenter, Bartlett, Morris, Wood, Quartagno and Kenward2023; Graham, Reference Graham2009; Van Buuren, Reference Van Buuren2018; Wang & Ware, Reference Wang and Ware2013).

Prognostic vs. diagnostic models

In both statistical and ML literature, a model that aims to predict an unknown or unobserved outcome y based on variables or features x is typically called predictive. This prediction of the outcome does not necessarily relate to a future observation, but can also be just an unknown characteristic. However, in the medical literature, the process of assessing a current unknown status of a patient is typically called diagnostic (Van Smeden et al., Reference Van Smeden, Reitsma, Riley, Collins and Moons2021). Many current applications of deep learning or AI techniques in medical imaging, such as radiology, are based on cross-sectional data on the association between predictor variables and outcome (Hosny et al., Reference Hosny, Parmar, Quackenbush, Schwartz and Aerts2018; Secinaro et al., Reference Secinaro, Calandra, Secinaro, Muthurangu and Biancone2021). These are, therefore, by their nature, diagnostic, as the disease is already present (and can be detected in the corresponding images). In the case of prognostic models for future outcomes, the association between predictor variables and the outcome is longitudinal, and the objective is to assess the individual risk of a patient for a future occurrence of a disease (Collins et al., Reference Collins, Reitsma, Altman and Moons2015).

In the case of our example on PND, this distinction can be easily illustrated by the underlying aim of the prediction model. If the goal is prognostic, the prediction model should identify women at high risk of developing depression in the future, allowing preventive measures to be initiated. From a preventive point of view, the crucial time point for applying the prognostic model could be the beginning of pregnancy, to allow early identification and intervention (Curry et al., Reference Curry, Krist, Owens, Barry, Caughey, Davidson, Doubeni, Epling, Grossman, Kemper, Kubik, Landefeld, Mangione, Silverstein, Simon, Tseng, C and Wong2019; RISEUP-PPD Guidelines Development Group, 2022). If the aim was diagnostic, the model should identify pregnant women who are currently suffering from depression, allowing timely clinical diagnosis and treatment.

This distinction significantly influences the selection of potential predictor variables. In cases involving longitudinal associations, such as in prognostic modeling, it is essential to respect the chronological order of events. For example, in our model of PND, only variables available in clinical records or easily obtained at the beginning of pregnancy should be considered, as that is the intended time for prognosis. In our example, derived from a large international observational study with extensive data collected throughout pregnancy, adhering to this chronological constraint drastically reduces the pool of potential predictor variables for a prognostic model (see Section “Variables”).

In contrast, a diagnostic model for depression can incorporate any available variable to determine whether a woman is currently experiencing depression. For instance, in our dataset, the most informative covariate for EPDS is Generalized Anxiety Disorder-7 (GAD-7), a widely used scale to assess anxiety symptoms that is strongly correlated with EPDS. However, since GAD-7 is measured at the same time as EPDS, it is suitable for a diagnostic model but not for a prognostic model.

It is evident that prognostic modeling presents a more challenging task in practice, as the longitudinal nature of associations may be subject to change over time (Cook, Reference Cook2008).

Variable selection

One of the most critical decisions in the development of a clinical prediction model is the selection of the predictor variables that will form the model (Collins et al., Reference Collins, Reitsma, Altman and Moons2015). This decision has an important impact on the accuracy achievable by the final model, as only a combination of highly informative variables can build a valid prediction model. However, the choice of the variable set also has very relevant consequences for the applicability of the model and its translation into clinical practice.

Clinical prediction models will only be implemented in real-world settings if the predictor variables underlying them are readily accessible to clinical decision-makers or can be easily and quickly assessed. To develop a prediction model for eventual use in clinical practice, after external validation, it is therefore essential not only to focus on selecting the set of predictors that yield the highest possible performance but also to bear in mind that healthcare professionals who will later employ this model will need access to all these data.

In our example on PND, this means that if the objective is to build a model for internal screening by healthcare providers to identify women at risk, the subset of predictor variables that could form the model should consist of information easily available to healthcare providers. For example, variables in questionnaires such as anxiety symptoms, while highly informative about depression, should not be included in the final model. By considering the chronology of events (as discussed in Section “Prognostic vs. diagnostic models”) and selecting only variables that could be available or easily obtained at the beginning of pregnancy, the number of potential predictors is reduced to only 15 of the more than 83 variables available in the data set.

In practice, it is sensible to pre-filter existing variables regarding their availability. This can be viewed as a subject-matter-driven variable selection (Heinze, Wallisch, & Dunkler, Reference Heinze, Wallisch and Dunkler2018; Staerk, Byrd, & Mayr, Reference Staerk, Klinkhammer, Wistuba, Maj and Mayr2024). Such filtering or also the complete variable selection may also rely on prior subject matter knowledge, such as known risk factors or findings from the literature (see Table 1). In contrast, many methods to estimate prediction models (see Section “Methods to develop prediction models”), incorporate some kind of data-driven variable selection. Optimizing the corresponding tuning parameter with respect to prediction accuracy directly affects the model complexity (either via a shrinkage parameter, the number of iterations of an algorithm, or an information criteria).

It is important to note that the selection of the most suitable set of variables is not automatically the same as identifying the most important risk factors for the event or outcome of interest (Shmueli, Reference Shmueli2010). The aim of a prediction model is to find the combination of factors leading to the highest prediction accuracy, taking into account their interdependence. In our example on PND, we therefore used a ML or statistical approach to select and estimate the best combination of the 15 potential predictor variables (see also “Section “Variables””).

Metrics and measures to evaluate prediction models

An essential step after developing a prediction model is to assess its accuracy (Steyerberg, Reference Steyerberg2019). This assessment should be performed using a separate validation cohort that was not used during model development or parameter estimation (see Section “Development and validation cohort”). But how can one determine whether a model is accurate? The choice of evaluation metric depends on the specific prediction target and the clinical question being addressed. For example, in the case of continuous outcomes, such as the EPDS in our illustrative example, the focus might be on point predictions (e.g., the severity of a disease as measured by the EPDS or the value of a specific medical marker) or on predicting reference ranges for specific populations (e.g., typical biomarker values for a given age group).

For binary outcomes or classification problems (e.g., EPDS ≥13 vs EPDS <13), the primary interest is often in predicting individual probabilities (e.g., the risk of developing clinically significant depressive symptoms). These probabilities can subsequently be categorized into discrete labels, such as high-risk or low-risk individuals, for practical application.

For both types of responses – binary and continuous – different evaluation metrics are commonly used. These metrics can be broadly categorized into those assessing a model’s calibration and those evaluating its discrimination (Cook, Reference Cook2008). Calibration refers to how well the predicted values align with the observed outcomes (Van Calster et al., Reference Van Calster, McLernon, van Smeden, Wynants and Steyerberg2019). This is often assessed graphically using calibration plots, which compare observed values with predicted values or probabilities (e.g., observed vs. predicted probabilities, as described in Collins et al., Reference Collins, Reitsma, Altman and Moons2015). In contrast, discrimination reflects a model’s ability to distinguish between individuals with different outcomes, such as correctly identifying those at higher versus lower risk (Pencina, D’Agostino, & Vasan, Reference Pencina, D’Agostino and Vasan2008).

For binary outcomes, the most commonly used discriminatory measures are the AUC, as previously mentioned, and the more general C-statistic, both of which range from 0.5 (no discrimination) to 1 (perfect discrimination). For continuous outcomes, a commonly used discriminatory measure is R ² for test data, which represents the proportion of variance in the outcome explained by the model (Staerk, Klinkhammer et al. Reference Staerk, Byrd and Mayr2024). Some metrics evaluate both calibration and discrimination (Steyerberg, Reference Steyerberg2019). For example, the Brier score for binary outcome measures the mean squared difference between observed outcomes (0 or 1) and predicted probabilities (Brier, Reference Brier1950). For continuous outcomes, the analogous measures are the Mean Squared Error of Prediction (MSEP) and its square root, the Root Mean Squared Error of Prediction (RMSEP).

Development and validation cohort

To assess the accuracy of prediction models, irrespective of whether they are prognostic or diagnostic, it is extremely important to validate the performance on unseen observations that have not been included in the model development, e.g., by utilizing an external validation cohort (Collins et al., Reference Collins, Dhiman, Ma, Schlussel, Archer, Van Calster, Harrell, Martin, Moons, Van Smeden, Sperrin, Bullock and Riley2024). Evaluating the model on already seen observations would most likely lead to drastically overoptimistic results and heavily biased conclusions about the abilities of the model.

Unfortunately, in some of the prediction models we found in our literature review, it cannot be ruled out that this issue is present (see Table 1), as no separate validation set was described. In this context, one could assume that models are trained toward overfitting, a problem often encountered in prediction models (Hastie, Tibshirani, & Friedman, Reference Hastie, Tibshirani and Friedman2009; James, Witten, Hastie, & Tibshirani, Reference James, Witten, Hastie and Tibshirani2013). In order to minimize the discrepancy between model and observations, classical fitting approaches (e.g., maximum likelihood procedures) for statistical models will estimate the best model for the development cohort. As a result, the model focuses too much on peculiarities in this underlying sample and performs poorly on observations outside this sample. Ignoring the distinction between in-sample and out-of-sample performance by validating a model on already seen observations from the development cohort makes the identification of overfitting behavior impossible.

To overcome potential biases and ensure a fair assessment of a prediction model, external validation is considered the gold standard. Ideally, the model should be tested on entirely separate data. This requires validation carried out by different researchers, in different regions or hospitals, to ensure robustness in various settings.

A second-best option is so-called temporal validation, where patients recruited until a specific time point constitute the development cohort, while those recruited afterward form the validation cohort. This procedure attempts to mimic external validation and typically leads to more realistic results than just randomly splitting the data (Collins et al., Reference Collins, Reitsma, Altman and Moons2015).

Randomly splitting the data once or multiple times, through techniques such as bootstrapping, cross-validation or subsampling, constitutes internal validation. These methods are widely used in practice, as confirmed by our literature search. Internal validation (e.g., via bootstrapping) is designed to correct for optimism in apparent model performance and can provide approximately unbiased estimates of predictive performance for future individuals drawn from the same underlying population as the development cohort. However, when model performance is intended to generalize to different settings (e.g., routine clinical practice and not a clinical trial), populations (other regions, countries) or time periods (prognostic models), performance estimates obtained through internal validation may still be optimistic. This reflects the fact that internal validation does not account for differences in case mix, clinical practice or other forms of unmeasured heterogeneity between development cohort and target populations.

In clinical practice, dependencies and associations may often differ slightly from those observed in the trial used to develop the model. In order to report performance measures of the new model that are closer to clinical reality, it may be favorable to try to obtain truly external observations (or at least temporally splitted data) even if this leads to poorer estimated performances.

In our example on PND this can be easily illustrated due to multinational structure of the survey. We fitted linear prediction models for the EPDS via different estimation approaches (Table 3), including a step-wise variable selection via the AIC (Heinze, Wallisch, & Dunkler, Reference Heinze, Wallisch and Dunkler2018), a component-wise gradient boosting approach with linear base-learners (Bühlmann & Hothorn, Reference Bühlmann and Hothorn2007) and also a LASSO L ₁-regularized model (Tibshirani, Reference Tibshirani1996). We performed different splits of the data into development and validation cohorts. First, we simply randomly split the data (70/30 random split) before we performed a 10-fold cross-validation. Additionally, we performed a temporal split by including all cases before November 2020 in the development cohort and all after this time point in the validation cohort. Afterward, we also used a regional split by including all observations from Western + Central European countries in the development cohort and all others (South America, Israel, Turkey) in the validation cohort.

Table 3.

Root mean squared error of prediction (RMSEP) evaluated on different test cohorts performing random splitting, 10-fold cross-validation, temporal split (before and after November 2020), and regional split: Western + Central European countries form the development cohort, all others (South America, Israel, Turkey) form the validation cohort

Table 3. Long description

From the top row downward, the table lists test cohorts: Random split, 10-fold C V, Temporal Split, and Regional split. For each cohort, root mean squared error of prediction values are given for Step-wise, Boosting, and LASSO methods. Random split: Step-wise 5.31, Boosting 5.31, LASSO 5.40. 10-fold C V: Step-wise 5.37, Boosting 5.37, LASSO 5.42. Temporal Split: Step-wise 5.44, Boosting 5.42, LASSO 5.45. Regional split: Step-wise 5.49, Boosting 5.49, LASSO 5.60. Regional split defines Western plus Central European countries as the development cohort, and South America, Israel, Turkey as the validation cohort.

Table 4.

Population characteristics and endpoints of the studies

Table 4. Long description

Beginning at the top row, the table presents studies in the leftmost column: Adler, Wang, Mohr, and Choudhury (2022) with 61 patients in treatment and 48 university students, mean patient age 37 years, region USA, endpoint EMA continuous. Next, Albagmi et al. (2022) with n equals 3,017, over half aged 20 to 39 years, region Saudi Arabia, endpoint G A D dash 7 binary and categorical. Bjertrup, Vae ver, and Miskowiak (2023) with n equals 87 pregnant women, mean age 33 years, region Denmark, endpoints MINI binary and E P D S continuous. Caye et al. (2022) with n equals 1,072, age 19 years, region USA, endpoint Y A P A binary. Chavanne et al. (2023) with n equals 156 participants with anxiety and 424 controls, ages 18 to 23 and 14 years, region not reported, endpoint G A D dash 7 binary. Chung and Teo (2023) with n equals 955, age and region not reported, endpoint mental health problems binary. Cunha et al. (2023) with n equals 1,442, mean age 18 years, region Brazil, endpoint D A W B A binary. Geffre, Carr, Meynadasy, and Sachs Ericsson (2023) with n equals 571, age over 60 years, region USA, endpoint COVID dash 19 related distress continuous. Kim, Park, and Kang (2022) with n equals 116 potential depression participants and 80 controls, mean ages 59 and 49 years, region Korea, endpoint B D I binary. Liu et al. (2022) with n equals 16,384, age over 18 years, region UK, endpoint P H Q dash 9 continuous. Matsuo et al. (2022) with n equals 10,013, mean age 31 years, region Japan, endpoint E P D S binary. Murri et al. (2022) with n equals 39,439, age over 55 years, region Europe, endpoint E U R O dash D binary. Noro n a dash Zhou et al. (2023) with n equals 1,389 mother child dyads, mean child age 9 years, region USA, endpoints S C A R E D continuous and binary, C D I dash 2 continuous and binary. Ostberg and Nordin (2022) with population based data, ages 18 to 79 years, region Sweden, endpoints H A D S continuous and binary. Qasrawi et al. (2022) with n equals 3,984, ages 10 to 15 years, region Israel, endpoints G A D dash 7 binary, D S R S binary. Roseberry et al. (2023) with n equals 721, age and region not reported, endpoints S A S dash 4 binary, S T A I binary. Sajjadian et al. (2023) with n equals 192, mean age 35.4 years, region Canada, endpoint M A D R S continuous and binary. Wang et al. (2023) with n equals 13,293, mean age 54 years, region USA, endpoint P H Q dash 9 binary. Wy et al. (2022) with n equals 129, age not reported, region Korea, endpoints P H Q dash 9 binary, G A D dash 7 binary. Yang et al. (2022) with n equals 981, mean age 31 years, region China, endpoint E P D S binary. Zhou and Reiter (2010) with n equals 2,574, age over 18 years, region China, endpoint P H Q dash 9 binary. Zhu (2022) with college students, age and region not reported, endpoint depression binary. Abbreviations in the table footnote include U S A for United States of America, E M A for Ecological Momentary Assessments, G A D dash 7 for General Anxiety Disorders, MINI for Mini International Neuropsychiatric Interview, E P D S for Edinburgh Postnatal Depression Scale, Y A P A for Young Adult Psychiatric Assessment, D A W B A for Development and Well Being Behavior Assessment, COVID dash 19 for coronavirus disease 2019, P H Q dash 9 for Patient Health Questionnaire dash 9, E U R O dash D for European Union initiative, S C A R E D for Screen for Child Anxiety Related Emotional Disorders, C D I dash 2 for Children’s Depression Inventory Second Edition, D S R S for Depression Self Reported Scale, S A S dash 4 for Simplified Anxiety Scale, M A D R S for Montgomery and Asberg Depression Rating Scale. Age at endpoint assessment is marked with an asterisk.

Note: USA, United States of America; EMA, Ecological Momentary Assessments; GAD-7, General Anxiety Disorders; MINI, Mini-International Neuropsychiatric Interview; EPDS, Edinburgh Postnatal Depression Scale; YAPA, Young Adult Psychiatric Assessment; DAWBA, Development and Well-Being Behavior Assessment; COVID-19, coronavirus disease 2019; PHQ-9, Patient Health Questionnaire-9; EURO-D, European Union initiative; SCARED, Screen for Child Anxiety Related Emotional Disorders; CDI-2, Children’s Depression Inventory, Second Edition; DSRS, Depression Self-Reported Scale; SAS-4, Simplified Anxiety Scale; MADRS, Montgomery and Åsberg Depression Rating Scale; *age at endpoint assessment.

It is notable that the difference in performance between the modelling methods is smaller than the different prediction errors achieved from different validation techniques: The root RMSEP is ranging from 5.31 to 5.60 for the different validations, see Table 3. Regarding the modelling methods, for example, for the temporal split, the difference is comparably marginal (5.42 to 5.45). This might be also due to the limited number of variables that we have to predict depression (Buckman et al., Reference Buckman, Cohen, O’Driscoll, Fried, Saunders, Ambler, DeRubeis, Gilbody, Hollon, Kendrick, Watkins, Eley, Peel, Rayner, Kessler, Wiles, Lewis and Pilling2023), which limits the abilities of the more advanced methods compared to the step-wise variable selection for logistic regression. The regional split, which is closest to a real external validation, leads to the poorest results (5.49–5.60). However, these results might be the most realistic ones. The single random split of the data, representing internal validation, leads to the best (5.31–5.40) but perhaps overoptimistic results, which might be heavily influenced by high variability.

Generalizability

While Section “Development and validation cohort” focused on validation strategies to assess predictive performance, an equally important challenge is the ability of prediction models to generalize beyond the specific conditions under which they were developed. Generalizability refers to the extent to which a model maintains acceptable performance across different populations, settings and time periods. This section therefore focuses on structural and design-related factors that influence generalizability and cannot be fully addressed through validation alone.

The most fundamental issue is that the development cohort covers a broad range of different settings that are diverse and representative of various clinical or population realities to which the model will later be applied. In our example on PND, where our objective is to develop a screening tool for pregnant women, the development cohort should cover the true heterogeneity of the general population. However, many cohorts from research studies tend to be biased toward participants with higher education and lack representation from individuals with low socioeconomic backgrounds. This also holds for our development sample (Motrico et al., Reference Motrico, Bina, Dominguez-Salas, Mateus, Contreras-Garcia, Carrasco-Portino, Ajaz, Apter, Christoforou, Dikmen-Yildiz, Felice, Hancheva, Vousoura, Wilson, Buhagiar, Cadarso-Suárez, Costa, Devouche, Ganho-Ávila, Gómez-Baya and Gude2021).

Additionally, to avoid overfitting in the first place, many statistical or machine-learning techniques are designed not to achieve the best fit on the development cohort but to provide good predictions on external observations and robustness to shifts in association structures. This is achieved by using penalization techniques, early stopping or careful tuning of hyperparameters based on prediction performance (Friedrich et al., Reference Friedrich, Groll, Ickstadt, Kneib, Pauly, Rahnenführer and Friede2023).

Prominent examples of the failure of prediction models to generalize well beyond the initial populations, particularly with respect to mental disorders, are genetic prediction models. These models are usually developed based on large cohort data or genome-wide association studies and aim to report the genetic predisposition of an individual toward a phenotype or trait. While these types of prediction models have made significant progress in recent years, their generalizability remains a substantial challenge. Typically, the underlying cohorts consist mostly of individuals with European ancestry, due to the dominance of genotyping performed in North American and European countries. Current research suggests that the performance of these models on individuals from non-European ancestries, which are underrepresented in the development cohorts, is very limited. One provocative study (Curtis, Reference Curtis2018) even concluded that a polygenic prediction model for schizophrenia was more strongly associated with ancestry than with schizophrenia. The reasons for this include different linkage (associations) and the prevalence of risk alleles in ancestry-based populations. Very current research tries to overcome these issues by incorporating methods from causal inference (Rothenhäusler & Bühlmann, Reference Rothenhäusler and Bühlmann2023; Rothenhäusler, Meinshausen, Bühlmann, & Peters, Reference Rothenhäusler, Meinshausen, Bühlmann and Peters2021).

In the context of our illustrative example on PND, we benefit from a highly diverse dataset that includes observations from different countries, populations and sociodemographic backgrounds. However, there is a bias toward higher educated women with a better sociodemographic profile. This is most likely due to the self-selection process of the participants. In addition, there are significant differences between countries and regions. The EPDS threshold defining clinically significant symptoms varies across countries and languages. However, a threshold of 13 or more is often used to indicate clinically significant symptoms of depression, as recommended for a cut-off point with higher specificity in the perinatal period based on data from several countries, including seven of the countries in our analysis (Brazil, Chile, Greece, Malta, Portugal, Spain, Turkey and the UK; Levis et al., Reference Levis, Negeri, Sun, Benedetti and Thombs2020). In the remaining countries included in our analysis, no cutoff point is mentioned in Argentina, Cyprus and Israel (Alvarado, Jadresic, Guajardo, & Rojas, Reference Alvarado, Jadresic, Guajardo and Rojas2015; Kandel-Katznelson, Maisel, Zilber, & Lerner, Reference Kandel-Katznelson, Maisel, Zilber and Lerner2000; Lopez-Janer, Quiros, & Cabestrero, Reference Lopez-Janer, Quiros and Cabestrero2021). Furthermore, our analysis of different validation approaches (Table 3) highlights the limited generalizability of models developed in one set of countries when applied to others, underscoring the role of population-specific factors. A solution to this might be to develop different regional or population-specific prediction models.