Selecting observations

It is possible that not all of the observations in the dataset are suitable to be included in the analysis. Reasons for this could include outliers, repeated measures (from which only one is required) or subgroups with few observations, such as males in a predominantly female sample. Decisions regarding which observations should be included and excluded should justified (e.g. based on good statistical grounds or findings from previous studies) and well documented when the methodology is described.

Pre-selecting features & dimensionality reduction

Whilst classic feature selection makes use of the outcome and uses statistical techniques to determine which features to include (see below: Step 3. Modelling), the analyst may also have to decide which features (or groups of features) are potentially relevant to the problem and therefore should be included for data processing. Where possible, domain knowledge should be used to justify the elimination of variables that are not expected to be relevant to the problem^{(Reference Badillo, Banfai, Birzele, Davydov, Hutchinson, Kam-Thong, Siebourg-Polster, Steiert and Zhang9,Reference Nguyen and Holmes42)} . For example, when investigating cardiovascular disease risk using a large cohort, specific biochemical measures may be included, whereas others deemed insufficiently relevant can be excluded. Importantly, any feature selection steps that make use of the outcome variable must be performed within validation loops (see below: Internal validation schemes in Step 3. Modelling).

Dimensionality reduction refers to reducing the number of features irrespective of the outcome variable (i.e. unsupervised approaches), such as through identifying redundancies in the data^{(Reference Harrell43,Reference Sorzano, Vargas and Pascual-Montano44)} . Reducing the dimensionality of a dataset can be desirable by reducing model complexity, computation time, problems related to collinearity, and overfitting^{(Reference Nguyen and Holmes42)}. Low-variance features may not contain enough information to be useful to the problem and are sometimes removed or combined with other features, if appropriate^{(Reference Sarker16)}. However, this should be done with caution since loss of information is possible. For example, in microbiome studies it is common to see bacteria present in fewer than a given proportion of the sample removed (e.g. <5%). However, in some cases the presence of a given bacterium in a small group of individuals may be informative for health outcomes or the problem being investigated. Such findings may be lost due to low-variance filtering.

Techniques for identifying similarities or redundancies within the data can also be employed, such as correlation-based approaches, PCA and variable clustering^{(Reference Zebari, Abdulazeez, Zeebaree, Zebari and Saeed45,Reference Xu and Lee46)} . It is commonly believed that it is not necessary to perform such unsupervised data reduction techniques within validation steps and, instead, that they can be safely applied to the entire dataset. This belief is based on the understanding that these techniques do not make use of the outcome variable, thus minimising the risk of information leakage^{(Reference Hastie, Tibshirani and Friedman14,Reference Sarker16)} . However, recent findings suggest that unsupervised dimensionality reduction steps on the whole dataset can still introduce bias^{(Reference Moscovich and Rosset47)}. Whilst more work is needed on the topic, analysts may consider repeating analyses where dimensionality reduction is confined within training splits of validation steps to assess the sensitivity of results to the timing of these steps.

Processing missing data

Missing data is common in many datasets and comes in different forms, including missing completely at random (MCAR), missing at random (MAR) and not missing at random (NMAR)^{(Reference Bennett48)}. Each reflects different underlying mechanisms: MCAR implies that the missingness is unrelated to any data, MAR suggests that the missingness is related to observed data but not the missing data themselves, and NMAR indicates that the missingness is related to the unobserved data^{(Reference Bennett48)}. These distinctions are crucial because they influence the methods used to handle missing data and the conclusions derived from the results.

One approach to missing data is to simply restrict the analyses to complete cases. This may be justified in certain circumstances, such as when data are MCAR and the number of observations with missing data is relatively small, however, when data are NMAR, removing observations with missing data can bias the results, and verifying the type of missing data in question is not always possible^{(Reference Sterne, White, Carlin, Spratt, Royston, Kenward, Wood and Carpenter49)}.

An alternative is to impute the missing data in various ways. Single imputation approaches with the mean, mode or other summary statistics are simple to implement, although they are almost never optimal and can distort the distribution of the features of the dataset and the relationships between them^{(Reference Zhang50,Reference Donders, van der Heijden, Stijnen and Moons51)} . In contrast, multiple imputation uses distributions of the observed data to provide multiple plausible complete datasets, thus accounting for uncertainty in the missing values^{(Reference Sterne, White, Carlin, Spratt, Royston, Kenward, Wood and Carpenter49)}. Analyses are repeated on each dataset and the results are pooled to account for the variability due to imputation.

Finally, model-based approaches, such as regression and machine learning techniques (e.g. neural networks, k-nearest neighbours and random forest) use the observed data to predict missing values^{(Reference Jerez, Molina, García-Laencina, Alba, Ribelles, Martín and Franco52,Reference Emmanuel, Maupong, Mpoeleng, Semong, Mphago and Tabona53)} . Various packages exist which facilitate the implementation of imputation techniques^{(Reference Yadav and Roychoudhury54)}. Imputation of missing data should be done within validation steps.

Processing outliers

Outliers can arise due to a variety of reasons, including equipment malfunction, human error during data entry, or extreme (but valid) data measurements^{(Reference Dash, Behera, Dehuri and Ghosh55)}. Regardless of their origin, the presence of outliers should be investigated during exploratory data analysis and, if necessary, appropriate action should be taken to account for them^{(Reference Badillo, Banfai, Birzele, Davydov, Hutchinson, Kam-Thong, Siebourg-Polster, Steiert and Zhang9)}. Outliers can sometimes be detected through manual observation of features through descriptive characteristics or plots^{(Reference Kuhn and Johnson56)}. For example, if the median LDL cholesterol levels in a sample were 2.5 mmol/L, with an interquartile range of 1mmol/L but a maximum value of 25 mmol/L, it would be likely that one or some values were unrealistically high, possibly reflecting a mistake during manual entry or measuring equipment malfunction. Other times, however, what defines an outlier is less clear, and even in cases where extreme values are found, it is not always the case that some type of treatment is required.

Whilst general approaches based on statistical properties can be simple and easy to implement (e.g. values greater than the third quartile plus 1.5 *interquartile range)^{(Reference Dash, Behera, Dehuri and Ghosh55)}, they often do not consider the characteristics of the data or the problem at hand and are unjustified in many situations. For example, a small number of people may report a much higher income than the rest in a sample; however, these may be perfectly valid observations that do not require corrective action, despite their appearance in descriptive statistics or plots. Whenever possible, domain knowledge and additional related information should be used to guide decision-making for identifying and processing outliers, and this should be documented in the methodology.

If observations are determined to be outliers, if and how they should be dealt with should be based on logical grounds and on a case-by-case basis, whenever possible. When this is not possible, such as when features cannot be easily interpreted or in high-dimensional settings, identifying and dealing with outliers should be done carefully to avoid loss of information or risk of introducing bias.

Finally, the processing of outliers may be done on either the whole dataset or within validation steps, depending on the purpose. For instance, approaches to correcting genuine errors in the dataset, such as mistaken data entries, can be performed on the whole dataset as there is little risk of introducing bias. However, if characteristics of the data are used to identify and correct potential outliers, this should be done within validation steps.

Transforming features

Feature transformation, such as normalisation or standardisation, is another common data processing step and may be required by some algorithms when the features are on different scales. Examples include the regularised regression techniques Lasso and Ridge regression, where the scale of the features is relevant to the penalty term^{(Reference Tibshirani35)}, and k-nearest neighbours, where the scale of the features is important for distance calculations^{(Reference Peterson57)}. Whether transformation is required for the algorithms used for data analysis should be known by the analyst beforehand. However, it should be noted that the choice of transformation technique used can have a significant impact on the results and should not be an arbitrary decision^{(Reference Singh and Singh58–Reference van den Berg, Hoefsloot, Westerhuis, Smilde and van der Werf60)}. Hence, if transformation techniques are required, it may be warranted to investigate how the results change with different transformation techniques in order to assess the sensitivity of the findings. Transformations that depend on the data distribution (e.g., Z-score normalisation) should be performed within validation steps to prevent data leakage.

Discretisation of continuous features

This refers to converting features on a continuous scale to categorical ones. A common example in nutrition is the conversion of BMI in kg/m² to underweight, healthy weight, overweight and obese. Despite the dangers of this practice being well-documented and long-known^{(Reference Harrell43,Reference Naggara, Raymond, Guilbert, Roy, Weill and Altman61–Reference Sauerbrei and Royston65)} , it remains widespread in the literature.

Discretisation almost inevitably leads to a loss of information, which can hinder the predictive capacity of the model^{(Reference Naggara, Raymond, Guilbert, Roy, Weill and Altman61,Reference Royston, Altman and Sauerbrei64)} . The fewer the number of levels (or bins) in the newly formed category, the greater the loss of information. Another consequence of discretisation is the introduction of step functions where the response ‘jumps’ when moving from one level to the next within a category^{(Reference Harrell43)}. Unless this is justified (e.g. the pH at which a biological reaction occurs), such situations are usually undesirable and, in biological settings, may be unrealistic^{(Reference Sauerbrei and Royston65)}.

The decision regarding the number of bins to use (‘binning’) and the numerical limits which define them is also problematic^{(Reference Royston, Altman and Sauerbrei64)}. This is sometimes done in terms of quantiles (e.g. quartiles) or groups that make sense to humans (e.g. age groups of 40–49 years, 50–59 years, etc.), despite that such groups may not make sense to the problem at hand^{(Reference Altman62,Reference Bennette and Vickers63)} . An additional problem related to binning concerns the sensitivity of the results to the limits defining the bins. In the absence of well-defined cut-points based on prior work^{(Reference Naggara, Raymond, Guilbert, Roy, Weill and Altman61)}, this decision is left in the hands of the analyst. Unintentionally or otherwise, this can lead to cut-points being selected that support hypotheses or iteratively trying enough combinations of bins and cut-points until favourable results are found. The consequences of such practices include an increased chance of spurious findings, false positives and biased results^{(Reference Naggara, Raymond, Guilbert, Roy, Weill and Altman61,Reference Sauerbrei and Royston65,Reference Heavner, Phillips, Burstyn and Hare66)} .

Finally, discretisation of the outcome variable can influence the information that can be obtained from the results. Vastly different observations may be grouped together, whereas observations with only minimal differences may end up in different levels^{(Reference Naggara, Raymond, Guilbert, Roy, Weill and Altman61,Reference Bennette and Vickers63)} . For example, discretising blood pressure measurements into levels of hypertension (e.g. normotensive, pre-hypertension and hypertension) could lead to individuals with dangerously high blood pressure and those who are barely hypertensive being classified into the same group, whereas those only 1 mmHg apart could be assumed to have different risks^{(Reference Naggara, Raymond, Guilbert, Roy, Weill and Altman61)}. This can have important consequences for the conclusions derived from the results, resource allocation, and treatment or intervention options.

Hyperparameter tuning

Hyperparameters are modifiable parameters that affect the learning process of an algorithm. Examples include maximum depth on decision trees and the learning rate in neural networks. How these hyperparameters are tuned can have a significant impact on model performance. The tuning process involves fitting many different models, each with different hyperparameter configurations, to see which set of configurations leads to the best performance. Grid search and random search^{(Reference Bergstra, Ca and Ca74)} are common and relatively basic optimisation techniques, though more advanced techniques also include Bayesian Optimisation, Hyperband, and evolution strategies^{(Reference Bischl, Binder, Lang, Pielok, Richter, Coors, Thomas, Ullmann, Becker and Boulesteix75)}.

Feature selection

Feature selection aims to reduce the initial feature space by eliminating features which are less important to the model output, ideally while preserving predictive performance^{(Reference Chandrashekar and Sahin76)}. This is increasingly sought-after as data in the modern world are being collected on a wide range of variables, sometimes from only a small number of samples. For example, thousands of microbes can be collected from each individual in gut microbiome studies, yet such studies rarely have so many participants.

Many feature selection approaches exist^{(Reference Theng and Bhoyar77,Reference Pudjihartono, Fadason, Kempa-Liehr and O’Sullivan78)} , although the difficulty of the task of feature selection in the high-dimensional setting is often underappreciated^{(Reference Fan and Lv79)}. Especially when the number of features is much higher than the number of observations, identifying features which generalise to unseen data and distinguishing those that are relevant to the problem from those that are not can be challenging^{(Reference Fan and Lv79–Reference Saeys, Inza and Larrañaga81)}.

An additional challenge is feature selection stability, which refers to how sensitive the selected features are to perturbations of the data^{(Reference Khaire and Dhanalakshmi82)}. Ideally, the feature subset would contain features that are selected across multiple repeats of feature selection, each using different splits of the data for training and testing. Feature selection stability can be estimated by repeating feature selection across multiple different subsamples of the data, as part of robust internal validation schemes^{(Reference Saeys, Inza and Larrañaga81,Reference Khaire and Dhanalakshmi82)} (as described below).

Internal validation schemes

Many techniques for hyperparameter tuning, feature selection and data processing steps make use of the outcome variable, meaning there is risk of information leakage^{(Reference Boehmke and Greenwell83,Reference Whalen, Schreiber, Noble and Pollard84)} . Information leakage describes the situation in which the same observations in a dataset are used to both construct and evaluate the model, resulting in an optimistic evaluation of model performance on unseen data^{(Reference Boehmke and Greenwell83,Reference Whalen, Schreiber, Noble and Pollard84)} .

To minimise information leakage, robust internal validation schemes preserve an entirely unseen portion of the data that was not involved in data processing or model-building for validation. Various approaches exist for this and are discussed below (see below: Step 4. Evaluation), though in general they involve splitting the data for training, where data processing, hyperparameter optimisation and feature selection are performed, and testing, where evaluation is performed. This procedure is then repeated multiple times to estimate uncertainty and stability in the results^{(Reference Lones85)}. This can protect against spurious findings and make the results more robust. A visual representation of how a robust internal validation scheme might look can be seen in Fig. 2. Examples of machine learning experiments with good internal validation schemes can be seen in^{(Reference Acharjee, Finkers, Gf Visser and Maliepaard86,Reference Acharjee, Larkman, Xu, Cardoso and Gkoutos87)} .

Figure 2.

A robust internal validation scheme using nested cross-validation. Data are first split using cross-validation (outer loop; step 1). In each fold of the outer loop, cross-validation is used on the training data (dark blue) for data processing, hyperparameter optimisation and feature selection. This is known as the inner loop (grey box; step 2). The performance of the model selected in the inner loop is then validated on the outer fold test data (dark orange; step 3). This process is depicted in the large, light blue box, and is repeated in each fold of the outer loop (step 4). The whole process is then repeated multiple times to account for instability of the results depending on how the dataset is split (step 5).

Metrics

The metrics by which machine learning modes are evaluated are important since they not only reveal how a model performs from different perspectives but also determine how they are optimised^{(Reference Dinga, Penninx, Veltman, Schmaal and Marquand88)}. Hence, it is important that metrics are chosen that reflect the desired performance of the models with respect to the problem^{(Reference Vollmer, Mateen, Bohner, Király, Ghani, Jonsson, Cumbers, Jonas, McAllister and Myles11)}. In some cases, performance with respect to some metrics may be more important than others. For example, in a classification problem, it may be more important that all true cases are correctly identified even at the expense of incorrectly labelling true negatives as positives (i.e. high sensitivity, low specificity), such as in identifying children at risk of obesity for targeted advertisement for events at a local sports centre. In other circumstances, false positives can be costly, such as when cases predicted as positive are selected to undergo surgery. In this case, a low specificity could lead to unnecessary surgery and related complications. Alternatively, if there is no particular preference for one metric over others, composite metrics which consider multiple aspects of performance (e.g. F1 score^{(Reference Dinga, Penninx, Veltman, Schmaal and Marquand88)}) may be preferred.

Similar differences exist for metrics in regression. For example, mean-squared error (MSE) and root MSE (RMSE) punish larger errors more than mean absolute error (MAE), which can be desirable in some cases^{(Reference Naser and Alavi89)}. The coefficient of determination, R-squared, on the other hand, measures the proportion of the variation in the outcome that is explained by the features^{(Reference Chicco, Warrens and Jurman90)}. Because R-squared is limited between 0 and 1, it is easier to compare performance between different datasets that may have different variables on different scales^{(Reference Chicco, Warrens and Jurman90)}. However, it should be noted that it is not always the case that optimal performance with respect to one single metric is desired; instead, models may be evaluated across different metrics with each evaluating different aspects of performance^{(Reference Chicco, Warrens and Jurman90)}.

Whereas in supervised machine learning labels are available for the data, providing a ground truth on which to score predictions, this is not the case with unsupervised machine learning, which uses different metrics for assessing model performance. The exact metrics depend on the specific unsupervised task and often involve measures of homogeneity or dissimilarity. For example, for clustering, one of the most common unsupervised machine learning tasks, metrics include silhouette score, Calinski-Harabasz coefficient, and Dunn index^{(Reference Palacio-Niño and Berzal91)}. The different ways in which these metrics determine the quality of clustering can lead them to arrive at different optimal solutions. Hence, unless there is rationale to prefer one over another, it may be interesting in unsupervised approaches to explore how results differ across different metrics and how this influences the eventual conclusions drawn from the machine learning experiment.

Train-test split

Splitting the dataset into a portion for training (model-building) and a portion for testing (evaluating model performance) is the most basic way in which a model can be evaluated^{(Reference Kirk, Kok, Tufano, Tekinerdogan, Feskens and Camps1)}. The data may be split entirely randomly or in a way that maintains certain aspects of the data characteristics in each portion, such as ensuring the same proportion of cases and controls in the training and test data. The cost of this simplicity is that the results obtained from a train-test split can be highly dependent on how the dataset was split, which can lead to significant variability in performance metrics, especially with smaller datasets. While train-test splits are more reliable on large datasets or those with good signal-to-noise ratios, alternative methods such as cross-validation (discussed below) offer a more robust evaluation of model performance. If train-test splits are to be used, they should be repeated in order to increase the stability of the results^{(Reference Raschka and Mirjalili92)}.

The importance of repeating validation techniques with different subsamples of the data is shown in Fig. 3, which uses a decision tree classifier with the Pima Indians Diabetes dataset^{(Reference Smith, Everhart, Dickson, Knowler and Johannes93)} to demonstrate the effect of validation technique, the number of times it is repeated, and sample size on the stability and uncertainty of the results. The code for the analysis and location of the dataset can be seen in the supplementary code. Fig. 3 makes clear the importance of repeating validation techniques on smaller datasets in order to increase the certainty of the results. However, this is not always observed in the literature, despite the fact that relatively small samples are often used in machine learning experiments.

Figure 3.

The effect of validation technique, the number of times it is repeated, and sample size on the stability and uncertainty of the results. One repeat of cross-validation (second row) is an immediate improvement over one repeat of train-test split (first row) because changes in the training and test data in each fold provide an indication of the stability of the AUC-ROC scores. Repeating the validation procedure multiple times with different subsamples also allows stability to be estimated, with this being more effective in cross-validation (fourth row) than train-test split (third row) because there are more test scores. Both instability and uncertainty tend to decrease as sample size increases.

Cross-validation

Cross-validation approaches are a group of resampling procedures that can be used for model selection and to estimate the generalisability of the model^{(Reference Hastie, Tibshirani and Friedman14)}. In k-fold cross-validation, the dataset is split into k folds so that each observation is used once for testing and k-1 times for training^{(Reference Hastie, Tibshirani and Friedman14)}. The number of folds k is often chosen based on computational efficiency and a suitable bias-variance trade-off, with values generally between 5 and 10 being used in practice^{(Reference Raschka and Mirjalili92)}. Different variations of cross-validation exist, such as leave-one-out cross-validation and stratified and grouped cross-validation (see Kirk et al.^{(Reference Kirk, Kok, Tufano, Tekinerdogan, Feskens and Camps1)}). Unlike train-test split, there are k number of test score results, which may be presented individually (as in Fig. 3) or aggregated into summary statistics (e.g. mean across all test folds). Different varieties of cross-validation exist to allow stratified cross-validation or account for dependent observations.

Cross-validation is an improvement over train-test split because all of the data are used for training and testing, making it less sensitive than a single split in the data. Even so, cross-validation can still be sensitive to how the data is split within each fold and test scores between each fold can differ greatly, especially with smaller datasets. To account for this, cross-validation can be repeated multiple times over, ensuring that on each repeat different splits of the data are used within each fold (Fig. 2)^{(Reference Lones85,Reference Raschka and Mirjalili92)} .

Nested cross-validation

One concern with cross-validation is that data processing steps, hyperparameter tuning and feature selection are performed on the same data used to evaluate the model performance. Nested cross-validation deals with this by adding another cross-validation loop (known as the inner loop) within the training data of each fold of the outer loop (see Fig. 2). The inner loop is used for tuning hyperparameters and feature selection, and this is then validated on the test data of the outer loop (for details see^{(Reference Wainer and Cawley94)}). This ensures a portion of the data that was not involved in any part of the model-building process is available to estimate performance on unseen data.

Nested-cross validation reduces the chance of information leakage and allows for an unbiased estimation of model performance^{(Reference Cawley and Talbot95)}. However, in doing so it greatly increases computational time, and whether this justifies the reduction in bias has been called into question^{(Reference Wainer and Cawley94)}. Analysts may prefer to first perform traditional cross-validation and then, if the results appear promising, validate that the findings are not the result of optimism due to information leakage by using nested cross-validation. This can circumvent wasting time and computer resource on machine learning experiments that would not be fruitful anyway.

Calibration

An important yet often overlooked concept for supervised classifiers is calibration, which refers to the alignment between the predicted probabilities and the observed outcomes^{(Reference Steyerberg and Harrell96)}. For example, a classifier that is trained to predict diabetes would be well-calibrated if the observed occurrence of diabetes was close to 10% for those whose predicted probabilities were close to 10%. Model miscalibration may not always be apparent in internal validation, but low calibration can mislead expectations and be problematic when looking at high or low-risk groups (where it may be needed most) or external datasets^{(Reference Van Calster, McLernon, Van Smeden, Wynants, Steyerberg, Bossuyt, Collins, MacAskill, Moons and Vickers97)}. This can have important implications for the action taken in response to the predictions made by the model.

In contrast to predicting class labels directly, there are advantages to working with predicted probabilities^{(Reference Van Calster, McLernon, Van Smeden, Wynants, Steyerberg, Bossuyt, Collins, MacAskill, Moons and Vickers97)}. Firstly, it is often more relevant to know the chance of an event occurring rather than a class label without further context. For example, whilst two individuals may both be predicted to belong to the same class, the probability for one being 55% and the other being 95% shows a clear difference in the confidence of their predicted class membership. This can have important implications, such as how resources are allocated in response to model predictions (e.g. children with a higher predicted probability of malnutrition are prioritised for corrective nutritional intervention). Using probabilities also means that custom thresholds can be more easily set and adjusted, which can be desirable when the cost or benefit of correct classification is not the same as that for misclassification. Finally, probabilities are also inherently more interpretable than class labels, which is a desired property of machine learning procedures^{(Reference Doshi-Velez and Kim98)}.

Miscalibration may occur due to the data or the model used to fit the data (i.e. overfitting)^{(Reference Van Calster, McLernon, Van Smeden, Wynants, Steyerberg, Bossuyt, Collins, MacAskill, Moons and Vickers97)}. It is most often diagnosed by plotting the predicted probabilities against the observed frequencies (known as calibration curves or reliability plots), with a straight-line y=x representing perfect calibration^{(Reference Dormann99)}. Due to differences in how they operate, some models drive probabilities away from 0 and 1 (e.g. support vector machines), whereas others more readily predict probabilities at the extremes (e.g. Naïve Bayes) and others still are naturally well-calibrated (e.g. logistic regression)^{(Reference Kull, De Menezes, Filho and Flach100,Reference Niculescu-Mizil and Caruana101)} . Following the identification of miscalibration, calibration correction techniques can be used. Two well-known approaches are Platt scaling^{(Reference Platt102)}, which is used for those with S-shaped calibration curves, such as those seen for support vector machines, and isotonic regression, which is capable of modelling more complex shapes but with an increased risk of overfitting^{(Reference Niculescu-Mizil and Caruana101)}.

External validation

A primary goal of machine learning applications is to make predictions using new data that were not available during training. While good internal validation schemes provide an estimate of how well prediction models do this, they can still be optimistically biased^{(Reference Vollmer, Mateen, Bohner, Király, Ghani, Jonsson, Cumbers, Jonas, McAllister and Myles11,Reference Ho, Phua, Wong and Bin Goh103)} . External validation involves validating the generalisability of the model on a dataset that reflects the target population but comes from another source^{(Reference Vollmer, Mateen, Bohner, Király, Ghani, Jonsson, Cumbers, Jonas, McAllister and Myles11,Reference Steyerberg and Harrell96,Reference Ho, Phua, Wong and Bin Goh103–Reference Siontis, Tzoulaki, Castaldi and Ioannidis107)} .

External datasets may differ in key characteristics such as the location, time and methods of data collection, as well as the individuals responsible for collecting the data. However, it is crucial that key features remain constant. For instance, when externally validating a model predicting disease outcomes, it is imperative that the disease is defined in the same way in both datasets to prevent differences in model performance reflecting disease definitions rather than generalisability. To further enhance robustness and reduce the chance of optimistic bias, external validation may be performed by an independent research group that was not involved in model development or data collection for the initial dataset^{(Reference Vollmer, Mateen, Bohner, Király, Ghani, Jonsson, Cumbers, Jonas, McAllister and Myles11)}.

It is worth noting that some studies sometimes erroneously claim that external validation was used, when in fact their ‘external validation’ set is simply a test set or an extension of the original dataset (e.g.^{(Reference Heshiki, Vazquez-Uribe, Li, Ni, Quainoo, Imamovic, Li, Sørensen, Chow and Weiss108)}). This is sometimes a semantic issue rather than intentional misrepresentation^{(Reference Siontis, Tzoulaki, Castaldi and Ioannidis107)}, but care should be taken when interpreting such results because external validation is a stronger indication of generalisability than internal validation with a larger dataset. Even still, it should be kept in mind that good external validation performance does not prove generalisability^{(Reference Cabitza, Campagner, Soares, García de Guadiana-Romualdo, Challa, Sulejmani, Seghezzi and Carobene105,Reference Van Calster, Steyerberg, Wynants and van Smeden109,Reference la Roi-Teeuw, van Royen, de Hond, Zahra, de Vries, Bartels, Carriero, van Doorn, Dunias and Kant110)} . External validation performance is still dependent on the external dataset used, and conclusions made must bear the characteristics of this dataset (e.g. location, time, sample characteristics, etc.) in mind. As external validation is repeated on a greater number of external datasets that are more different from the dataset used to develop the model, confidence in the generalisability of the model increases^{(Reference Van Calster, Steyerberg, Wynants and van Smeden109,Reference la Roi-Teeuw, van Royen, de Hond, Zahra, de Vries, Bartels, Carriero, van Doorn, Dunias and Kant110)} .

Feature importance

It is often desirable to know which features were important for machine learning models in generating their output, despite that there is usually no ground truth for feature importance and the concept itself is poorly defined^{(Reference Linardatos, Papastefanopoulos and Kotsiantis111)}. Different approaches estimate feature importance in different ways, often arriving at different conclusions^{(Reference Molnar, König, Herbinger, Freiesleben, Dandl, Scholbeck, Casalicchio, Grosse-Wentrup and Bischl112)}, though popular approaches with good statistical properties include SHAP^{(Reference Lundberg, Allen and Lee113)}, Lime^{(Reference Ribeiro, Singh and Guestrin114)}, and permutation-based feature importance^{(Reference Altmann, Toloşi, Sander and Lengauer115)}, amongst others^{(Reference Linardatos, Papastefanopoulos and Kotsiantis111)}.

Some algorithms can provide feature importance estimates as part of their architecture (so-called ‘built-in’ feature importance). Such built-in feature importance estimates, whilst convenient, can come with significant drawbacks. This is particularly well-described for random forest^{(Reference Breiman71)}, which can provide biased results based on the scale of continuous features and number of categories in categorical ones, as well as when multicollinearity is present^{(Reference Strobl, Boulesteix, Zeileis and Hothorn116–Reference Toloşi and Lengauer118)}. While no feature importance method is perfect^{(Reference Linardatos, Papastefanopoulos and Kotsiantis111)}, it is concerning how often random forest-derived feature importances are reported in scientific literature without consideration for their potential limitations or how the results may be different had other feature importance techniques been used^{(Reference Kirk, Catal and Tekinerdogan2)}, creating a false sense of certainty for feature importance estimates calculated with this approach.

Inappropriately calculated feature importance estimates can lead to both false positives (i.e. unimportant features falsely identified as important) and false negatives (i.e. important features falsely identified as unimportant), both of which reduce the quality of the literature. In response to this, analysts should first think carefully about how their choice of feature importance technique relates to their data^{(Reference Molnar, König, Herbinger, Freiesleben, Dandl, Scholbeck, Casalicchio, Grosse-Wentrup and Bischl112)}. It should be known if there are feature interactions or collinearity present, and how the chosen feature importance techniques may be affected by this^{(Reference Linardatos, Papastefanopoulos and Kotsiantis111)}. Analysts should also be open to comparing results across various suitable techniques and place their findings in the context of the model and explainable AI technique used^{(Reference Saarela and Jauhiainen119)}, rather than making conclusions about feature importance in a general sense.

Additionally, an important but sometimes underappreciated fact is that feature importance estimates on models with poor predictive performance cannot be trusted, and analysts should resist reporting feature importance estimates in such cases^{(Reference Molnar, König, Herbinger, Freiesleben, Dandl, Scholbeck, Casalicchio, Grosse-Wentrup and Bischl112,Reference Murdoch, Singh, Kumbier, Abbasi-Asl and Yu120)} . Moreover, similarly to the results from predictions generated by machine learning models, feature importance estimates may also be dependent on how the dataset was split for training and testing^{(Reference Wang, Yang and Luo121,Reference Kalousis, Prados and Hilario122)} . It can be informative to estimate the stability of feature importances by seeing how they change across many repeats of model fitting on different splits of the data^{(Reference Linardatos, Papastefanopoulos and Kotsiantis111,Reference Wang, Yang and Luo121,Reference Kalousis, Prados and Hilario122)} . Finally, the subfield of explainable AI is not without issues and has been criticised for the consequences of unfaithful explanations, complex explanations, failing to consider that important features may change over time, and ambiguity regarding terminology^{(Reference Rudin123–Reference Yang, Wei, Wei, Chen, Huang, Li, Li, Yao, Wang and Gu125)}. Hence, researchers should be aware of the shortcomings of explainable AI techniques they decide to use and the potential costs of these in the context of the problem.