3.2.1. Cohort Studies

H&C actuaries are likely familiar with cohort studies, where a cohort of people or policies is followed over time, as this is the default study design for experience analysis of an insurance portfolio. Cohort studies are useful for studying incident cases and establishing causal relationships. For most actuarial cohort studies, exposure data is collected prospective to an outcome of interest, which reduces potential recall bias. Recall bias typically occurs when exposure is collected retrospective to the outcome. For example, an individual may be more or less likely to recall the risk factors they were exposed to after they experience an outcome of interest. When the outcome of interest is rare, these studies require progressively larger cohorts and longer follow-ups to ensure sufficient statistical power (Section 3.6). An example of a cohort study is following a cohort of people over time to compare lung cancer incidence rates between smokers and non-smokers.

3.2.2. Case Control Studies

Case-control studies are more efficient study designs for rare outcomes or where follow-up time is limited. Case-control studies compare individuals with a particular outcome (cases) to those without it (controls) to analyse potential risk factors. Unlike cohort studies, there is no follow-up or exposure time. Case-control studies are efficient but suffer from various potential biases (e.g. recall bias), especially when data is retrieved retrospective to the outcome. An example of a case-control study is a study comparing fraudulent claims (cases) to genuine claims (controls) to identify predictors of fraudulent claims.

3.2.3. Cross-Sectional Studies

Cross-sectional studies are observational studies that analyse data on exposure and outcome from a population at a single point in time. Cross-sectional studies can analyse the prevalence of a condition, but not the incidence, due to the lack of a temporal element. For this reason, cross-sectional studies are useful for demographic distributions and outcomes where the individual remains within the data set to be observed (such as low mortality diseases like diabetes). Cross-sectional studies are not useful for high mortality conditions such as stroke since, at any given time, the number of prevalent cases will be low. An example of a cross-sectional study is an analysis of life insurance policy ownership by socioeconomic status.

Note that interventional study designs (clinical trials) and descriptive study designs (case reports, case series and ecological studies) are out of scope of this framework as these study designs are generally not useful for actuarial projects.

3.3.1. Reproducible Data

Ensuring reproducible data involves maintaining consistent data sources, documenting pre-processing steps, and storing both raw and processed data sets in repositories accessible to internal reviewers. Techniques such as data versioning, hashing for integrity checks, and structured metadata (e.g., using Findable, Accessible, Interoperable and Reusable (FAIR) principles as per Wilkinson et al., Reference Wilkinson, Dumontier, Aalbersberg, Appleton, Axton, Baak, Blomberg, Boiten, da Silva Santos, Bourne, Bouwman, Brookes, Clark, Crosas, Dillo, Dumon, Edmunds, Evelo, Finkers, Gonzalez-Beltran, Gray, Groth, Goble, Grethe, Heringa, ’t Hoen, Hooft, Kuhn, Kok, Kok, Lusher, Martone, Mons, Packer, Persson, Rocca-Serra, Roos, van Schaik, Sansone, Schultes, Sengstag, Slater, Strawn, Swertz, Thompson, van der Lei, van Mulligen, Velterop, Waagmeester, Wittenburg, Wolstencroft, Zhao and Mons2016) help ensure that analyses can be replicated by colleagues under identical conditions.

3.3.2. Reproducible Methods

Version control tools such as Git enable actuaries to maintain an audit trail of the code base, revert to earlier analysis stages, systematically review model changes over time and facilitate collaborative workflows. Git can also support data lineage tracking by indexing data pre-processing.

Randomness in data science methods (e.g. tree-based models, neural networks, clustering algorithms, bootstrapping and cross-validation) can be controlled by setting a random seed. Despite this, fixing a seed does not guarantee full reproducibility due to variations in hardware and software dependencies, underscoring the importance of containerisation tools like Docker or virtual environments for computational consistency. For these reasons, it is also best practice to document the software environment and package versions used.

Model and workflow ownership, as per TAS 100 (Financial Reporting Council, 2023), also supports reproducibility by having a clear point of contact available for queries.

3.3.3. Reproducible Documentation

Well-structured workflows and comprehensive documentation support reproducibility. This includes using programming tools such as R Markdown and Jupyter Notebook that integrate code, output and explanatory text; including documentation around key decisions made during data pre-processing and analytics, into a single shareable document.

3.4.1. Data Selection Criteria

Well-designed studies include data selection criteria that define which subjects should be included and excluded. Data selection criteria should be defined to ensure appropriateness of the data. Important criteria involve geographical region, date of birth, commencement year (of insurance policy), calendar years of follow-up and medical history. Failure to specify data selection criteria can result in non-representative or irrelevant study participants. For example, when using external data sets such as a data set from the Continuous Mortality Investigation (CMI), actuaries should take care to understand the underlying selection rules used by the original data collectors, such as the exclusion of rated lives in analyses of standard-term assurances.

Data selection criteria vary by study design as cohort studies and cross-sectional studies enrol subjects based on an exposure (such as a hospital visit or being a policy-holder), while case-control studies enrol subjects based on an outcome (such as a fraudulent claim).

3.4.2. Data Bias

Data bias may arise from inherently biased data sampling methods, historical bias within the data (including bias due to over-representation of certain groups) and/or omission of key predictive attributes from the data (Financial Reporting Council, 2024).

Data bias can arise from various factors, examples are:

1. 1. Anti-selection, where individuals with poorer health or pre-existing conditions are more likely to buy or retain coverage.

2. 2. Data drift, where the statistical distributions of input features change over time. For example, the BMI distribution changing over time and the proportion of smokers decreasing over time.

3. 3. Concept drift, where the relationship between an input feature and the outcome changes by issue year. For example, inflation affects the relationship between sum assured and outcomes over time and improvements in underwriting could affect the impact of duration on outcomes.

4. 4. Reporting bias where not all data is captured consistently or accurately. For example, underreporting of smoking in the data set.

5. 5. Outcome definition bias, where outcome definitions (e.g. ICD-11 codes), or even conditions covered can change over time.

6. 6. Omission of key predictive features, where key predictive features are missing from the data. For example, if smoking is not captured in a life insurance claim analysis, part of the effect of smoking could be assigned to features correlated with smoking such as males and low sums assured.

Mitigating data bias often requires bespoke solutions and can be adjusted for within a model or data set. For example, sum assured can be inflation-adjusted in historic data. Various strategies including up-sampling and weights can be considered (Section 3.12) to deal with data drift, and data selection criteria can also be utilised to deal with data bias, for example by excluding unreliable or non-representative data. Section 3.5 discusses some techniques in identifying unreliable or non-representative data.

3.4.3. Ethical and Privacy Controls

Actuaries should ensure that all data sets comply with relevant data protection regulation including UK GDPR (Regulation (EU) 2016/679, 2016) and the Data Protection Act 2018. Data collection and use should also comply with ethical approval and participant consent, where appropriate. Additionally, data analytics projects will be subject to internal governance processes, including legal, regulatory and audit requirements, as well as internal risk management guidance and professional guidance such as TAS 100 (Financial Reporting Council, 2023). The regulatory landscape is subject to constant change. For example, whilst direct use of protected characteristics is widely prohibited, indirect discrimination by proxy variables or complex algorithms is currently a regulatory grey area (Xin & Huang, Reference Xin and Huang2024). Therefore, actuaries are encouraged to keep up to date with regulatory developments.

Advanced algorithms and big data may elevate privacy risks and inadvertent use of protected characteristics by proxy. Structured ethical checkpoints throughout the analytics project (problem definition, data, modelling, evaluation and deployment), as discussed in recent actuarial literature (Huang, Reference Huang2025) may help guard against these risks. These checkpoints help embed principles such as accountability, transparency and privacy. Transparency is further aided by model explainability (Section 3.15). Bias and fairness are further covered in Section 3.14.6.

3.5.1. Structural Issues

Structural issues include duplicated rows, columns and lives, incorrect variable types, and redundant columns that can introduce inefficiencies and distort analytical outcomes. Duplicated rows may arise from data entry errors or merging data sets, leading to overrepresentation of certain observations. Duplicated lives can arise from a claim by the same person on multiple policies, or tranches of the same policy. Similarly, redundant columns, often created during data processing, can add unnecessary complexity without providing additional information.

Incorrect variable types, such as numerical values or dates stored as text, can interfere with calculations and statistical modelling as well as visualisations. Addressing these structural issues early prevents downstream errors and improves data integrity. Automated scripts and validation checks can help identify and correct such problems efficiently.

3.5.2. Content Issues

Content issues in data cleaning include missing data and outliers as well as inconsistent and implausible data. Outliers can be identified by visualising histograms or performing range checks, but their interpretation depends on context. For instance, in lab tests and biometrics, outliers can be due to different units (e.g. mmol/L vs. mg/dL for cholesterol levels), or system conversions (imperial versus metric). In financial data sets, negative values can indicate refunds rather than errors. Addressing outliers or missing data ideally involves identifying their root cause (e.g. by speaking with the data supplier) before determining whether correction, transformation, flooring/capping or exclusion is appropriate.

Missing data can be missing completely at random (MCAR), missing at random (MAR) and missing not at random (MNAR) (Mack et al., Reference Mack, Su and Westreich2018).

1. 1. MCAR occurs when missingness is unrelated to any observed or unobserved variables. An example of this is a batch of policy holders missing smoking information due to a data entry error. Although statistical power is reduced, there is no introduction of bias into the analysis.

2. 2. MAR occurs when missingness is systematically related to observed, but not unobserved data. For example, younger (observed) policyholders being less likely to disclose smoking habits. MAR can introduce bias if not properly adjusted for but can often be corrected for.

3. 3. MNAR occurs when missingness is related to unobserved factors. For example, high-risk (unobserved) individuals omitting disclosure of health conditions. MNAR is most problematic since it cannot be corrected for since the factors influencing missingness are unobserved. MNAR is most likely to introduce bias in any subsequent model.

Dealing with outliers and missing data requires a tailored approach, as these issues are often non-random and can introduce bias if mishandled. One common solution, imputation, involves replacing missing values (or outliers) using statistical or machine learning techniques in order to retain useability for analysis or modelling. Simple imputation methods, such as replacing missing values with the mean or median, can distort distributions and weaken predictive models. Multiple imputation by chained equation (MICE) (White et al., Reference White, Royston and Wood2011) has traditionally been regarded as the gold standard for imputation of missing data. Recently, various more sophisticated techniques utilising tree-based models such as missForest are available and have been outperforming MICE in certain studies (Luo, Reference Luo2022; Waljee et al., Reference Waljee, Mukherjee, Singal, Zhang, Warren, Balis, Marrero, Zhu and Higgins2013). Imputation may not produce reliable results after a certain threshold of missingness is met. Unfortunately, this threshold is highly dependent upon type of missingness and is specific to each data set. For categorical features (e.g. smoking status) a simple solution can be to introduce a new category: “unknown.” Should different types of missingness (e.g. blank values vs. missing values) be present, these should be considered separately as these could represent different underlying issues.

3.5.3. Data Consistency Checks

Various types of consistency checks can be considered (see Table 3).

Table 3.

Consistency checks classes

3.6.1. Credibility

Credibility is the weighting of different estimates to come up with a combined estimate. Generally, credibility combines observed experience with a more stable, yet less individualised estimate (i.e. the a priori assumption). Credibility is especially useful when observed data is limited or volatile, which may result in unreliable model predictions. Traditionally, limited fluctuation (LF), greatest accuracy (GA) and Bayesian methods have been used for credibility (Atkinson, Reference Atkinson2019). More recently, LASSO and random effects models allow for credibility to be integrated within the model itself.

3.6.2. Sample Size Calculation

When applying for access to certain health and care data sets, sample size calculations may be required by ethics committees to ensure studies are robust and well-designed.

Sample size calculations prevent unreliable, inconclusive, and non-reproducible results as well as reduce the risk of false negative results (Ioannidis, Reference Ioannidis2005). This is similar to actuarial credibility theory -where a data set must reach a certain size before we can rely on observed experience rather than external assumptions. Sample size calculations may not be relevant for actuarial projects where the objective is to extract the maximum number of insights from a data set, or for pricing exercises.

Sample size calculations require (Sharma et al., Reference Sharma, Mudgal, Thakur and Gaur2020):

• • A null hypothesis and alternative hypothesis

• • Acceptable significance level (the probability of incorrectly rejecting the null hypothesis), typically set at 5%

• • Study power (the probability of correctly rejecting the null hypothesis), typically set at 80%

• • Expected effect size, typically expressed as a relative risk, odds ratio or hazard ratio

• • Underlying event rate in the population

• • Margin of error

• • Standard deviation in the population

• • A one tail and two tail inferential statistical test

• • A design effect

3.9.1. Numeric Features

Numeric feature engineering involves transforming numeric variables (both discrete and continuous) to better capture their relationships with the target variable. Since relationships between predictors and outcomes are rarely perfectly linear in real-world data, these transformations are critical for improving model performance, especially for linear models, including GLMs and their regularised variants, such as Ridge, Lasso and Elastic Net.

Typically, the process begins with an exploratory data analysis to understand the distribution of each numeric feature and its empirical relationship with the target variable. Visualisations such as scatter plots and density plots can help with deciding if any transformation is required. For instance, logarithmic transformations may be useful for features with significant skewness, whilst non-linear relationships with the target may justify including additional polynomial terms. The principal methods of transforming numeric features are listed in Table 4.

Table 4.

Overview of feature engineering methods for numeric features

Unlike linear models, most machine learning models such as neural networks and tree-based models can discover non-linear relationships on their own without explicit transformation. But feature engineering is still important because the model performance can still depend on how the input data is represented (Goodfellow et al., Reference Goodfellow, Bengio and Courville2016). Particularly for neural network models, the likelihood of convergence is higher and the rate of convergence is faster during model training when all features are standardised. A popular way of doing this is Z-score normalisation (LeCun et al., Reference LeCun, Bottou, Orr, Müller, Orr and Müller1998b), where $${z_i}$$ is the normalised version of the original feature $${x_i}$$ , $${\mu _i}$$ is the mean of $${x_i}$$ and $${\sigma _i}$$ is the standard deviation of $${x_i}$$ :

$${z_i} = \;{{{x_i} - \;{\mu _i}} \over {{\sigma _i}}}$$

3.9.2. Categorical Features

Categorical features are not ingested by most models and require encoding into numeric values. The default is often dummy encoding, or one-hot encoding. Both dummy encoding and one-hot encoding generate binary columns for each of the categories. Contrary to one-hot encoding, dummy encoding drops one category to avoid issues with multicollinearity in linear models (the dropped category becomes the intercept or baseline). These encodings are problematic with high-cardinality features because of the numerous columns created that increase the risk of overfitting to training data and model training computational resources. This problem is exacerbated further when variable interactions (Section 3.9.3) are allowed, whether done automatically by the model or manually by domain experts. To mitigate this issue, dimensionality reduction techniques such as PCA can be used post-categorical encoding to condense the full feature set into orthogonal principal components.

More sophisticated encoding techniques can be used to transform a categorical column into a dense numeric feature. For instance, target encoding can be used by mapping each category to the average value of the target variable (Section 3.10.2). However, using this method naively can cause data leakage (Section 3.7), which in turn leads to overfitting to the training data. An effective means of reducing data leakage is to apply cross-validation on target coding of each categorical variable, as shown in Figure 1. For unseen data, the target encoded values are based on the entire training data set.

Figure 1.

How to incorporate cross-validation into target encoding to prevent data leakage in training data with 3 random folds.

Categorical embedding provides an alternative to target encoding by isolating the individual effect of each category. This technique is widely used in Natural Language Processing (NLP), in which words are converted into dense numeric vectors with considerably fewer dimensions than one-hot vectors. Word embeddings are learnt by training neural networks on tasks like predicting context words (Mikolov et al., Reference Mikolov, Chen, Corrado and Dean2013). In a similar fashion, categorical embeddings can be trained on prediction tasks. The resulting model’s coefficients assigned to each category will become its numeric representation.

Some ML models do not need a separate procedure to encode categorical features, as they can natively encode those features. Examples of these kind of models are CatBoost (Prokhorenkova et al., Reference Prokhorenkova, Gusev, Vorobev, Dorogush and Gulin2018) and LightGBM (Ke et al., Reference Ke, Meng, Finley, Wang, Chen, Ma, Ye and Liu2017). Also, grouping rare or similar categories together decreases cardinality and reduces overfitting. The cost of grouping, however, is reduced granularity that may result in losing valuable information differentiating the original categories. Grouping can be performed either manually with domain knowledge or automatically with unsupervised learning techniques like K-means.

3.9.3. Feature Interactions

Feature interaction happens when the influence of a feature on the target variable relies on the values of other features. In these instances, the combined effect cannot be isolated by individual features.

Most machine learning models can automatically learn feature interactions. This includes tree-based models and neural network models. On the other hand, models traditionally used by actuaries are those that can be represented explicitly using a regression formula, or more generally speaking, have an additive structure: $$g\left( {E\left[ y \right]} \right) = \;{\beta _0} + \sum {f_i}\left( {{x_i}} \right)$$ , where $$E\left[ y \right]$$ is the expected value of the target variable, $$g$$ is the link function and $${x_i}$$ is a feature. These two model classes represent two distinct modelling cultures, which are discussed in Section 3.10.1.

A main drawback of using additive models is that interaction terms need to be explicitly defined. Construction of these interaction terms used to be a cumbersome, time-consuming process requiring deep domain expertise. However, with the advancement of data science, an automated way of developing additive models is to (Tam & Luteijn, Reference Tam and Luteijn2025):

1. 1. Create a baseline model that contains just individual effects;

2. 2. Employ GBMs to identify interaction effects by training the model to predict residuals

3. 3. Include the interaction terms and retrain the additive model

An interaction term most H&C actuaries are familiar with is the interaction between smoking status and age in relation to mortality. Figure 2 shows the expected term assurance mortality rates by policyholder age across all genders and durations (Continuous Mortality Investigation, 2021), splitting between those underwritten as smokers and non-smokers. The relative mortality of smokers compared to non-smokers increases with age up to around the 80s, after which it begins to decline slightly. Therefore, the smoker excess mortality risk by age cannot be captured by a single loading.

Figure 2.

CMI expected term assurance mortality rates by age: smokers (orange) versus non-smokers (blue) (all genders, all durations; authors’ analysis).

3.10.1. Regularisation

Regularisation is a statistical technique that improves model performance by adding a penalty for complexity which encourages simpler models with more reliable predictions. It reduces the influence of less significant predictors which prevents overfitting. This is similar to actuarial credibility (Section 3.6.1), which balances individual and collective experience to improve estimates. In both cases, the goal is to achieve a robust and generalisable result by controlling for noise and over-reliance on sparse or unreliable data. Regularisation has been implemented in various machine learning algorithms such as gradient boosting, neural networks and penalised regression. Regularisation can be:

1. 1. L1 regularisation (LASSO) (Tibshirani, Reference Tibshirani1996), where a penalty is added relative to the absolute size of the coefficients. L1 regularisation can perform feature selection by shrinking regression coefficients to zero, eliminating them from the model.

2. 2. L2 regularisation (ridge), where a penalty is added relative to the squared values of the coefficients, shrinking them towards zero, but not eliminating them from the model.

LASSO regression analysis has widespread use in the insurance space due to its transparency, innate protection against the risk of overfitting and resulting parsimonious models.

3.10.2. Ensemble learning

Ensemble learning can be used either as:

1. 1. A meta-model combining predictions from independent models of different types

2. 2. A distinct model class combining weaker learners (i.e. models that perform slightly better than random guessing) of the same type systematically into an overall model (e.g. Random Forest or GBMs).

In both cases, ensemble models make use of individual models to make predictions more accurately than any one model on its own. This enhanced predictive ability of a group of models is an example of the “wisdom of the crowd.” The effectiveness of ensemble learning hinges on two primary factors: model diversity and model accuracy. Greater diversity and accuracy among models enhance an ensemble’s predictiveness (Ali & Pazzani, 1995). In other words, constituent models should be accurate but fail on different examples. This allows the ensemble to average out individual model biases.

Ensemble learning can also reduce the variance of model predictions (Wyner et al., Reference Wyner, Olson, Bleich and Mease2017). In this context, variance measures how sensitive a model is to changes in the training data. High variance means small changes in the training data will lead to large changes in the model’s predictions and is not desirable in insurance applications. For example, when a pricing model has high variance, a model refresh may lead to significant changes in premiums for policyholders upon renewals, even when there have been no material changes in their personal attributes, potentially leading to confusion and dissatisfaction among existing customers.

There are three main categories of ensemble learning, summarised in Table 5.

Table 5.

Main ensemble learning methods

3.10.3. Gradient Boosting Machines

GBMs are a special type of ensemble model that are very effective in analysing tabular data prevalent in actuarial analysis.

GBMs construct multiple decision trees on an iterative basis. Each new tree aims to predict residual errors that the previous ones have not been able to collectively accounted for. This approach builds upon AdaBoost, one of the first boosting algorithms that popularise boosting as a modelling technique. As AdaBoost is designed for binary classification problems (Freund & Schapire, Reference Freund and Schapire1997), GBMs extend the boosting methodology from exponential loss to any loss function that is differentiable (Friedman, Reference Friedman2001). Thus, GBMs are now suitable for both regression and classification tasks.

Since the first GBM methodology was proposed, several best practices have been developed to make it more robust and less prone to overfitting. These include stochastic gradient boosting with random sampling on training data or features (Friedman, Reference Friedman2002) and introduction of regularisation parameters to control model complexity as well as early stopping and learning rate (Bühlmann & Hothorn, Reference Bühlmann and Hothorn2007). More recently, XGBoost, LightGBM, and CatBoost have emerged as the most popular GBM methods with their own open-source packages (Mooney, Reference Mooney2022), each with its own distinct training regimes and functionalities. Each GBM type has its own bespoke way of growing decision trees:

• • XGBoost constructs trees level-by-level to their full depth and then cuts those branches for which loss improvement is below a minimum threshold (Figure 3).

  Figure 3.

  XGBoost tree structure – before and after pruning, with min split loss $$\;\left( \gamma \right) = 0.05$$ .

  

• • LightGBM grows trees leaf-wise, expanding the leaf that reduces loss by the largest amount, making them asymmetric (Figure 4).

  Figure 4.

  Light GBM grows trees leaf-wise (asymmetric), while CatBoost grows trees level-wise with symmetric structure.

  

• • CatBoost grows trees level by level and at each level, every node uses the identical feature and splitting value (Figure 4).

What are the practical implications of different tree growing strategies? LightGBM’s leaf-wise method has faster training time, as it takes fewer splits to achieve the same loss reduction when compared to the level-wise one. But a downside is that LightGBM is more prone to overfitting with smaller data sets. XGBoost and CatBoost are usually more robust against overfitting because of their level-wise method. Furthermore, CatBoost’s symmetric tree structure acts as an extra regularisation mechanism, thereby reducing tree complexity.

The second key difference is that both LightGBM and CatBoost natively handle categorical variables, but XGBoost requires encoding of categorical variables into numerical format before model training. For each of LightGBM’s splits during model training, it calculates the gradient statistics for each category and uses them to sort categories. Gradient statistics are indicative of residual errors. Then it finds the optimal two-way grouping using an algorithm with polynomial time complexity (Ke et al., Reference Ke, Meng, Finley, Wang, Chen, Ma, Ye and Liu2017).

CatBoost employs random permutations of the training set to produce various artificial timelines. When applying target encoding on categorical features (3.9.2 Categorical features) for a decision tree, the encoded value for every data point is calculated from those that have appeared earlier on a randomly-selected timeline (Prokhorenkova et al., Reference Prokhorenkova, Gusev, Vorobev, Dorogush and Gulin2018). This remediates the problem of data leakage in target encoding. However, the current version of CatBoost’s target encoding does not accommodate sample weights (Yandex, 2025). This may cause problems in situations when data points have different weights, for instance, in mortality modelling.

Table 6 compares other key GBM functionalities by the following characteristics:

• • Offset: incorporating a local bias for each data point, which is essential for residual risk modelling.

• • Interaction constraints: restricting feature interactions for linear decomposition of model predictions, useful when separating the effect of control variables from that of genuine features.

• • Monotonic constraints: enforcing increasing or decreasing relationships between features and predictions, important for conforming model behaviours to domain knowledge or regulatory constraints.

• • Incremental training: continuing training from existing models, with a use case being continuously updating models using new data, instead of training models from scratch.

• • Survival modelling: allowing for building survival models such as Cox Proportional Hazards (Cox, Reference Cox1972) or Accelerated Failure Time (Wei, Reference Wei1992).

Table 6.

Comparison of functionalities between XGBoost, LightGBM and CatBoost

Despite GBMs being able to model more complex relationships compared to linear models, H&C actuaries often prefer linear models for their complete transparency. Several adaptations of GBMs have been developed to improve their explainability. For instance, Explainable Boosting Machines (EBMs) use gradient boosting with shallow decision trees to enforce an additive structure with pairwise interactions between features and model predictions : $$g\left( {E\left[ y \right]} \right) = \;{\beta _0} + \sum {f_i}\left( {{x_i}} \right) + \;\sum {f_{ij}}\left( {{x_i},\;{x_j}} \right)$$ , where $$E\left[ y \right]$$ is the expected value of the target variable, $$g$$ is the link function and $${x_i}$$ is a feature (Lou et al., Reference Lou, Caruana, Gehrke and Hooker2013).

Nevertheless, EBM offers a narrower set of functionalities compared with the standard GBM packages. For instance, it does not allow for training survival models, nor does it support custom loss functions that allow users to tackle a broader range of modelling problems (InterpretML, 2025). Furthermore, its bespoke boosting algorithm that round-robin cycles through features with a very low learning rate and its lack of support for GPU training mean that model training will take materially longer than with the standard packages.

Perhaps surprisingly, enforcing an explainable, additive structure merely requires the ability to constrain feature interactions and continue training from existing models. The imposition of interaction constraints ensures that the model learns just individual effects and pairwise interactions. The ability to continue model training enables pairwise effects to be learnt only after the individual effects are learnt in previous models. For these reasons, both XGBoost and LightGBM can train explainable GBMs akin to EBMs, while coming with the broader functionalities that may be useful for tackling domain-specific problems.

3.10.4. Neural network

Neural networks (NNs), at their core, are composed of interconnected units called neurons, with the architecture defining the structure of the neurons and the connection weights as the model parameters. Viewing through this lens, NNs can be seen as an extension of GLMs, which have direct connections from input features to the output and do not contain any hidden layers. NNs are inspired by the biological process by which brains strengthen synaptic connections through learning from experience.

The Perceptron (Rosenblatt, Reference Rosenblatt1958), the first trainable neural network, had a single-layer architecture and used a custom learning algorithm for updating its weights. With just one layer, the model struggled to learn complex patterns from data. The advent of the backpropagation algorithm (Rumelhart et al., Reference Rumelhart, Hinton and Williams1986) made multi-layer neural network training viable, leading to breakthroughs in Convolutional Neural Networks (CNNs) for computer vision (LeCun et al., Reference LeCun, Bottou, Bengio and Haffner1998a) and Recurrent Neural Networks (RNNs) for NLP (Mikolov et al., Reference Mikolov, Karafiát, Burget, Cernocký and Khudanpur2010). Further innovations in the form of the attention mechanism and transformer architecture (Vaswani et al., Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017) provided the foundation for building multi-modal AI chatbots.

Modelling of tabular data presents different challenges compared to unstructured data. For such work, practitioners usually start by using a standard feedforward architecture. This architecture connects each neuron in a layer to all the neurons in the next layer. This dense connectivity may lead to the model learning spurious patterns, resulting in overfitting especially when using small data sets.

To reduce model complexity of the feedforward models, simpler architectures can be developed by selectively dropping connections. One approach is a GLM-like architecture, where the first layer captures individual feature effects and the second layer automatically learns any residual pairwise interaction effects (Figure 5). This approach mirrors that of Explainable Boosting Machine discussed in Section 3.10.1.3. L1 and L2 regularisation (Section 3.10.1.1) can also be incorporated into the loss function to further mitigate overfitting.

Figure 5.

A comparison of a fully connected, feedforward NN (left) and a GLM-like architecture with sparser connectivity (right), both using two hidden layers.

Beyond this kind of general architectural adaptations, actuaries have also been developing new NN architectures with actuarial applications in mind. One such example is LocalGLMnet (Richman & Wüthrich, Reference Richman and Wüthrich2023), which learns context-dependent coefficients for each feature by using fully connected layers. Unlike classical GLMs, in this approach each coefficient can vary depending on the other feature values, essentially learning interaction effects that are more easily interpreted by model users.

Another method is the Combined Actuarial Neural Network (CANN) (Schelldorfer & Wuthrich, 2019), which in effect trains a neural network with GLM predictions as local biases. This combines the interpretability of GLMs with the flexibility of NNs. More recently, Credibility Transformer (Richman et al., Reference Richman, Scognamiglio and Wüthrich2025) adapts the transformer architecture for claim frequency predictions by incorporating credibility theory (Section 3.6.1) into the architecture.

3.10.5. Gradient Boosting Machines vs Neural Networks

There has also been similar development in the general data science community to create NNs optimised for tabular data. Recent architectures developed for tabular data include TabNet (Arik & Pfister, Reference Arik and Pfister2021), DNF-Net (Katzir et al., Reference Katzir, Elidan and El-Yaniv2021) and Neural Oblivious Decision Ensembles (NODE) (Popov et al., Reference Popov, Morozov and Babenko2020). However, these models struggled to generalise beyond their original data sets used in their respective papers, with XGBoost outperforming them on 8 of 11 diverse data sets (Shwartz-Ziv & Armon, Reference Shwartz-Ziv and Armon2022).

A larger-scale study across 176 data sets produced more nuanced findings, where the NNs and GBMs were more evenly matched overall. Digging deeper, though, GBMs consistently outperformed NNs on data sets where features are irregular (e.g. heavy-tailed, skewed) or have high variance (McElfresh et al., Reference McElfresh, Khandagale, Valverde, Prasad, Feuer, Hegde, Ramakrishnan, Goldblum and White2023). GBMs are also generally easier to train, less sensitive to hyperparameter choices, and requiring less feature engineering (e.g. scaling numerical variables, imputing missing values) than NNs. Nonetheless, modern deep learning packages (e.g. Keras, Torch) enable easier training of non-standard models, such as zero-inflated models that handle excessive zeros in count data. Whether these strengths will make NNs a standard tool for actuarial applications remains to be seen.

3.10.6. Target Variable, Weights and Offset

The target variable (or dependent variable) should be meaningful and relevant to the research question. Target variable classes can be binary (e.g. claim or survival for a single subject), integers (e.g. claim count for a group of subjects, number of hospital visits) or continuous (e.g. blood pressure, claim amount). It is advised to visualise the target variable to obtain insights on the probability distribution and whether any transformation (e.g. log transformation, scaling) is required.

Weights can be used to address a biased sample by increasing the weight of under-sampled groups, or class imbalance (Section 3.12). Offsets are often used in regression models to account for exposure (e.g. time, population at risk, or an expected baseline or prior), allowing the model to estimate the rate or deviation from that baseline, rather than raw counts. This is very common in H&C data analysis, where outcomes often manifest over time and therefore have a direct relationship with observation time. For example, in a cohort study, the offset can be length of follow-up or expected number of claims (i.e. baseline). When the offset is time, the model will fit incidence rates. In the insurance space in particular, the offset is often an expected number of claims – if the target variable is set as the actual claims, the model will fit a set of adjustments to the expected. The case-control and cross-sectional study designs do not require a time component, although the offset can still be used to represent a prior or baseline risk.

3.10.7. Probability Distribution of Target Variable

Setting a correct probability distribution for the target variable (e.g. claims) reflects the underlying structure of the data and enables accurate predictions. The choice of distribution directly determines the appropriate loss function for model training. Commonly used probability distributions are logistic for binary outcomes (e.g. claims, lapses), Poisson for count outcomes (e.g. aggregated claims and lapses) and Gamma for continuous, positive valued outcomes (e.g. claim amounts).

Probability distributions are subject to underlying assumptions, for example a Poisson distribution assumes the mean equals the variance and the probability of zeros should equal $${e^{ - \lambda }}$$ , where $$\lambda$$ is the Poisson mean. In practice, excessive zeros are common in insurance data when the target variable has a very low incidence rate, e.g. mortality claims. If Poisson assumptions are violated, alternative distributions such as Negative Binomial or Zero Inflated Poisson can be a better fit (Winkelmann, Reference Winkelmann2008)