2.1.1 Boards

The Board of a company is ultimately responsible for approving the firm’s Internal Model for use. It needs to ensure that on an ongoing basis the design and operations of the Internal Model are fit for purpose; that the model appropriately reflects the company’s risk profile; the model meets the relevant tests and standards and that the output from the model is credible for use in managing the business and for regulatory purposes.

2.1.2 Supervisory authorities

The supervisory authorities will wish to be provided with evidence which demonstrates that the model meets all the relevant tests and standards of Solvency II in order that they are able to approve the model for use in calculating regulatory capital requirements.

2.1.3 Risk committees, senior management

The Board may use the output from reviews by its risk committees to inform its final decision on whether to accept the model or to require changes to it. Senior management will be users of the model in the day to day management of risk in the business and have a role in ensuring it is fit for purpose. Members of senior management will also need to have a detailed understanding of the components of the Internal Model used in their own areas of the business. Senior management may establish technical committees whose membership includes executives from different parts of the business to ensure that recommendations made to the Board on the methodology and assumptions used in the model take appropriate account of business needs in addition to being technically sound.

2.1.4 Risk management function

Under Solvency II, the risk management function is responsible for putting in place an effective risk management system to identify, measure, monitor, manage and report on the risks to which a company is exposed and their interdependencies.

Where an Internal Model has been approved by the supervisory authorities for the calculation of the SCR, the Solvency II Directive states that the risk management function is responsible for the design and implementation of that Internal Model as well as for its testing and validation. This includes the documentation of the Internal Model together with any subsequent changes made to it. The risk management function is also required to analyse the performance of the model and produce corresponding reports. These reports include informing the Board about the performance of the model, areas that need improvement and providing updates on actions aimed at improvement of previously identified weaknesses.

The risk management function is also responsible for the policies relating to the governance of the Internal Model, including the policy for changes to the Internal Model and the framework for validation of the Internal Model.

In practice, the risk management function may delegate some of the day to day activities to the actuarial function, subject to oversight by the risk management function. For example, due to the requirement of Solvency II for independent validation (see section 2.1.6), the design and implementation of the Internal Model together with responsibility for maintaining the related documentation is often delegated to the actuarial function, with oversight provided by the risk management function.

The risk management function is responsible for the developing of proposals for the validation framework for review, challenge and, ultimately, approval by the Board and for the production of regular validation reports to the Board. The resulting validation process will often include a review of the choice of copula and its parameterisation by individuals independent of the development of those proposals. It will also typically include (i) an assessment of the adequacy of the fit of the proxy model by the risk management function or (ii) the definition of a set of tests to be performed by the actuarial function to assess the fit of the proxy model and the criteria for any subsequent adjustments to model outputs with a review of the outcome by the risk management function.

2.1.5 Actuarial function

The actuarial function is often responsible for the design, maintenance, testing and day to day operation of the Internal Model under oversight of the risk management function. The actuarial function will have an interest not only in ensuring the technical soundness of the model and its compliance with the company’s own policies and the relevant regulatory tests and standards, but also that it is appropriate for use in the business. This means that the model should not only be suitable for the calculation of regulatory capital requirements, but also that it does not contain unnecessary areas of prudence which could lead to inappropriate decisions or result in unnecessary constraints on the business. The actuarial function is therefore likely to establish its own “first line” system of review which may include peer review by other technical specialists, technical review forums including other finance experts on areas such as asset liability management, tax or IFRS reporting, and final review by the Chief Actuary. The actuarial function will therefore be highly interested in both the technical soundness of the model as well as ensuring that the outputs it produces provide a realistic measure of the risks.

The actuarial function may also be responsible for maintaining the documentation of the Internal Model, subject to review and approval by the risk management function. This could include preparing papers recommending methodology and assumptions for approval by the Board, together with papers seeking approval from the Board for the results of the Internal Model, including the SCR. These papers should include an assessment of the strengths and limitations of the Internal Model, a description of the significant expert judgements and sensitivities to valid alternative assumptions. In particular, documentation provided to the Board should draw out key judgements relating to the choice of copula, its parameters, how account has been taken of tail dependence, the rationale for those judgements, the associated limitations and sensitivities to valid alternative judgements. The documentation should also draw the attention of the Board to limitations of the proxy model including fitting errors and, where applicable, how these limitations have been allowed for through adjustments to the proxy model result together with the rationale for those adjustments. These judgements should be communicated in a way which is accessible and engaging and which identifies the judgements where the Board can significantly influence the outputs of the model by choosing alternative assumptions.

2.1.6 Independent validation

The Solvency II regulations require regular validation of the Internal Model, including its specification, performance and comparison of its results against actual experience, through a validation process which is independent of those responsible for the development and operation of the model. Responsibility for the validation of the model lies with the risk management function. As indicated in section 2.1.4, the requirement for independence of validation process from those responsible from the development and operation of the Internal Model often means that the latter responsibilities are delegated to the actuarial function under oversight of the risk management function.

Personnel involved in the validation process will be interested in ensuring that the model meets all the relevant tests and standards and is suitable for use in managing the business. Personnel charged with the validation process must regularly report on the outcome of their reviews to the Board. They will therefore wish to have a good understanding of the mathematical basis for the model, detailed evidence which demonstrates that the model it meets the tests and standards of Solvency II, and that the outputs from the model provide a reasonable basis for the measurement and management of risk. They will also be interested in understanding the limitations of the model and circumstances under which it may not be effective, whether an appropriate range of alternative methods have been considered and how the limitations are mitigated.

2.1.7 Internal audit

The internal audit function is responsible for the evaluation of the adequacy and effectiveness of the company’s internal control system, including whether the actuarial function and risk management function have properly performed their respective roles. The internal audit function may therefore carry out its own testing in order to provide assurance to the Board. This could include aspects such as a review of the effectiveness of the processes and controls designed to ensure the Internal Model meets the required tests and standards, the processes around expert judgement (e.g. the selection of correlation assumptions) and whether the process for calibration and adjustment of the output from a proxy model is operated in accordance with the approved specification.

2.1.8 External advisors

Some firms may seek additional assurance from external advisors regarding the design of the model or the underlying assumptions, in particular in specialist areas where the firm feels it does not have sufficient expertise internally or where it wishes to obtain additional insight into market practice.

2.1.9 External auditors

Where the SCR is calculated using an approved Internal Model, according to PRA Consultation Paper CP 43/15 “Solvency II: external audit of the public disclosure requirement” (2015b) (the consultation on which had not been concluded at the time of writing this paper), the PRA does not intend to require the SCR to be subject to external audit. This avoids duplication of the independent validation and the PRA’s own Internal Model approval process. It is for the Boards of such firms to determine the extent of involvement of external auditors in review of the SCR. For example, the Board of some firms may determine that no further external assurance is required. Alternatively, a Board may request external auditors to perform one of a range of possible assurance exercises: (i) review and comment on specific aspects of the SCR calculation; (ii) provide a limited assurance opinion on whether specific items of the SCR have been calculated in accordance with a basis of preparation defined by the firm; or (iii) provide a reasonable assurance opinion on whether the full SCR calculation has been performed in accordance with a basis of preparation defined by the firm. In each case, the basis of preparation would be the specification of the Internal Model approved by the college of supervisors.

2.2.1 Board members

Where the Internal Model is used for calculating the SCR under Solvency II, the Board are collectively responsible for ensuring that the Internal Model is fit for purpose and that it meets all the relevant tests and standards. They will therefore wish to make sure that it is appropriate for use in the management of risk in the business as well as for the production of regulatory capital requirements.

This does not mean that members of the Board have to be experts in the mathematics and statistics underlying the capital models. Rather they will need to understand at a high level why a particular model was selected, what that model does, its key features, its strengths and limitations, the significant judgements involved, the related uncertainties and the impacts of reasonable alternative models and assumptions so they can exercise review and challenge where appropriate.

It will therefore be important when explaining recommendations to the Board to focus on the most significant areas of judgement and to avoid technical jargon. Graphical techniques such as scatter plots, charts and histograms provide effective tools to explain and motivate choices in a compact way. Simple worked examples may also be helpful to illustrate concepts. Boards must ensure that they have sufficient understanding of the model, the underlying judgements, their limitations and the sensitivities of the outputs to valid alternative in order to form a view on whether the model is fit for purpose. The Board will also need to be involved in the design of the independent validation process and approve it for use to ensure that proposals have received an appropriate level of technical challenge.

In addition to submitting proposals for approval by the Board at a formal meeting, it will be appropriate to ensure that the Board is well informed in advance of the meeting. A series of educational events, exploring different aspects of the model, may therefore be useful to allow members to build up an understanding and have the opportunity to ask questions prior to making a decision.

Members of Boards are likely to have diverse backgrounds and experience, as well as differing levels of interest in the components of the model. It may therefore be appropriate to offer one-to-one sessions with individual members to provide an opportunity for more detailed exploration of specific areas of interest.

The membership of a Board also changes over time. Firms may therefore wish to maintain an appropriate suite of educational material which can be used to support new directors or as the basis of regular Board “refreshes”.

For the purposes of evidencing effective governance by the Board, it will be appropriate to keep a record of the review and challenge exercised by the Board and track progress against any actions. Firms may also wish to maintain a log of the educational support provided to members of the Board.

2.2.2 Supervisory authorities

Regulators will wish to ensure that the Internal Model meets all the relevant tests and standards of Solvency II prior to approval for use to calculate the SCR. They will therefore expect to be provided with documentation which demonstrates that the model meets the requirements of Articles 120–126 of the Solvency II Framework Directive (2009/138/EC) together with the related requirements of the Delegated Regulations and EIOPA Guidelines. It may therefore be useful to use a checklist or standard documentation template to verify that the documentation to be provided does evidence compliance with all the relevant standards.

The supervisory authorities will want to ensure that the undertaking has a thorough understanding of the techniques used (including the mathematical basis), their limitations and what measures the undertaking has taken to mitigate those limitations. This will include evidence of having taken account of relevant data and the application of appropriate validation tests, including statistical testing, where relevant. The supervisory authorities also have teams of technical specialists whose expertise can be drawn upon to review submissions from firms. Therefore, undertakings may expect to have to present detailed technical documentation to support their methodology.

Regulators will also expect undertakings to have identified all the choices made, the potential impact of alternatives and understand why an undertaking has chosen to go down one particular route rather than another. This includes the identification of the most significant assumptions (e.g. correlations) and a quantification of the impact on the SCR of adopting plausible alternative assumptions.

The selection of a dependency structure and the approach to calibration and adjustment of a proxy model necessarily involve expert judgement. Undertakings should be able to demonstrate that there is a robust and systematic process in place for the selection of those assumptions and their validation, including the adjustment of any assumptions or the outputs to allow for limitations of the model (e.g. the lack of tail dependence in a Gaussian copula; fitting error in a proxy model).

The supervisory authorities will also expect users of the outputs of the model to be aware of any significant limitations of the model so that appropriate account of these limitations can be taken when making decisions informed by the model. They will therefore expect documentation provided to the users to highlight such limitations.

The PRA published some principles on creating good-quality documentation in December 2013 (PRA, 2013). The PRA has also developed a Quantitative Framework which it has used in its assessment of Internal Model applications. It has released some details of the Quantitative Indicators which form part of that framework in two executive director updates dated 9 March 2015 (PRA, 2015a) and 15 January 2016 (PRA, 2016). These updates include an outline of some of the factors considered in the PRA’s assessment of dependency structures. Actuaries involved in the development of internal models may wish to refer to these documents to help understand the PRA’s expectations.

3.5.1 Data inspection

A first step in illustrating dependency is through plots of time series and scatter plots. These relatively simple charts are often the simplest and most effective tool in demonstrating evidence of association to stakeholders and motivating subsequent choices.

Scatter plots of the data can assist in:

• ∙ The identification of the presence of a relationship.

• ∙ The nature of that relationship (e.g. the sign and broad magnitude of any correlation, the extent of any symmetry or lack of it, and any clustering of extreme values that may indicate the presence of tail dependence).

• ∙ Identification of any data points, which could be outliers.

Charts of both the raw observations and pseudo-observations are useful. The pseudo-observations (defined in Appendix A.2) use a non-parametric transformation of ranks to filter out the marginal distributions and can be compared with scatter charts of standard copulas.

As noted in Shaw et al. (Reference Shaw, Smith and Spivak2011), by excluding the time dimension, scatter plots mask temporal effects which may be present in the data and which one may wish to take in account when selecting assumptions (e.g. trends or a change in regime).

Figure 2 shows a time series plot of equity returns versus increases in credit spreads. Figure 3 shows scatter plots of pairs of increases in value of our three risk factors and the corresponding pseudo-observations.

Figure 2 Monthly equity returns and increases in credit spreads over period 31 December 1996 to 31 December 2014

Figure 3 Equity returns and credit spreads raw and pseudo-observations

Figure 2 shows no obvious trend in the relationship between equity returns and credit spreads. There is an obvious anti-symmetry in the “peaks” with increases in credit spreads frequently being mirrored by negative equity returns. The strong negative correlation is also apparent in Figure 3. The chart of pseudo-observations for EQ/CR shows some clustering in the upper left and lower right tails along the “−45% ray” – that is, extreme falls in equity values show a greater tendency to be accompanied by extreme increases in credit spreads (and vice versa). However, it also shows some clustering along the other diagonal (the “+45% ray”) giving rise to “star” shape. This behaviour is typical of a Student’s t copula with a low degrees of freedom parameter and indicates the presence of “arachnitude” – see section 3.10.

From the charts of pseudo-observations, at least visually, the assumption of an elliptic copula for each pair does not appear unreasonable.

3.5.2 Parametric fitting techniques

There are various statistical techniques for estimation of the copula parameters which extend the MoM or maximum likelihood estimate (MLE) method that are familiar in fitting models for one-dimensional random variables. We describe these techniques briefly here. Readers are referred to standard texts, for example, section 7.5 of McNeil et al. (Reference McNeil, Frey and Embrechts2015), for further details.

3.5.3 MoMs type approaches

These are based on (first order) rank invariants of the copula such as Spearman’s rank correlation or Kendall’s τ statistic.

For the d-dimensional Gaussian copula, one can calculate Spearman’s rank correlation or Kendall’s τ for the sample data and invert to solve for the correlation parameter using the formulae below:

(Equation (1)) $$\rho \,\, {\equals}\, \,2\sin \left( {{{\pi \rho _{S} } \over 6}} \right)$$

(Equation (2)) $$\rho \,\, {\equals}\, \,\sin \left( {{{\pi \rho _{\tau } } \over 2}} \right)$$

where, for a given pair of risk factors, ρ _S is the Spearman’s rank correlation, ρ _τ the Kendall’s τ statistic and ρ the corresponding parameter of the correlation matrix underlying the copula. The relationship in equation (1) is precise only for the Gaussian copula, although in practical situations, it does not appear to lead to significantly different conclusions.

The approach based on the relationship involving Kendall’s τ statistic described by equation (2) (the “inverse Kendall’s τ” technique) holds more generally for any elliptic copula such as the Student’s t copula. However, pairwise inversion of the Kendall’s τ statistic using equation (2) may not produce a PSD copula correlation matrix. It may therefore be necessary to adjust the resulting matrix using techniques such as those described in section 3.14 to obtain a PSD correlation matrix. This technique also results only in values for the correlation matrix. Other techniques such as maximum likelihood estimation or use of higher order rank statistics such as those described in section 3.10 must be used to estimate the degrees of freedom parameter.

3.5.4 Maximum likelihood approaches

There are two slightly different approaches to the estimation of copula parameters using maximum likelihood techniques:

1. (a) Inference from margins (IFM) approach – see Joe (Reference Joe2015)

   This approach assumes parametric models for each of the distributions of changes in individual risk factors as well as for the copula. The usual maximum likelihood approach, given a set of sample data for changes in the risk factors, would then be to express the likelihood (or log likelihood) of the joint distribution as a function in the parameters of the copula and all the marginal distributions. This will generally result in a high-dimensional space in which to seek a solution.

   The IFM approach splits this optimisation process into two separate steps:

   • ∙ First, the parameters of each of the one-dimensional marginal distributions of changes in risk factors are estimated using maximum likelihood.

   • ∙ The fitted parameters of the marginal distributions are then kept fixed so that the (log) likelihood function is then expressed in terms of the copula parameters only. The values of these parameters are then chosen to maximise the (log) likelihood.

   The values of the copula parameters therefore depend on the models and parameters chosen for the individual risk factor distributions.

2. (b) MPL – see Genest & Rivest (Reference Genest, Rémillard and Beaudoin1993) and McNeil et al. (Reference McNeil, Frey and Embrechts2015)

This method avoids making assumptions about the marginal distributions by using non-parametric techniques to estimate their distributions. The sample data are replaced by the corresponding “pseudo-observations” – see Appendix A.2 for definitions. The corresponding pseudo-observations are then used as inputs to the probability density function of the copula when forming the (log) likelihood function. The resulting likelihood function then depends only on the parameters of the copula and, unlike the IFM approach, not on the assumed models and parameters of the marginal distributions. The copula parameters can then be selected using optimisation techniques.

3.5.5 Worked example

We have fitted correlation parameters (“ρ”) and degrees of freedom parameters (“Nu”) of the Gaussian and Student’s t copulas to our data set using:

1. (a) the inverse Kendall’s τ technique to estimate ρ (with MLE to estimate Nu) using the “QRM” package in R;

2. (b) the MPL method using the “copula” package of R.

In both cases, we have fitted to pairs of risk factors as well as the triple. Tables 1 to 4 show the results.

Table 1 Bivariate

Table 2 Trivariate

Table 3 Bivariate – Maximum Pseudo-Likelihood

Table 4 Trivariate – Maximum Pseudo-Likelihood

Inverse Kendall’s τ

MPL

We note the following:

• ∙ Fitting a Gaussian copula using either technique results in a correlation parameter which is appropriate to the full distribution. The resulting value of the correlation is identical to that fitted to a Student’s t model if the inverse Kendall’s τ method is used. It does not differ significantly from the corresponding parameter for a Student’s τ copula if the MPL approach is used, even where the degrees of freedom parameter is low. This suggests that, if using a Gaussian copula, further adjustments to the correlations parameter may be required to allow for the effect of tail dependence in the extreme tail – see sections 3.8 and 3.9.

• ∙ For example, for EQ/CR, both the MPL and MoM fit for the bivariate t copula produce a degrees of freedom parameter of <3. However, the correlation parameter of the bivariate Gaussian and bivariate Student’s t fitted using the MPL technique only differ by <3 percentage points (−48.8% compared to −46.5%, respectively).

• ∙ Where a Student’s t copula is fitted for each pair of risk factors separately, the degrees of freedom parameters for the different pairs vary significantly. The degrees of freedom parameter in the trivariate case (4.51) is in some sense an “average” of the three bivariate values (2.60, 9.40 and 6.08).

3.6.1 Different time periods

In practice, the correlation between increases in two risk factors varies over time. Judgement is therefore required in selecting both the period of time on which the estimate is based and any allowance made for any uncertainty in the estimate. Note that the latter is a margin for prudence in the estimate of the copula parameter. In the case of a Gaussian copula, this differs conceptually from any further allowance which may be made to adjust for the absence of tail dependence, although the outcome may be similar. We discuss adjustments to the parameters of a Gaussian copula for tail dependence in sections 3.8 and 3.9.

Charts of correlations over different time periods may be helpful in illustrating the resulting uncertainty to stakeholders in a manner which is compact and amenable to explanation, for example, by pointing out the consequences of certain extreme market events. Such charts may also help identify any potential trends in correlations which the undertaking may wish to take into account when selecting assumptions.

For example, one might produce a chart showing rank correlations over different windows of time, such as:

1. (a) From a varying start date to a fixed end date (e.g. the end of the period for which data are available).

2. (b) From a fixed start date (e.g. the start of the period for which data are available) to a varying end date.

3. (c) Over a window of fixed length moving through the data period.

4. (d) Some other set of time periods chosen to test differences in behaviour.

Under (c), the length of the window could be selected using judgement as being sufficient to form a view on “short-term” correlations and give an idea of how correlations could vary between benign and stressed conditions. The choice of window is a compromise between the length of the window and the uncertainty in the resulting estimates. A longer window will result in a lower sample error but on the other hand may not pick up short-term behaviour. Modellers who use this approach may wish to assess the effect of using different window lengths, particularly where it is used to inform material assumptions.

The charts produced may also be enriched by superimposing information about confidence intervals which may be derived using techniques such as those described in section 3.6.2.

3.6.2 Confidence intervals

It may be useful to illustrate the uncertainty in-sample estimates using confidence intervals. There are a number of techniques available, for example:

1. (i) Fisher Z-transform. This uses asymptotic properties of the distribution of transformed data to provide analytic formulae for the upper and lower bounds of a confidence interval. There are various versions of the formulae, which are intended to adjust the result to allow for the finite sample size.

2. (ii) Bootstrapping. A large number of synthetic data sets is generated by re-sampling the original data with replacement. The rank correlation for each of the synthetic data sets is then calculated. This process generates a large number of simulated values of the rank correlation from which appropriate percentiles may be drawn to determine the confidence interval.

The paper by Ruscio (Reference Ruscio2008) provides a useful survey of these approaches. Functions provided in some statistical packages such as R also produce confidence intervals.

3.6.3 Graphical tools

Figure 4 is an example of a chart of type (a) described in section 3.6.1 for the Spearman’s rank correlation of our EQ/CR data set from a varying start date to a fixed end date of 31 December 2014. We have superimposed 95% confidence intervals produced using both the Fisher Z-transform (described by formula (3) of Ruscio, Reference Ruscio2008) and bootstrapping techniques (with 1,000 simulations). Visually, in this case, the confidence intervals produced by the different techniques are almost indistinguishable.

Figure 4 Equity returns and credit spreads Spearman’s rank correlation – chart of type (a); CI=confidence interval

Note that this approach provides information only about the rank correlation – a scalar statistic which is “global” – rather than relevant to a specific area of the joint distribution such as the tail. It is therefore more useful in informing one’s best estimate view of a correlation. If it is considered appropriate to include a margin for uncertainty in the correlation, one potential approach would be to use the confidence intervals as a guide to select a higher or lower value for the correlation, taking into account the exposures in the “biting scenario”. (The “biting scenario” is a scenario which, in some sense, represents the average simulated scenario which gives rise to losses equal in magnitude to the SCR. For example, some undertakings produce such a scenario by applying a smoothing process to simulated scenarios, the ranks of whose corresponding losses lie in a “window” around the SCR.)

Inspection of rolling short-term correlations such as using charts of type (c) may be useful to inform views of correlations in “stressed circumstances” and of any further allowance for tail dependence which may be appropriate. However, we note that the copula used in the calculation of the SCR (and the tail dependence embedded within it) is a static quantity. The change in correlation over time is conceptually different to tail dependence.

Figure 5 illustrates correlations for a rolling 24 month window for our EQ/CR data set. It shows the correlation reaching almost −75% for periods beginning in 2009 during the financial crisis.

Figure 5 Equity returns and credit spreads – correlations for rolling 24 month periods, start dates 31 December 1996 to 31 December 2012

One crude method of allowing for tail dependence would be to select an assumption based on confidence intervals or “stressed correlations” informed by charts such as Figures 4 or 5. However, as noted, these are conceptually different from tail dependence so such an approach would have to be carefully justified. Alternative approaches are discussed in sections 3.8, 3.9 and 3.10.

3.6.4 Rounding

To reflect the uncertainty in the chosen parameter values and the use of judgement, as well as for the practical reason of avoiding the frequent re-calibration of a large set of parameters, it is a common practice to round correlation assumptions according to a convention chosen by the undertaking (e.g. round to integer multiples of 10%). This rounding may lead to internal inconsistencies in the “raw” correlation matrix with adjustments required to make it PSD prior to use in simulation – see section 3.14.

The rounding convention is itself a choice which should be justified. A notch size which is too small may be spuriously accurate. On the other hand, a notch size which is too large may provide insufficient granularity and could lead to unnecessary prudence or unintended imprudence as well as inconsistencies in the raw matrix which require larger adjustments to produce a PSD matrix.

3.7.1 Communicating the concept of tail dependence

Figure 6 shows a graphical method for illustrating the meaning of the coefficient of lower tail dependence.

Figure 6 Coefficient of lower (finite) tail dependence

In the diagram, the events have been expressed on a quantile (or rank) scale. The coefficient of lower finite tail dependence evaluated at q, λ _L (q), is the ratio of:

1. (i) The proportion (or probability) of events in the square ABCD; to

2. (ii) The proportion (or probability) of events in the rectangle AEFD

The probability of events occurring in the rectangle AEFD is (1−q), by definition.

As the events become more extreme (i.e. q increases), the square ABCD and the rectangle AEFD shrink. The coefficient of tail dependence is the limiting value of the ratio of the proportion of events that occur in the shrinking square to the proportion of events that occur in the shrinking rectangle.

Note that the probability of events occurring in the vertical rectangle ABGH is also by definition (1−q). The definition of λ _L (q) in equation (4) is therefore clearly symmetric in X and Y.

The coefficient of finite upper tail dependence, λ _U (q), may be illustrated in an analogous way.

It is a standard result that the coefficients of tail dependence of a Gaussian copula are zero while those of a Student’s t copula are non-zero. For example, the coefficient of tail dependence of a bivariate Student’s t copula with correlation parameter ρ and ν degrees of freedom is given by:

(Equation (7)) $$2t_{{\nu {\plus}1}} \left[ {{\minus}\sqrt {\left( {\nu {\plus}1} \right)\left( {1{\minus}\rho } \right)\,/\,\left( {1{\plus}\rho } \right)} } \right]$$

where t _ν+1 is the cumulative distribution function of a standard Student’s t distribution with (ν+1) degrees of freedom – see equation (7.38) of McNeil et al. (Reference McNeil, Frey and Embrechts2015).

3.7.2 Communicating the implications of tail dependence

So what does the presence of tail dependence mean in practice? The definition of the coefficient of tail dependence in terms of a limiting value makes the concept more difficult to explain to stakeholders. When explaining the implications of tail dependence to stakeholders, it may therefore be more useful to provide some simple quantitative indicators of what different copula models and parameters mean for the likelihood of “extreme events happening at the same time”, by illustrating the consequences in terms of joint exceedance probabilities or conditional probabilities. As we shall see in section 3.8 the use of conditional probabilities in the form of the coefficient of finite tail dependence provides a useful graphical tool to inform the selection or validation of parameters.

3.7.2.1 Joint exceedance probabilities

Tables 5 to 7 are based on tables 7.2 and 7.3 of McNeil et al. (Reference McNeil, Frey and Embrechts2015), although we have chosen parameters more typical of those commonly seen in a life insurance context. They show:

• ∙ A comparison of joint exceedance probabilities at differing percentiles produced by a bivariate Student’s t copula with those produced by a bivariate Gaussian copula for various correlation and degrees of freedom parameters.

• ∙ Each table shows the probabilities for the Gaussian copula and the factors by which those probabilities must be multiplied to obtain the corresponding probabilities for the Student’s t copula. For example, assuming a correlation parameter of 50%, the probability that both risk factors exceed their “1 in 100 year” values at the same time under a Student’s t copula model with 5 degrees of freedom is twice that under a Gaussian copula model. An event with a probability of 0.00129 or around 1 in 770 years under the Gaussian copula now has a probability of 1 in 385 years under the Student’s t copula. See the highlighted cells in the table.

• ∙ Looking at Table 5 for a correlation of 0%, the probability of both variables simultaneously exceeding their 95^th percentiles is 0.25% under a Gaussian model. However, under a Student’s t copula with 5 degrees of freedom, the probability increases by a factor of 2.24 to 0.56%. An event with probability <1 in 200 under the Gaussian model (1 in 400) has a probability >1 in 200 under the Student’s t model (around 1 in 180).

• ∙ Tables 6 and 7 provide a comparison of joint exceedance probabilities for d-tuples of risk factors to illustrate how tail dependence influences behaviour in higher dimensions. The tables show the joint exceedance probabilities for the Gaussian copula and corresponding multiples for the Student’s t copula with various degrees of freedom. For each copula, the off-diagonal correlation parameters are all equal to the value shown. We show values for 2, 5, 10 and 25 dimensions and at the 75^th and 90^th percentiles (to illustrate dependence on event severity).

Table 5 Joint Exceedance Probabilities (Gaussian and Bivariate Student’s t Copula Shown as Multiples of the Gaussian)

DOF=degrees of freedom.

Table 6 Joint Exceedance Probabilities at 75^th Percentile

DOF=degrees of freedom.

Table 7 Joint Exceedance Probabilities at 90^th Percentile

DOF=degrees of freedom.

For example, taking 90^th percentile (or “1 in 10 year” events) in each risk factor and assuming a correlation parameter of 25%, a simultaneous event in ten risk factors, each of which is at least as severe as a 1 in 10 year event, is 5.7 times more likely if they follow a Student’s t model with 5 degrees of freedom compared to a Gaussian. The equivalent multiplier for two dimensions is 1.27. See the highlighted cells in the table using 25% correlation.

The results illustrate the importance of considering tail dependence and making appropriate adjustments to the copula parameters to allow for this. As we saw in section 3.5.5, the application of standard fitting techniques can produce very similar correlation parameters for a Gaussian and Student’s t model. Yet, as shown by Tables 5 to 7, the two models can produce very different joint exceedance probabilities at extreme percentiles. When calculating the SCR or using an Internal Model to generate other outputs at extreme percentiles, it is therefore essential to consider tail dependence when selecting the parameters of the model. The selection of copula parameters necessarily involves expert judgement. We discuss some techniques for informing that judgement in the case of the Gaussian and Student’s t copula in sections 3.8, 3.9 and 3.10.

3.7.2.2 Conditional probabilities

An alternative approach to illustrating the effects of tail dependence is to show the effects on conditional probabilities (i.e. the coefficients of finite tail dependence). For example, Table 8 shows the coefficients of finite tail dependence for a bivariate Gaussian copula and bivariate Student’s t copulas with 5 and 10 degrees of freedom linking random variables X and Y with a common value of 50% for the correlation parameter. It shows that the probability of Y exceeding its 97.5^th percentile value given that X has exceeded its 97.5^th percentile value under the assumption of Student’s t copula with 5 degrees of freedom is 156% of that assuming a Gaussian model.

Table 8 Coefficients of Finite Tail Dependence (Bivariate Gaussian and Bivariate Student’s t Copulas, 5 and 10 Degrees of Freedom)

DOF=degrees of freedom.

Note that the ratios of the conditional probabilities of the Student’s t model to the Gaussian model in Table 8 are equal to the corresponding ratios of joint exceedance probabilities in Table 5. This is because in the conditional probabilities in numerator and denominator are both obtained by dividing the joint exceedance probability by the probability of Y exceeding the corresponding percentile – see equation (3).

From examining the ratios of joint exceedance probabilities in the tables, it is apparent that, for an equi-correlation matrix and all other things equal, the amplifying effect of tail dependence increases as:

• ∙ The percentile of the joint event increases (where the joint event is assumed to be a combination of equi-probable events in each of the risk factors).

• ∙ The correlation parameter reduces.

• ∙ The degrees of freedom parameter reduces.

• ∙ The number of dimensions increases.

3.7.2.3 Charts of coefficients of finite tail dependence

The implications of tail dependence for conditional probabilities can be summarised compactly using simple charts showing the value of the coefficients of upper and lower tail dependence λ _U and λ _L as a function of the percentile q. We will see in section 3.8 that, where a Gaussian model has been chosen, such charts can provide a useful tool in explaining to stakeholders the selection of correlation assumptions and adjustments made to those parameters to allow for tail dependence.

Figure 7 illustrates the behaviour of the coefficient of finite tail dependence for the bivariate Gaussian copula and bivariate Student’s t copulas with 5 and 10 degrees of freedom with correlation parameters of 25%, 50% and 75%. (The right hand plot in Figure 7 is restricted to events more extreme than the 90^th percentile.) As these copulas are radially symmetric, the charts of λ _U and λ _L are coincident.

Figure 7 Coefficients of finite tail dependence, ρ is correlation (expanded view 90^th percentile on the right)

The following is apparent from the charts:

• ∙ The correlation parameter is the principal factor that determines conditional probabilities.

• ∙ There is little difference in conditional probabilities in the body of the distribution.

• ∙ However, differences in the shape of the functions become apparent in tails. For a given correlation, the coefficient of finite tail dependence is greater for a Student’s t copula than for the Gaussian copula and increases as the degrees of freedom parameter reduces.

• ∙ The coefficients of finite tail dependence tend to zero for the Gaussian copula but to non-zero values for the Student’s t copula.

A coefficient of tail dependence of zero does not imply that extreme changes in one risk are less likely to be accompanied by extreme changes in another. In the case of the Gaussian copula, a positive correlation assumption between increases in X and increases in Y means that, on average, large increases in Y will tend to occur if a large increase in X has occurred. Recall that the conditional distribution is given by:

(Equation (8)) $$\left. Y \right|X\,\sim\,N\left( {\mu _{Y} {\plus}\rho {{\sigma _{Y} } \over {\sigma _{X} }}\left( {X{\minus}\mu _{X} } \right),\left( {1{\minus}\rho ^{2} } \right) \sigma _{Y}^{2} } \right)$$

A large value of X results in a large value of the mean of Y|X and therefore an increased tendency for Y to take large values.

A coefficient of tail dependence equal to zero also does not necessarily mean that the corresponding pair of risk factors are asymptotically “independent in the tail”, even for a Gaussian copula. See for example, section 2.2 of Malevergne & Sornette (Reference Malevergne and Sornette2003).

3.8.2 Empirical coefficients of finite tail dependence

The empirical coefficient of lower finite tail dependence functions is obtained by taking the ratio of the number of pseudo-observations in our sample that fall into the shaded square ABCD to the number of pseudo-observations that fall into the rectangle AEFD in Figure 6. For example, if we have a sample of N observations (X _i , Y _i )i=1, … , N from (X,Y) with ranks (R _i ,S _i ) then the empirical lower tail dependence function $\hat{\lambda }_{L} \left( q \right)$ is given by the ratio:

(Equation (9)) $$\hat{\lambda }_{L} \left( q \right)\, {\equals}\, {{\,\#\,\left\{ {{\raise0.7ex\hbox{&#x0024;{R_{i} }&#x0024;} \!\mathord{\left/ {\vphantom {{R_{i} } N}}\right.\kern-\nulldelimiterspace}\!\lower0.7ex\hbox{&#x0024;N&#x0024;}}\leq (1{\minus}q) {\rm \ AND\ } {\raise0.7ex\hbox{&#x0024;{S_{i} }&#x0024;} \!\mathord{\left/ {\vphantom {{S_{i} } N}}\right.\kern-\nulldelimiterspace}\!\lower0.7ex\hbox{&#x0024;N&#x0024;}}\leq (1{\minus}q)} \right\}} \over {\,\#\,\left\{ { {\raise0.7ex\hbox{&#x0024;{R_{i} }&#x0024;} \!\mathord{\left/ {\vphantom {{R_{i} } N}}\right.\kern-\nulldelimiterspace}\!\lower0.7ex\hbox{&#x0024;N&#x0024;}}\leq (1{\minus}q)} \right\}}}$$

where “#” denotes the number of observations which satisfy the condition(s) inside the curly brackets. This is simply the number of actual observations in our sample where the values of both risk factors are less than the (1−q)^th quantile divided by the number of observations of the first variable (R) which are less than the (1−q)^th quantile.

The empirical upper tail dependence function is constructed in an analogous way.

As an example, if (X _k , Y _k ) is a specific observation of (−Equity Return, Increase in Credit Spread) with rank (R _k , S _k ) then the value of the lower tail dependence function at 1−R _k /N is obtained as follows:

1. (a) Let A be the number of observations of the (X _i , Y _i ) whose ranks (R _i , S _i ) satisfy R _i ≤R _k AND S _i ≤S _k .

2. (b) Let B be the number of observations of the (X _i , Y _i ) whose ranks (R _i , S _i ) satisfy R _i ≤R _k .

3. (c) The value of the empirical lower tail dependence function is given by A/B.

In practice, we can choose the labels i so that the R _i are ordered in non-decreasing rank (i.e. assuming there are no ties, R ₁=1, R ₂=2, etc.).

The algorithm above then simplifies so that the value of the lower tail dependence function at the point 1−k/N is then obtained as follows:

1. (a) A=number of observations where i≤k AND S _i ≤k.

2. (b) B=k.

That is

(Equation (10)) $$\hat{\lambda }_{L} \left( {1{\minus}{k \over N}} \right)\, {\equals}\, {{\,\#\,\left\{ {S_{i} \leq k, i\, {\equals}\, 1,\,\ldots\!, k} \right\}} \over k}$$

3.8.3 Overview of approach

The approach proceeds as follows:

1. (i) Select a pair of risk factors and corresponding sample data.

2. (ii) For the chosen set of sample data (with N observations, say) chart the empirical coefficient of finite upper and lower tail dependence functions $\hat{\lambda }_{U} \left( q \right)$ and $\hat{\lambda }_{L} \left( q \right)$ . This data remains fixed in the following stages.

3. (iii) Choose a copula model and a parameterisation. (In practice, the approach described here is more likely to be used in parameterising a Gaussian copula, although it may prove useful in validating the parameters of a Student’s t copula.)

4. (iv) Superimpose the coefficients of finite upper and lower tail dependence λ _U (q) and λ _L (q) of the proposed model on the chart.

5. (v) Generate an envelope of confidence intervals for the values of the empirical tail dependence functions $\hat{\lambda }_{U} \left( q \right)$ and $\hat{\lambda }_{L} \left( q \right)$ assuming the dependency structure follows the proposed model. Confidence intervals can be produced using bootstrapping techniques – see Efron & Tibshirani (Reference Efron and Tibshirani1994), for example.

6. (vi) Compare the coefficients of finite tail dependence for the assumed model (and the envelope of confidence intervals around them) with the empirical coefficients of finite tail dependence derived from the sample data. If the empirical values lie outside the confidence intervals, this may indicate a poor model fit.

7. (vii) Adjust the parameterisation and/or model chosen in (ii) until an acceptable fit is found.

In adjusting the model in step (vii) above, one may specify a quantitative criterion to be met. For example, one might choose the model to target the empirical conditional probability at a chosen percentile. This percentile may be chosen based on the biting scenario.

Alternatively, taking into account the limited volume of data and the uncertainty in the empirical values, the targeting may be approximate and based on a visual inspection of the charts at percentiles close to the biting scenario. If a particular rounding convention for correlation parameters has been chosen, one might consider the effect of changing correlation parameters in discrete increments.

Use of charts (e.g. see Figure 8) is also helpful in understanding the consequences of any decision on the model for conditional probabilities within the body of the distribution as well as within in the tail. For a particular model, it may not be possible to produce simultaneously a fit which is considered satisfactory within both the body and the tail. The choice of parameterisation may therefore depend on the purposes for which it is used.

Figure 8 EQ/CR bivariate Gaussian coefficient=0.5

Note that the Gaussian and Student’s t copula are elliptic and therefore radially symmetric so that the model coefficients of finite tail dependence are equal; that is, λ _L (q)=λ _U (q) for all q.

3.8.4 Practical considerations

The approach assumes that the choice of copula has already been made and is used to adjust a best estimate view of the correlation parameters to make an allowance for tail dependence. Depending on the numbers of risk pairs involved, it may not be practical to assess each pair individually. It may therefore be desirable for practical reasons to perform the full analysis only for those pairs of risks where the sensitivity of the output to a defined change in the correlation assumption exceeds a certain threshold.

One option to further reduce the volume of detailed analysis required would be to fit a Student’s t copula to each risk pair and perform more detailed analysis if the fitted degrees of freedom parameters falls below a specified threshold.

A limitation of this approach is that it only considers conditional probabilities based on both risk factors simultaneously exceeding their q ^th percentile – that is, the percentile is identical for both risk factors. It therefore considers conditional probabilities only along a ray extending from the origin at an angle of 45° into the “north-east” and “south-west” quadrants. In order to ensure that the approach looks at the appropriate tail, it is necessary to consider the undertaking’s exposure to each risk pair (e.g. by consideration of the biting scenario). It may be necessary to adjust the data by multiplying one of the risk factors by (−1) in order to ensure that the analysis takes into account the direction of changes which are expected to bite.

3.8.5 Worked example

We illustrate the technique using our equity and credit spread (EQ/CR) data set. We assume that the undertaking’s exposure is to a fall in equity values combined with an increase in credit spreads. We have therefore multiplied equity returns by (−1) for the purposes of the analysis. As noted in section 3.5.4, the MPL fit for a bivariate Gaussian is given by ρ=0.488 and for bivariate Student’s t distribution is given by (ρ,υ)=(0.465,2.6). A potential initial candidate model may therefore be a Gaussian copula with correlation coefficient 50% (or −50% when we adjust back to our original coordinate system). This model is illustrated in Figure 8 which contains the following charts (Table 9).

Table 9 Additional Details for Figure 8

Figure 8 shows some asymmetry in the tail with the conditional probabilities associated with extreme upwards movements in credit spreads and falls in equity values somewhat higher than those corresponding to movements in the opposite direction.

The central solid line in Figure 8 tends towards zero. This is consistent with the Gaussian copula having a coefficient of tail dependence of zero.

The confidence intervals expand as events become more extreme, reflecting an increasing funnel of doubt as the volume of data in the tails decreases.

Although the circles of the lower tail do not appear inconsistent with the confidence intervals, they are at the boundary, particularly just below the 80^th percentile and exceed the boundary around the 90^th percentile. If a Gaussian copula is to be used, this may suggest use of a stronger correlation assumption if the SCR biting scenario includes percentiles of 80^th or above in credit spreads and equity risk factors.

Figure 9 is similar but with the Gaussian copula model replaced by the bivariate Student’s t copula estimated using MPL techniques.

Figure 9 EQ/CR bivariate Student’s t ρ=0.465, nu=2.6

Comparing the two charts, it is apparent that the Student’s t copula produces greater values of conditional probabilities in the tail of the distribution and the solid central line no longer converges to a value of zero, consistent with a non-zero value for the coefficient of tail dependence.

The Student’s t copula appears consistent with the sample data across a wider range of percentiles – in particular, both in the extreme tail and in the body of the distribution.

If it has been decided to use a Gaussian copula model, then one may use a correlation assumption estimated from data using one of the techniques described in sections 3.5 and 3.6 to assist in informing one’s central view of an appropriate assumption. However, if analysis suggests that the biting scenario is likely to be in the tail of the distribution, then it may be considered appropriate, given the low degrees of freedom parameter for the fitted Student’s t copula, to make an adjustment to the correlation parameter to allow for tail dependence.

One approach would be to inspect various alternative parameter values – for example, by “flicking through” a set of graphics comparing the coefficients of finite tail dependence for the assumed parameterisation and the envelope of confidence intervals around it with the empirical coefficient of finite tail dependence. Figures 10 to 12 show how the coefficient of finite tail dependence changes as the correlation assumed in the Gaussian copula is strengthened in increments of 10 percentage points. The coefficients of finite tail dependence for the copula model, together with the envelope of confidence intervals, increase, allowing a judgement to be made on a parameter value which places the model output in an appropriate place relative to the empirical values.

Figure 10 EQ/CR bivariate Gaussian coefficient=0.6

Figure 11 EQ/CR bivariate Gaussian coefficient=0.7

Figure 12 EQ/CR bivariate Gaussian coefficient=0.8

If one had a view on the percentiles underlying the SCR biting scenario, one could aim to use a correlation assumption which produced values of the coefficient of finite tail dependence which were broadly consistent with the empirical values in the neighbourhood of that percentile.

For example, if the biting scenario involved a combination of equity and credit spread stresses at around the 95^th percentile, one might judge that a correlation assumption of 60% might undershoot the conditional probabilities suggested by the data whilst a correlation assumption of 80% might overshoot. A correlation assumption of 70% might be judged a reasonable compromise.

Alternatively, if there was a desire to retain a model with explicit tail dependence, one could use such plots to inform the choice of parameters. For example, Figure 13 illustrates one potential choice of parameters for a Student’s t copula which appears to more closely match the lower tail (depicted by circles).

Figure 13 EQ/CR bivariate Student’s t ρ=0.6, nu=3

3.10.1 Background

Another approach to copula fitting – see Shaw et al. (Reference Shaw, Smith and Spivak2010) – is conceptually similar to a MoMs approach in that the parameters of the copula are selected so that certain higher order rank invariants of the model are equal to sample values derived from the data. Its main application currently is in the case where a Student’s t copula has been chosen for modelling and the correlation matrix and degrees of freedom parameter υ has to be selected, although the approach generalises to other copulas. Under the approach set out by Shaw et al. (Reference Shaw, Smith and Spivak2010), the correlation matrix and a degrees of freedom parameter υ are selected using an algorithm, which aims to produce a match between certain rank invariants of the model and their corresponding sample values. The two rank invariants are the Spearman’s rank correlation – a first-order rank statistic – and a higher order rank invariant called “arachnitude”. The latter statistic is so-named as it measures “off diagonal” dependencies that give rise to the “star” shape observed in some scatter plots of bivariate Student’s t copula or “spider’s legs” extending into the corners of the hypercube [0,1] ^d in higher dimensions.

The rank invariant “arachnitude” is defined in Smith & Sweeting (Reference Smith2011) as: ρ((2F _X (X)−1)², (2F _Y (Y)−1)²), where ρ is Pearson’s (linear) correlation.

The equivalent sample statistic is given by the formula:

(Equation (11)) $${\rm arachnitude}\, \, {\equals}\, \, {{45} \over {4N\left( {N^{2} {\minus}1} \right)\left( {N^{2} {\minus}4} \right)}}\left[ {\mathop \sum\limits_{k\, \, {\equals}\, \, 1}^N \left( {2R_{k} {\minus}N{\minus}1} \right)^{2} \left( {2S_{k} {\minus}N{\minus}1} \right)^{2} {\minus}{{N\left( {N^{2} {\minus}1} \right)^{2} } \over 9}} \right]$$

where the {R _i } and {S _j } are the ranks of the sample data {x _i } and {y _j } (Smith, 2014). Arachnitude takes values between −1 and 1 and is large when extreme high or low values of X tend to coincide with extreme high or low values of Y. It is therefore a measure of dependency along both diagonals rather than just along the 45° line as in the case of tail dependence.

A summary of the theoretical basis underlying the approach is provided in Appendix A.3. An outline of the calibration approach is provided in section 3.10.2.

3.10.2 Parameter selection algorithm

In practice, the copula we wish to parameterise will relate to a d-dimensional distribution (where d≥2) such as a set of market risks. We therefore have to solve for d(d−1)/2+1 parameters (correlation parameters for d(d−1)/2 pairs of risk factors and 1 degrees of freedom parameter). The degrees of freedom parameter is common to all pairs of risk factors, which acts as a constraint on the system of equations. This means that in practice it is not possible for all combinations of (rank correlation, arachnitude) pairs to be “reached” simultaneously by the parameters of a Student’s t copula.

The most commonly used approach to solving this constrained system of equations is as follows:

1. (A) Produce a two-dimensional scatter plot showing the sample values of (rank correlation, arachnitude) for each risk pair.

2. (B) Generate a one-dimensional family of curves (parameterised by a single degrees of freedom parameter), for which each curve describes arachnitude as a function of rank correlation, using the following algorithm:

   1. (i) Fix a grid of correlation parameters and a grid of degrees of freedom parameters for a bivariate Student’s t copula;

   2. (ii) For each degrees of freedom parameter υ:

      1. (a) Select a copula correlation parameter ρ from the grid and generate a set of simulated values from the bivariate Student’s t copula with parameters ρ and υ.

      2. (b) Calculate the rank correlation and arachnitude for this set of simulations and plot it.

      3. (c) Loop back to step (a) and select the next correlation parameter from the grid.

      4. (d) Repeat until all copula correlation parameters have been used.

   3. (iii) Draw a curve through the plotted (rank correlation, arachnitude) pairs for the given value of υ.

   4. (iv) Repeat (ii) and (iii) for each υ in the grid.

3. (C) The algorithm of step (B) generates a one-dimensional family of curves parameterised by υ showing the relationship between arachnitude and Spearman’s rank correlation. Use judgement to choose a value of υ such that an appropriate balance is obtained between the number of sample values of (rank correlation, arachnitude) that lie above the corresponding curve and the number that lie below.

4. (D) The judgement at step (C) could take into account exposures to each risk. For example, υ could be chosen so that the resulting curve was closer to those points corresponding to risks for which exposures were significant.

5. (E) This fixes a degrees of freedom parameter υ for the d-dimensional Student’s t copula, which is common to each risk pair. For each pair of risk factors, back solve for the correlation parameter of the bivariate Student’s t copula with degrees of freedom parameter υ which gives the corresponding Spearman’s rank correlation on the fitted curve (i.e. solve for the correlation parameter which produces the (rank correlation, arachnitude) pair resulting from “dropping” the sample (rank correlation, arachnitude) vertically onto the fitted curve). As the relationship of equation (1) is exact only for a Gaussian copula, the back solving involves numerical techniques (e.g. linear interpolation using the grid created in Step B(i)).

6. (F) Adjust the correlation matrix obtained in step (E) to make it PSD (e.g. using one of the techniques described in section 3.14).

The approach is illustrated in Figure 15 which is reproduced from Shaw et al. (Reference Shaw, Smith and Spivak2010). The example is based on monthly total returns for equity indices of 18 different geographies (i.e. 153 distinct pairs of risk factors) over the period 31 December 1969 to 31 December 2009 and shows curves of (rank correlation, arachnitude) for a constant correlation matrix and varying degrees of freedom parameter υ. For this particular data set, it would appear that υ=5 provides a reasonable balance between data points that lie above and below the curve. However, a different choice may be appropriate if exposures are significantly weighted towards a subset of the risks.

Figure 15 Monthly total returns for equity indices of 18 different geographies (i.e. 153 distinct pairs of risk factors) over the period 31 December 1969 to 31 December 2009 (reproduced from Shaw et al., Reference Shaw, Smith and Spivak2010 with permission)

An alternative approach to parameterisation modifies the approach described in this section by fixing the copula correlation matrix at the first (rather than final) stage in the process. This could be done, for example, by using the relationship of equation (2) involving Kendall’s τ which is exact for an elliptic copula such as a Student’s t. The resulting matrix is then made PSD, if required. With the copula correlation matrix fixed, one can then produce a family of (rank correlation, arachnitude) curves parameterised by the degrees of freedom parameter υ using the techniques described in Step B(ii) and select an appropriate value for υ as in Step (C).

3.14.1 Background

All valid correlation matrices satisfy an internal consistency condition known as “PSD”. Broadly speaking this means the correlations between all possible n-tuples of correlations are consistent. For example, if the correlation between risk pairs (X, Y) and risk pairs (Y, Z) are large and positive, one would expect the correlation between pair (X, Z) to be large and positive – see appendix A2 of Shaw et al. (Reference Shaw, Smith and Spivak2011), for more details.

Mathematically, this means that the eigenvalues of the matrix must all be non-negative. The correlation matrix approach applies a formula and will always produce a result when applied to the vector of capital requirements corresponding to each of the individual risk factors. However, if the correlation matrix is not PSD, the “sum of squares” approach will produce a zero or negative result when applied to certain capital vectors.

(Note that some copula simulation algorithms such as those based on the Cholesky decomposition as described in algorithm 6.10 of McNeil et al. (Reference McNeil, Frey and Embrechts2015) will only work if the matrix is strictly positive-definite. This is a stronger condition and requires all the eigenvalues to be strictly positive, or, equivalently that the matrix be PSD and all its columns are linearly independent. If the correlation is not strictly positive-definitive, then an alternative to the Cholesky decomposition must be used for simulating; e.g. techniques based on the decomposition of the correlation matrix in terms of a diagonal matrix of eigenvalues and an orthogonal matrix of eigenvectors.)

Correlation matrices derived from a multivariate data set where all risks are sampled over an identical period of time at coincident dates will always be PSD by construction. In the context of life assurance, this is unlikely to be the case for the following reasons:

• ∙ Where data are available, the period of time over which it is available for all the risk factors under consideration is likely to be very short. This results in the selection of assumptions on a pair-wise basis in order to maximise the use of the relevant data for each pair of risk factors.

• ∙ Use of judgement in the selection of assumptions, particularly where there is very scant or no relevant data.

• ∙ Possible inclusion of additional allowances for tail dependence.

• ∙ Any rounding convention applied when selecting correlations.

• ∙ Any further adjustments which may have been made (e.g. if the assumption suggested by the data are not considered to reflect the future relationship).

3.14.2 Adjustments

The initial candidate matrix may therefore not be PSD and so requires adjustment prior to use. There are several techniques available which vary in complexity. One of the more straightforward techniques involves elimination of negative eigenvalues. The candidate matrix is first diagonalised using standard eigen-decomposition techniques of linear algebra. Any negative or zero eigenvalues appearing along the diagonal of the diagonal matrix of eigenvalues are replaced by a small positive value chosen by judgement. The resulting matrix is then transformed back to the original coordinate system and the diagonal entries adjusted so they are all equal to one. This process is described in more detail in algorithm 7.57 of McNeil et al. (Reference McNeil, Frey and Embrechts2015).

More sophisticated techniques which seek to find the PSD correlation matrix which is in some sense “nearest” to the initial candidate matrix are also available. These techniques can be modified to apply weights to particular columns and rows, which may be useful if one has strong views that those assumptions are appropriate, or constrain an existing PSD sub-matrix to remain unchanged. See for example, Higham (Reference Higham2013).

3.14.3 Validation

Whichever technique for “PSD-ing” is used, users will need to be satisfied that the resulting matrix remains consistent with the views reflected in the selection of the “raw” candidate matrix originally approved. One may do this by inspection or introduce a process based on quantitative acceptance criteria. For example, one may inspect a histogram of percentage point increases in correlations or require that a specified proportion of changes fall within certain limits. However, a process based solely on changes to correlation assumptions does not take into account the implications for capital requirements. One may be willing to accept a larger movement in a correlation assumption between two insignificant risk factors that has a relatively small impact on capital requirements. One could therefore require that the effect of changes on capital requirements produced by a correlation matrix approach did not exceed a certain monetary limit. (The correlation matrix approach is used as the formula still produces a result even if the original matrix is not PSD.)

If the adjusted matrix does not meet the specified acceptance criteria, one may then have to: (i) seek approval to apply the adjusted matrix; (ii) use a more sophisticated adjustment technique, which may not be a practical option in the time available; or (iii) find an alternative set of adjustments by inspection, which may require a number of iterations. Failure to produce a suitable PSD matrix may indicate a more fundamental inconsistency within the original candidate matrix which requires more detailed investigation. It is therefore preferable to perform any adjustments necessary outside the production cycle, for example, as part of the calibration process.

3.15.1 Peer review

It is good practice for the proposed assumptions to be subject to review by one or more individuals with relevant expertise from around the business. Meetings could be held with the purpose to review and challenge proposals. These meetings could examine the rationale for particular assumptions, perhaps identifying relationships between changes in risk factors which had not been fully considered or whether conclusions drawn from analysis of data were reasonable or required further adjustment. The review should also consider whether the assumptions were appropriate on a prospective basis and were not unduly driven by historical data, taking into account the scarcity of data, the uncertainty in the analysis and expectations regarding the future relationship between changes in risk factors.

3.15.2 Independent review

To mitigate the risk of recommendations made by individuals or an expert judgement panel such as that described in 3.15.1 becoming biased by a desire for consensus and dismissing alternative views (“groupthink”), it is good practice for proposals to be reviewed by individuals who are independent from the formulation of the original proposals and subject to a different reporting line. The idea behind this is that the reviewers should be free of any influence from those responsible for development of the model, which, in theory, should lead to a more objective review. The Internal Model requirements of Solvency II require such an independent validation process. In some cases, the Board may wish to seek additional assurance through an external review of some or all of the assumptions, particularly those which are material.

3.15.3 Sensitivity testing

Assumption sets underlying the dependency structures used in life company internal models typically have high dimensions. For example, if there are 25 risk factors, the correlation matrix underlying a Gaussian copula will have 300 distinct parameters.Footnote ² The Towers Watson Limited (2015) Risk Calibration Survey indicates that some firms use up to 10,000 correlation parameters. In testing assumptions, it is important to focus on those which have the most significant impact on the model output. These can be identified by testing sensitivities to individual assumptions. In general, a correlation matrix approach may be adequate in ranking assumptions for this purpose rather than re-running the full copula + proxy model simulation many times.

When testing sensitivities, one should have regard to the relationships between changes in risk factors. For example, strengthening the correlation between risk factors X and Y may suggest strengthening other correlations involving X or Y for reasons of maintaining internal consistency. A sensitivity to a change in one correlation in isolation may not give a reasonable view of the total change if one were to make corresponding changes in all related assumptions.

3.15.4 Scenario analysis

This involves showing stakeholders the type of real-life situation which could give rise to losses of a magnitude similar to the SCR and asking them to form a view on whether that scenario is reasonable, given their knowledge of risk profile of the business. Any unexpected features of the scenario could indicate an inappropriate choice of parameters and trigger further investigation.

For example, one could examine scenarios which, when ranked by losses, lie in a “window” around the scenario corresponding to the SCR. Alternatively, one could average out the scenarios in the window, perhaps using kernel smoothing techniques which apply different weights to each scenario, to determine a “smoothed average” or “biting” scenario which is representative of the scenarios giving rise to losses equal to the SCR. That scenario could then be expressed in terms of the corresponding changes in risk factors which would be meaningful to stakeholders; for example, a fall in equity values by x%, a reduction in the level of interest rates of y basis points at term t; an increase in credit spreads on A rated corporate bonds of z basis points, an improvement in life expectancy of males aged 65 of w years, etc.

One could then invite stakeholders to assess whether a scenario was reasonable. For example, if a risk factor to which the business had a significant exposure featured relatively weakly in the biting scenario, this could prompt questions about whether the strength of the associations between that risk factor and others was appropriate. Alternatively, if a certain risk factor featured relatively strongly in the scenario(s), is that result reasonable given exposure to this risk factor and expectations of stakeholders regarding the strength of its relationship with other risk factors?

Conversely, one could ask stakeholders to postulate a scenario involving simultaneous changes in several risk factors, evaluate the “heavy” or proxy model on that scenario and determine its corresponding ranking or percentile in the overall distribution of losses. One could then ask stakeholders to form a view on whether the ranking of that particular scenario seemed reasonable.

In practice, it may be very challenging to ask stakeholders to assign a probability to losses under a particular scenario. However, presenting scenarios in terms of “real world” changes in risk factor values which are meaningful to the stakeholders such as falls in equity values, increases in interest rates or changes in persistency rates can make the calculations feel more “real”, help engage stakeholders in discussing the relationships between risks and provide a high level sense check on the results. Indeed, the ability of a simulation-based approach to identify a range of scenarios giving losses of magnitude comparable to the SCR is one of the main advantages of this approach compared to the correlation matrix approach.

3.15.5 Industry benchmarking

Several actuarial consultancies produce annual surveys comparing practices, models and calibrations. The validity of analyses based on survey results will necessarily be subject to limitations as they may not always compare “like with like”. For example, companies do not all adopt identical definitions of all risk factors, which may lead to different calibrations for models describing changes in similarly named risk factors or different correlation assumptions. Companies with insignificant exposures to a given risk factor may find it proportionate to adopt strong assumptions for correlations between that risk factor and other risk factors rather than spend resource on detailed analysis, whereas companies with more significant exposures may prefer to perform a more detailed analysis in order to avoid excessive prudence. The actual copula model used may also differ between one company and another. A correlation assumption adopted by a company using a Student’s t copula may not be directly comparable with the corresponding assumption of a company using a Gaussian copula, particularly if the latter has chosen to adjust the assumption to make an allowance for tail dependence.

Nevertheless, comparisons based on surveys can be useful in highlighting any assumptions which appear out of line and help focus validation effort on the rationale for those assumptions.

3.16.1 Goodness of fit tests

For example, the Table 14 provides a list of copula models with some potential tests and further references.

Table 14 Copula Models and Potential Tests

The first three tests are restricted to a specific family of copula: the Gaussian or Student’s t. They each involve “dimensional reduction” by condensing the information included in the data into values of one or two one-dimensional test statistics whose p-values are then generated assuming the null hypothesis.

The version of Mardia’s test involves first converting the empirical copula into a test observation from a multivariate Normal by applying the inverse distribution functions of a standard Normal distribution to each marginal of the copula.

The tests of Malevergne and Sornette are based around the squared Mahalanobis distance for each observation in the sample. A variety of test statistics based on the usual Kolmogorov–Smirnov or Anderson–Darling statistics may be defined and their empirical distributions derived by bootstrapping assuming the null hypothesis of the Gaussian copula holds.

The test of Kole, Koedijk and Verbeek extends that of Malevergne and Sornette to a Student’s t copula through a modification of the Mahalanobis distance.

The “blanket” tests of Genest, Rémillard and Beaudoin are so named because they can be applied to any family of copula. Instead of using an intermediate statistic to reduce the dimension, the test statistic is generated directly from the empirical copula. Again, various forms of the test statistic may be used, the most common being the Cramér von Mises statistic which is a measure of the L ² distance between the empirical copula and hypothesised copula. The distribution of the test statistic and p-values must be generated using bootstrapping techniques. Test functions are available in the “copula” package of R.

3.16.2 Model filters

These provide a method for assessing the appropriateness of using a more complex model with additional parameters. For example:

1. (i) Likelihood ratio tests for nested models.

2. (ii) Penalised likelihood functions such as Akaike information criteria (AIC), Bayesian information criteria (BIC) or other information criteria.

3.16.3 Usefulness of tests in practices

In practice, in a life insurance context, a meaningful analysis is possible only for a subset of market risks. Even then, the results of the tests are inconclusive – see for example, Makin & Stevenson (Reference Makin and Stevenson2014). It is possible that such tests may have greater power when applied to larger (and more homogeneous) sample sets such as monthly returns on equity indices for different geographies. However, for the relatively small data sets which are most often used in a life insurance context, we have found that such tests provide little useful additional insight.

We have also found when comparing two versions of the blanket tests to our sample data (the standard Cramer von Mises test and the Cramer von Mises test with the Rosenblatt transform – see Genest et al., Reference Genest and Rivest2009), that the “stronger” version of the test based on the Rosenblatt transformed resulted in non-rejection of both the Gaussian and Student’s t model whereas the standard version of the test resulted in rejection. We believe this may be due to limitations in deriving the bootstrapped distribution function of the test statistic when applied to small samples.

4.4.1 Design considerations

In the following sub-sections we identify questions that a firm designing its proxy model would consider. For each, we outline some of the factors that a firm could take into account when answering these questions.

4.4.2 Summary of proxy model design

The above considerations will influence the design of the proxy model. In summary, a firm designing its proxy model will need to decide:

• ∙ The hierarchical structure of the model, which will determine at what levels we may be able to simulate and order losses and thus calculate diversified results.

• ∙ The lowest level of the hierarchy at which proxy functions are fitted. This sets the most granular level at which results can be analysed (e.g. product groups, legal entities).

• ∙ How to reflect any constraints to the diversification that can be achieved, for example, due to ring-fencing restrictions.

• ∙ Where and how the tax impacts on profits and losses can be calculated.

• ∙ The risk factors to be used in the model, and to which products/entities these apply.

Where firms are using third party software supplied by an external vendor, there may be constraints imposed by that software, for example, a limited range of mathematical functions (as well as statistical distributions and copulas) that can be used. However, a firm will still need to ensure that the proxy model provides a materially accurate representation of their business.

Note that the above design decisions have not yet considered the methodology for fitting proxy functions. We discuss this in the next section.

4.5.1 Objectives for model fitting

We now consider the choices around how to fit the model. First, we define a possible range of objectives of the fitting process.

• ∙ Well fitted: The model must fit the firm’s heavy models well, over an appropriately wide range of scenarios. This may focus on achieving strong fit at particular points or scenarios, for example, around the 99.5^th percentile value at risk (VaR) to meet SCRs, or at other points as used by the business for risk measurement. However, it is a requirement of Solvency II that firms produce a full Probability Distribution Forecast so it will be necessary to achieve a strong fit across a range of quantiles. Testing of this accuracy objective is part of the model validation process.

• ∙ Parsimonious: The model should not be more complicated than is necessary. The fitting approach should not lead us to implement proxy functions that are more complex than is required. For example, higher order polynomial coefficients and joint terms should only be included if they materially improve the fit of the model. Testing of the parsimony objective is inherent in some mathematical fitting approaches – for example, AIC or Bayes’ information criterion (section 4.5.3). Some firms use a bespoke information criterion to select the model. If firms are using expert judgement to determine the structural form of the proxy functions, it would be more difficult to prove the satisfaction of this objective.

• ∙ Avoid over-fitting: This would normally be defined as an approach that overly focusses on fitting to the sample observations. In applications where observations include large statistically random errors, this means over-fitting to those errors (i.e. fitting to the “noise” rather than the “signal” in the data). This is less likely to be relevant in insurance applications, where the asset and liability calculations are either deterministic or based on stochastic models with sufficiently many simulations to ensure convergence. However, over-fitting in the context of proxy models would mean placing too much emphasis on achieving an accurate fit in the areas that the fitting runs are performed. Further, over-fitting can lead to results in the tails of the distribution that are not sensible due to turning points in the proxy function that occur just outside the fitting range. The range of scenarios over which the proxy model is valid should be clearly specified as a limitation and a trigger framework produced so that the curve fitting process is repeated where necessary. Graphical validation can provide a quick and instructive view of the curves that are fitted (at least up to three dimensions) so that the behaviour can be sense checked. Charts illustrating the use of graphical validation for this purpose are included in section 4.6.3.

• ∙ Practical: The fitting process also needs to lead to practical proxy functions which satisfy any constraints imposed by the simulation software used and can therefore be incorporated into the firm’s simulation model. This may preclude certain functional forms or polynomial orders.

4.5.2 Choice of proxy function form

Whilst the coefficients are a result of the proxy function fitting, the functional form itself is influenced by the approach taken to curve fitting.

For example, the functional form for a univariate proxy function may consist of the following basis functions:

• ∙ Polynomials up to a particular order (x, x ², x ³, …)

• ∙ Another function, such as the exponential function.

The general form of this proxy function, p(x), which explains balance sheet movements under changes in risk factor x, is as follows:

(Equation (12)) $$p\left( x \right)\, {\equals}\, a_{1} x{\plus}a_{2} x^{2} {\plus}a_{3} x^{3} {\plus}\,\ldots\,{\plus}be^{x} $$

Interaction terms can be allowed for similarly, through polynomials or other functions in two or more risks.

Approaches to choosing functional form can be categorised according to whether they are theory driven or data driven.

Adopting a theory-driven approach would mean using the firm’s knowledge about its assets and liabilities, and its exposure to risk factors to determine the functional form of the model, or at least to restrict the number of possible coefficients of the model through expert judgement. For example, there may be an economic theory or business intuition that the impact of a particular risk factor on a particular metric will be linear.

The other extreme is a data-driven approach, which would be to run a very large range of stresses in the heavy models and use mathematical techniques to reduce the proxy functions down to a suitable form. This approach could start with a very large number of possible terms, and the aim of the mathematical technique would be to gradually reduce the proxy functions whilst maintaining an adequate level of fit. Alternatively, it could build up the model adding one term at a time. An example of such a technique is stepwise regression, which is described in section 4.5.3.2.

A firm will need to decide the most appropriate method for selecting its proxy function forms and should justify and record why it believes the chosen approach is appropriate (as well as what alternatives were considered and why these were judged to be less appropriate). It may also be possible to perform the form selection exercise off-cycle (out of the production period), if the exposures modelled are stable over time.

4.5.3 Fitting tools

In this sub-section, we provide an overview of some methods that can be used to fit proxy functions.

4.6.2 Validation scope

The proxy model is usually part of a firm’s wider economic capital model, which may be used for regulatory purposes (e.g. Solvency II Internal Model) or for internal reporting (e.g. Economic Capital model). Multiple validation tools exist to validate these models, including back-testing, reverse stress testing and statistical testing. It is important to be clear what exactly is being tested and how the chosen validation tool design achieves the test objective.

Figure 18 shows the typical components of a capital model used in producing a Probability Distribution Forecast, and where the proxy model validation fits in. For example, under Solvency II, internal models are required to produce a suitably accurate Probability Distribution Forecast which quantifies the movement in Own Funds over a full probability distribution. The focus of this section is on statistical testing to validate the proxy model component and if relevant, how it is “rolled forward” prior to generating the Probability Distribution Forecast, that is, where the proxy model is initially calibrated prior to the valuation date, and then rolled forward to that date. The approach to roll-forward is discussed in section 4.6.10.

Figure 18 Internal model components and validation points

Hence, the scope of this section is validating the components at points labelled (1) and (2) in Figure 18 (Table 16).

Table 16 Proxy Model Validation Points

4.6.3 In-sample testing (goodness of fit and visual inspection)

An element of validation should occur during the fitting process. After regression is used to fit the curves, firms should output a number of fitting statistics to inform how well they have been able to fit the observed points. This is likely to include:

• ∙ R ² or R – a measure of overall correlation and explanatory power.

• ∙ Mean squared error (MSE), or SSE. Both measures of the absolute fitting error.

• ∙ Maximum (absolute) error.

• ∙ Number of points outside a desired range – either an absolute or percentage amount.

These kinds of statistics are commonly used as part of OOS testing, so are discussed in the next section. Firms are also likely to use visual inspection to assess the fit. Statistical testing of in-sample points has limits on its credibility, because it does not cover the risk of un-modelled points being incorrect. If the user has sufficient expertise, graphical inspection can help to identify areas where the fit is inappropriate. For example, the fitted curve may show over-fitting or turning points outside the fitting range that we may not expect.

This is illustrated in Figures 19 and 20, which shows the hypothetical equity risk proxy function from Figure 16 in section 4.2 fitted using a polynomial of degree 5 (Figure 16 showed a polynomial of degree 2, and the same fitting points have been used to fit the degree 5 polynomial). Figure 19 illustrates how over-fitting to fluctuations between fitting points can result in an unintuitive proxy function within the fitting range. While Figure 20 shows the proxy function over a slightly wider range, which shows the effect of the unexpected turning point introduced is to suggest a £4bn loss in the event of +140% equity returns.

Figure 19 Degree 5 polynomial equity proxy function illustrating over-fitting

Figure 20 Degree 5 polynomial equity proxy function illustrating turning points beyond fitting range

4.6.4 OOS testing

OOS testing is an example of an effective statistical process for validating the proxy model. Scenarios that are not used in fitting are evaluated using the heavy model and compared with the results using the proxy model. The same sorts of fitting statistics can be used as for in-sample testing. Graphical inspection of the fitted curves themselves may have limited use in OOS testing because we are generally concerned with looking at multivariate points.

OOS testing tests for the impact of the following potential errors in proxy models (Table 17).

Table 17 Types of Errors Encountered in Proxy Modelling

Proxy models inherently contain an element of approximation and, as discussed in Hursey et al. (Reference Hursey and Scott2014), errors in individual points do not themselves invalidate the model. What is more important is that there is no bias in the fitted functions, so that the model does not systematically under- or overstate the result; or fail to adequately rank risks. Proxy model errors when assessing the Probability Distribution Forecast can be put into two categories:

1. (1) Proxy functions are biased so that the magnitude of losses is incorrect, though the ranking may be broadly correct, for example, where extreme losses are overstated by loss functions but would still be extremes in the ranking if the loss functions were accurate;

2. (2) Proxy functions are biased so that both magnitude and ranking of losses is wrong. This can arise, for example, due to turning points calculating profits when there should be losses (and vice versa) or where interactions are inaccurately modelled such that some combinations of risk have effects which are completely unexpected, for example, where a fit is accurate for scenarios when two risks both increase, but the wrong sign when the risks move in different directions.

In the former case, the model’s ability to rank risk may be appropriate, but the financial outcomes are not. For example, Economic Capital may be over or understated. Validation statistics would show that the proxy model is systematically over or understating the heavy model evenly across the full Probability Distribution Forecast.

In the latter case, the validation statistics would show errors that are not similar across the full Probability Distribution Forecast.

The reminder of this section focusses on the choices that need to be made when conducting OOS testing, and how to communicate the results.

4.6.5 Choosing scenarios for OOS testing

4.6.5.1 Number of scenarios

The Working Party is not aware of any specific approach to deriving a theoretically robust number of scenarios to test, for example, a formula based on statistical significance. Such approaches rely on the underlying assumption of randomness of errors which does not necessarily apply when replicating an underlying model.

Clearly the more scenarios tested the greater the confidence in the view formed from the tests on whether or not the proxy model is a good representation of the underlying heavy models.

The following principles are relevant in determining a suitable sample size ( Table 18).

Table 18 Principles to Determine the Number of Out-of-Sample Scenarios

A 2014 survey (Deloitte MCS Limited, 2014) of nine UK firms showed that the number of scenarios tested at the calibration date ranged from 20 to 1,000 (median 50), with a smaller number of scenarios also tested at the validation date. Clearly, time and cost are key limitations and constrain what companies can do; technological advances may mitigate this.

We expect the industry will refine its view of the suitable volume of tests as firms gain further insights from the testing they perform. They may then also become more focussed in their testing. Until then the approach is likely to be driven by the maximum achievable number within reasonable constraints.

4.6.5.2 Allocation of the scenario budget

It is important to test the full range of the scenarios across the Probability Distribution Forecast because different regions are relevant for different uses of proxy models.

Because we are looking to test the Probability Distribution Forecast, which is a representation of the full distribution of all the risk factors to which a company is exposed, the emphasis should be on testing multivariate points covering all risks. This ensures that the points tested can be mapped to the overall percentiles (and in fact are likely to be derived from percentile outputs of the model). This is useful for ensuring that interactions between risks are captured.

Scenarios that consist of a subset of risks can be appropriate when we are looking at a specific section of the model (e.g. asset, product or entity), which may be the case if the model is being used for a specific purpose, or if there are investigations that need to be performed after issues have been identified with the multivariate scenarios.

Notwithstanding the need to assess the full Probability Distribution Forecast, firms may choose to model a higher concentration of points at different percentiles, as ultimately different points will be more relevant for different uses of the model. The following table sets out an example target allocation of the number of scenarios across percentiles for a reporting entity. For this purpose, the percentile range has been split into six sections broadly aligned to usage. The scenarios are allocated across these percentiles in each fund to test various uses (Table 19).

Table 19 Allocation of Out-of-Sample Scenarios

SCR/Economic Capital: The simulations around the 99.5^th percentile are most relevant for calculating the aggregate 1-in-200 capital requirement. The SCR or other Economic Capital measures are typically calculated as the 99.5^th ranked loss across a large number of simulations.

The SCR can be under or overstated if there is a systemic error across many scenarios. For example, the SCR will be understated if proxy function fitting errors cause losses on simulations ranked at percentiles lower than the 99.5^th to be understated, where, if the fitting error was not present, those simulations would result in a loss greater than the SCR. The understatement in SCR will depend on the extent of understatement in each simulation and how close they are to the 99.5^th ranked loss.

As the 99.5^th percentile will not be known until the proxy model has been run, a firm will choose OOS scenarios using results from previous calibration of the model (which they may roll-forward to the current reporting date, as we will touch on in section 4.6.10).

To test ranking errors, simulations near the 99.5^th percentile are important. The further away from this region the greater the error needs to be to influence the ranking, although large errors are still possible where turning points exist. Systemic errors can be detected from anywhere in the range.

Diversified risks (or allocated capital): Diversified losses by risk are referenced for a number of purposes, including materiality definitions, risk appetite and prioritisation for risk management. A typical method of calculation is to take the average level of risk across simulations ranked from say 1,000 simulations around the 99.5^th percentile, possibly weighted to give more weight to simulations near the 99.5^th. The range of (in this example) 1,000 simulations is often known as “critical region” or “smoothing window”.

The accuracy of scenarios across the smoothing window is important to the accuracy of the diversified losses, and a firm may therefore wish to ensure these are covered by its OOS testing.

Short-term solvency estimation: Proxy models are commonly used to estimate changes in the economic balance sheet position between full calibrations. Typically, these are smaller movements that are actually experienced, both positive and negative. Testing scenarios at other percentiles will provide useful information on the accuracy of proxy functions in scenarios that might be encountered for this purpose. There is no specific percentile that needs to be focussed on for this purpose, but a range of levels close to the median (e.g. between the 30^th and 70^th percentiles) should be tested.

Risk appetite limits: Some insurers use proxy models to periodically set the level of capital needed to cover SCR after a 1-in-X level event, typically a 1-in-5 (80^th percentile), a 1-in-10 (90^th percentile) and a 1-in-25 (96^th percentile). Proxy models are used to estimate the impact of these events on Own Funds and on SCR by assessing a 1-in-200 loss after the 1-in-X event.

Therefore, the adverse range below the 99.5^th smoothing window needs to be tested for the Own Funds impact. Scenarios above 99.5^th will therefore be relevant to validate the capability of loss functions to calculate the SCR after a 1-in-X event.

Stress and scenario testing (SST): Firms can use proxy models to assess Own Funds and SCR impacts after a defined scenario – which may be a 1-in-X scenario generated or a real-life defined scenario such as a pandemic. The testing in regions described for risk appetite setting is also suitable for validating the use of a proxy model for SST.

4.6.6 Acceptance criteria

In order to assess whether the proxy model is fit for purpose, there is a need to define what is an acceptable deviation from the heavy model results, that is, what dictates a validation “pass” or “fail”.

The selection of tolerances for this purpose is one of the most important steps in the fitting and validation of proxy models. Tolerances determine the level of effort required in terms of the number of valuation model runs and the required complexity of the proxy model. In general, the lower the tolerances the higher will be the number of scenarios required in fitting and validation.

Setting arbitrarily tight tolerances may be spurious accuracy that will drive significant additional cost without altering management decisions. Setting tolerances which are too loose may lead to inappropriate decisions.

4.6.7 Validation measures

Firms should consider a range of statistics when analysing their OOS results. This should be a combination of descriptive and analytical statistics, and can include:

Measures of fit for individual samples:

• ∙ Monetary error (versus heavy model) in relevant currency.

• ∙ Percentage error, for example, as percentage of true heavy model result.

  Descriptive measures for the full sample set:

• ∙ Number of samples outside a specified range (defined as monetary value).

• ∙ Number of samples outside a specified range (defined as percentage).

• ∙ Number of under- or overstatements.

• ∙ MSE.

• ∙ Mean absolute error.

Analytical statistics:

• ∙ R ² (coefficient of determination)Footnote ³ . Measures how much of the variation in the heavy model result is explained by the proxy model. A value close to 1 (say ˃0.99) is expected if the proxy model is a good predictor of the heavy model.

• ∙ R (Pearson product–moment correlation coefficient). This measures the linear dependence between values from the fitted function and the in-sample modelled values.Footnote ⁴

• ∙ Coefficient of skewness: This is a measure of the extent to which the errors are biased in either direction (over-/understatement of heavy model result). This can tell us at an overall level whether the proxy model tends to under- or overstate results.

For the majority of these statistics, we can view the results across the whole loss distribution, or for specific regions of interest. This is particularly pertinent when looking at the skew of the errors. The skewness statistics could be low for the distribution as whole, but may contain clustering of bias at certain points of the curve. This is where graphical inspection of the OOS points can be instructive, since patterns in the residuals can be plotted and assessed accordingly.

4.6.8 How should the errors be investigated?

Individual errors can flag types of risk combination that are particularly inaccurate, for example, due to a missing or inaccurately modelled interaction (e.g. this can be identified by plotting OOS error against risk factor values and looking for systematic features). It may be that such combinations are rare and therefore the fail is not particularly important or it may be that the error occurs in many simulations and therefore causes errors across the Probability Distribution Forecast.

The cause of the failing individual scenario should be understood (or at least the unique features of the scenario compared with the passing scenario should be identified) and the simulation set examined to establish how frequent that type of scenario is to determine if further remedial action is justified due to pointing to a more systemic issue.

As well as looking at individual errors, a firm should examine the errors across all scenarios collectively. While individual errors may be within tolerances, this analysis could highlight systematic misstatements such as bias at a particular part of the loss distribution. For example, a firm could test for whether the number of over- or under-statements is statistically significant under the null hypothesis that the number of each follows a binomial distribution with an equal probability of over- or under-statement (parameter p=0.5).

4.6.9 Communicating validation results

The validation results can be communicated in the form of a plot showing the probability level of the chosen scenarios and how they relate to uses of the Probability Distribution Forecast.

An example of an effective plot for showing validation results is Figure 21. This is a scatter plot of proxy model result (y-axis) against the heavy model result (x-axis) for a set of OOS scenarios. If the proxy model were to produce exactly the same results as the heavy model, all points would fall on the dashed line. The plot allows us to compare the proxy model result to the heavy model result at different points of the loss distribution, and can be used to identify features visually such as:

• ∙ size and spread of residuals – seen from the vertical distance of a scenario, or cluster of scenarios, from the dashed line;

• ∙ large individual errors – as shown for two scenarios where the proxy model produces a result which is far from that of the heavy model;

• ∙ systematic under or overstatements – for example, in the region of the 50^th–80^th percentiles above where OOS results are clustered above the dashed line, indicating possible systematic overstatement of losses in that region.

Figure 21 Plot of proxy model losses against heavy model losses; SCR, solvency capital requirement

This plot can be used to communicate the suitability of the proxy models to the users of the model for each of the use types. The plot helps to highlight the percentiles and situations where the fit of the model is good and where it is poor/uncertain and the impact that can have on the key output for each model use. This will help management understand the limitations of the model and therefore assist the user to adjust use accordingly rather than over rely on it where it is less accurate.

A box–whisker plot, or a histogram of errors, can also provide an informative summary of OOS results.

The graphical plots can also assist in understanding the distribution of residuals across the range of percentiles. This can bring out any issues that are present in certain areas of the curve, but which are lost when statistics are calculated for the full sample set (such as, e.g. the large individual errors shown on Figure 21).

4.6.10 Roll-Forward

4.7.1 Mitigating a poor proxy model fit

Above we described ways in which the fit of a proxy function may not meet a firm’s tolerances. We noted that it may not be possible to re-perform the fitting process within the reporting cycle. As such, this section outlines some ways in which a firm may mitigate a poor fit.