2.1. Classical hedging problem and model-based approach

We first review the classical hedging problem for variable annuities and its model-based solution to introduce some notations and to motivate the RL approach.

2.1.3. Net liability of insurer

The liability of the insurer thus has two parts. The liability from the GMMB rider at the maturity for the i-th policyholder, where $i=1,2,\dots,N$ , is given by $\left(G_M^{\left(i\right)}-F_T^{\left(i\right)}\right)_+$ if the i-th policyholder survives beyond the maturity, and is 0 otherwise. The liability from the GMDB rider at the death time $T_{x_i}^{\left(i\right)}$ for the i-th policyholder, where $i=1,2,\dots,N$ , is given by $\left(G_D^{\left(i\right)}-F_{T_{x_i}^{\left(i\right)}}^{\left(i\right)}\right)_+$ if the i-th policyholder dies before the maturity, and is 0 otherwise. Therefore, at any time $t\in\left[0,T\right]$ , the future gross liability of the insurer accumulated to the maturity for these N contracts is given by

\begin{equation*}\sum_{i = 1}^{N}\!\left(\left(G_M^{\left(i\right)}-F_T^{\left(i\right)}\right)_+J_T^{\left(i\right)} + \frac{B_T}{B_{T_{x_i}^{\left(i\right)}}}\!\left(G_D^{\left(i\right)}-F_{T_{x_i}^{\left(i\right)}}^{\left(i\right)}\right)_+\unicode{x1d7d9}_{\{T_{x_i}^{\left(i\right)} \lt T\}}J_t^{\left(i\right)}\right).\end{equation*}

Denote $V_t^{\textrm{GL}}$ , for $t\in\left[0,T\right]$ , as the time-t value of the discounted (via the risk-free asset B) future gross liability of the insurer; if the liability is 0, the value will be 0.

From the asset-value-based fees collected by the insurer, a portion, known as the rider charge, is used to fund the liability due to the GMMB and GMDB riders; the remaining portion is used to cover overhead, commissions, and any other expenses. From the i-th policyholder, where $i=1,2,\dots,N$ , the insurer collects $m_e^{\left(i\right)}F_t^{\left(i\right)}J_t^{\left(i\right)}$ as the rider charge at any time $t\in\left[0,T\right]$ , where $m_e^{\left(i\right)}\in\!\left(0,m^{\left(i\right)}\right]$ . Therefore, the cumulative future rider charge to be collected, from any time $t\in\left[0,T\right]$ onward, till the maturity, by the insurer from these N policyholders, is given by $\sum_{i=1}^{N}\int_{t}^{T}m_e^{\left(i\right)}F_s^{\left(i\right)}J_s^{\left(i\right)}\!\left(B_T/B_s\right)ds$ . Denote $V_t^{\textrm{RC}}$ , for $t\in\left[0,T\right]$ , as its time-t discounted (via the risk-free asset B) value; if the cumulative rider charge is 0, the value will be 0.

Hence, due to these N variable annuity contracts with both GMMB and GMDB riders, for any $t\in\left[0,T\right]$ , the time-t net liability of the insurer for these N contracts is given by $L_t=V_t^{\textrm{GL}}-V_t^{\textrm{RC}}$ , which is $\mathcal{F}_t$ -measurable.

One of the many ways to set the rate $m^{\left(i\right)}\in\!\left(0,1\right)$ for the asset-value-based fees, and the rate $m_e^{\left(i\right)}\in\!\left(0,m^{\left(i\right)}\right]$ for the rider charge, for $i=1,2,\dots,N$ , is based on the time-0 net liability of the insurer for the i-th policyholder. More precisely, $m^{\left(i\right)}$ and $m_e^{\left(i\right)}$ are determined via $L_0^{\left(i\right)}=V_0^{\textrm{GL},\left(i\right)}-V_0^{\textrm{RC},\left(i\right)}=0$ , where $V_0^{\textrm{GL},\left(i\right)}$ and $V_0^{\textrm{RC},\left(i\right)}$ are the time-0 values of, respectively, the discounted future gross liability and the discounted cumulative future rider charge, of the insurer for the i-th policyholder.

2.1.4. Continuous hedging and hedging objective

The insurer aims to hedge this dual-risk bearing net liability via investing in the financial market. To this end, let $\tilde{T}$ be the death time of the last policyholder; that is, $\tilde{T}=\max_{i=1,2,\dots,N}T_{x_i}^{\left(i\right)}$ , which is random.

While the net liability $L_t$ is defined for any time $t\in\left[0,T\right]$ , as the difference between the values of discounted future gross liability and discounted cumulative future rider charge, $L_t=0$ for any $t\in\!\left(\tilde{T}\wedge T,T\right]$ . Indeed, if $\tilde{T}<T$ , then, for any $t\in\!\left(\tilde{T}\wedge T,T\right]$ , one has $T_{x_i}^{\left(i\right)}<t\leq T$ for all $i = 1,2,\dots, N$ , and hence, the future gross liability accumulated to the maturity, and the cumulative rider charge from time $\tilde{T}$ onward are both 0 so are their values. Therefore, the insurer only hedges the net liability $L_t$ , for any $t\in\left[0,\tilde{T}\wedge T\right]$ .

Let $H_t$ be the hedging strategy, i.e. the number of shares of the risky asset being held by the insurer, at time $t\in\left[0,T\right)$ . Hence, $H_t=0$ , for any $t\in\left[\tilde{T}\wedge T,T\right)$ . Let $\mathcal{H}$ be the admissible set of hedging strategies, which is defined by

\begin{align*}\mathcal{H}&=\left\{H=\left\{H_t\right\}_{t\in\left[0,T\right)}\;:\;(\textrm{i})\;H\textrm{ is }\mathbb{F}\textrm{-adapted, }(\textrm{ii})\;H\in\mathbb{R},\;\mathbb{P}\times\mathcal{L}\textrm{-a.s., and }\right.\\[5pt] & \qquad\qquad\qquad\qquad\; \left.(\textrm{iii})\;\textrm{for any }t\in\left[\tilde{T}\wedge T,T\right),\;H_t=0\right\},\end{align*}

where $\mathcal{L}$ is the Lebesgue measure on $\mathbb{R}$ . The condition (ii) indicates that there is not any constraint on the hedging strategies.

Let $P_t$ be the time-t value, for $t\in\left[0,T\right]$ , of the insurer’s hedging portfolio. Then, $P_0=0$ , and together with the rider charges collected from the N policyholders, as well as the withdrawal for paying the liabilities due to the beneficiaries’ inheritance from those policyholders who have already been dead, for any $t\in\!\left(0,T\right]$ ,

\begin{equation*}P_t=\int_{0}^{t}\!\left(P_s-H_sS_s\right)\frac{dB_s}{B_s}+\int_{0}^{t}H_sdS_s+\sum_{i=1}^{N}\int_{0}^{t}m_e^{\left(i\right)}F_s^{\left(i\right)}J_s^{\left(i\right)}ds -\sum_{i=1}^{N}\!\left(G_D^{\left(i\right)}-F_{T_{x_i}^{\left(i\right)}}^{\left(i\right)}\right)_+\unicode{x1d7d9}_{\{T_{x_i}^{\left(i\right)}\leq t \lt T\}},\end{equation*}

which obviously depends on $\left\{H_s\right\}_{s\in\left[0,t\right)}$ .

As in Bertsimas et al. (Reference Bertsimas, Kogan and Lo2000), the insurer’s hedging objective function at the current time 0 should be given by the root-mean-square error (RMSE) of the terminal profit and loss (P&L), which is, for any $H\in\mathcal{H}$ ,

\begin{equation*}\sqrt{\mathbb{E}^{\mathbb{P}}\left[\!\left(P_{\tilde{T}\wedge T}-L_{\tilde{T}\wedge T}\right)^2\right]}.\end{equation*}

If the insurer has full knowledge of the objective probability measure $\mathbb{P}$ , and hence the correct dynamics of the risk-free asset and the risky asset in the financial market, as well as the correct mortality model in the actuarial market, the optimal hedging strategy, being implemented forwardly, is given by minimising the RMSE of the terminal P&L:

\begin{equation*}H^*=\mathop{\textrm{arg min}}_{H\in\mathcal{H}}\sqrt{\mathbb{E}^{\mathbb{P}}\left[\!\left(P_{\tilde{T}\wedge T}-L_{\tilde{T}\wedge T}\right)^2\right]}.\end{equation*}

2.2. Pitfall of model-based approach

However, having correct model is usually not the case in practice. Indeed, the insurer, who is the hedging agent above, usually has little information regarding the objective probability measure $\mathbb{P}$ and hence easily misspecifies the financial market dynamics and the mortality model, which will in turn yield a poor performance from the supposedly optimal hedging strategy when it is implemented forwardly in the future. Section 2.4 outlines such an illustrative example which shall be discussed throughout the remaining of this paper.

To rectify this, we propose a two-phase (deep) RL approach to solve an optimal hedging strategy. In this approach, an RL agent, which is not the insurer themselves but is built by the insurer to hedge on their behalf, does not have any knowledge of the objective probability measure $\mathbb{P}$ , the financial market dynamics, and the mortality model; section 2.5 shall explain this approach in details. Before that, in the following section 2.3, the classical hedging problem shall first be reformulated with a Markov decision process (MDP) in a discrete time setting so that RL methods can be implemented. The illustrative example outlined in section 2.4 shall be revisited using the proposed two-phase RL approach in sections 4 and 6.

In the remaining of this paper, unless otherwise specified, all expectation operators shall be taken with respect to the objective probability measure $\mathbb{P}$ and denoted simply as $\mathbb{E}\!\left[\cdot\right]$ .

2.3. Discrete and Markov hedging

2.3.1. Discrete hedging and hedging objective

Let $t_0,t_1,\dots,t_{n-1}\in\left[0,T\right)$ , for some $n\in\mathbb{N}$ , be the time when the hedging agent decides the hedging strategy, such that $0=t_0<t_1<\dots<t_{n-1}<T$ . Denote also $t_n=T$ .

Let $t_{\tilde{n}}$ be the first time (right) after the last policyholder dies or all contracts expire, for some $\tilde{n}=1,2,\dots,n$ , which is random; that is, $t_{\tilde{n}}=\min\left\{t_k,\;k=1,2,\dots,n\;:\;t_k\geq\tilde{T}\right\}$ , and when $\tilde{T}>T$ , by convention, $\min\emptyset=t_n$ . Therefore, $H_t=0$ , for any $t=t_{\tilde{n}},t_{\tilde{n}+1},\dots,t_{n-1}$ . With a slight abuse of notation, the admissible set of hedging strategies in discrete time is

\begin{align*}\mathcal{H}=&\;\left\{H=\left\{H_t\right\}_{t=t_0,t_1,\dots,t_{n-1}}:\;(\textrm{i})\;\textrm{for any }t=t_{0},t_{1},\dots,t_{n-1},\;H_t\textrm{ is }\mathcal{F}_t\textrm{-measurable, }\right.\\[5pt] &\;\left.\quad\quad\quad\quad\quad\quad\quad\quad\quad\;\;(\textrm{ii})\;\textrm{for any }t=t_{0},t_{1},\dots,t_{n-1},\;H_t\in\mathbb{R},\;\mathbb{P}\textrm{-a.s., and}\right.\\[5pt] &\;\left.\quad\quad\quad\quad\quad\quad\quad\quad\quad\;\;(\textrm{iii})\;\textrm{for any }t=t_{\tilde{n}},t_{\tilde{n}+1},\dots,t_{n-1},\;H_t=0\right\};\end{align*}

again, the condition (ii) emphasises that no constraint is imposed to the hedging strategies.

While the hedging agent decides the hedging strategy at the discrete time points, the actuarial and financial market models are continuous. Hence, the net liability $L_t=V_t^{\textrm{GL}}-V_t^{\textrm{RC}}$ is still defined for any time $t\in\left[0,T\right]$ as before. Moreover, if $t\in\left[t_k,t_{k+1}\right)$ , for some $k=0,1,\dots,n-1$ , $H_t=H_{t_k}$ ; thus, $P_0=0$ , and, if $t\in\!\left(t_k,t_{k+1}\right]$ , for some $k=0,1,\dots,n-1$ ,

(1) \begin{align}P_t&=\left(P_{t_k}-H_{t_k}S_{t_k}\right)\frac{B_{t}}{B_{t_k}}+H_{t_k}S_{t}+\sum_{i=1}^{N}\int_{t_k}^{t}m_e^{\left(i\right)}F_s^{\left(i\right)}J_s^{\left(i\right)}\frac{B_{t}}{B_{s}}ds \nonumber \\[5pt] & \quad -\sum_{i = 1}^{N}\frac{B_t}{B_{T_{x_i}^{\left(i\right)}}}\!\left(G_D^{\left(i\right)}-F_{T_{x_i}^{\left(i\right)}}^{\left(i\right)}\right)_+\unicode{x1d7d9}_{\{t_k<T_{x_i}^{\left(i\right)}\leq t<T\}}.\end{align}

For any $H\in\mathcal{H}$ , the hedging objective of the insurer at the current time 0 is $\sqrt{\mathbb{E}\!\left[\left(P_{t_{\tilde{n}}}-L_{t_{\tilde{n}}}\right)^2\right]}$ . Hence, the optimal discrete hedging strategy, being implemented forwardly, is given by

(2) \begin{equation}H^*=\mathop{\textrm{arg min}}_{H\in\mathcal{H}}\sqrt{\mathbb{E}\!\left[\left(P_{t_{\tilde{n}}}-L_{t_{\tilde{n}}}\right)^2\right]}=\mathop{\textrm{arg min}}_{H\in\mathcal{H}}\mathbb{E}\!\left[\left(P_{t_{\tilde{n}}}-L_{t_{\tilde{n}}}\right)^2\right].\end{equation}

2.4. Illustrative example

The illustrative example below demonstrates the poor hedging performance by the Delta hedging strategy when the insurer miscalibrates the parameters in the market environment. We consider that the insurer hedges a variable annuity contract, with both GMMB and GMDB riders, of a single policyholder, i.e. $N=1$ , with the contract characteristics given in Table 1.

Table 1.

Contract characteristics.

The market environment follows the Black-Scholes (BS) in the financial part and the constant force of mortality (CFM) in the actuarial front. The risk-free asset earns a constant risk-free interest rate $r>0$ that, for any $t\in\left[0,T\right]$ , $dB_t=rB_tdt$ , while the value of the risky asset evolves as a geometric Brownian motion that, for any $t\in\left[0,T\right]$ , $dS_t=\mu S_tdt+\sigma S_tdW_t$ , where $\mu$ is a constant drift, $\sigma>0$ is a constant volatility, and $W=\left\{W_t\right\}_{t\in\left[0,T\right]}$ is the standard Brownian motion. The random future lifetime of the policyholder $T_x$ has a CFM $\nu>0$ ; that is, for any $0\leq t\leq s\leq T$ , the conditional survival probability $\mathbb{P}\!\left(T_x, \gt s\vert T_x>t\right)=e^{-\nu\left(s-t\right)}$ . Moreover, the Brownian motion W in the financial market and the future lifetime $T_x$ in the actuarial market are independent. Table 2 summarises the parameters in the market environment. Note that the risk-free interest rate, the risky asset initial price, the initial age of the policyholder, and the investment strategy of the policyholder are observable by the insurer.

Table 2.

Parameters setting of market environment.

Based on their best knowledge of the market, the insurer builds a model of the market environment. Suppose that the model happens to be the BS and the CFM as the market environment, but the insurer miscalibrates the parameters. Table 3 lists these parameters in the model of the market environment. In particular, the risky asset drift and volatility, as well as the force of mortality constant, are different from those in the market environment. For the observable parameters, they are the same as those in the market environment.

Table 3.

Parameters setting of model of market environment, with bolded parameters being different from those in market environment.

At any time $t\in\left[0,T\right]$ , the value of the hedging portfolio of the insurer is given by (17), with $N=1$ , in which the values of the risky asset and the single-jump process follow the market environment with the parameters in Table 2. At any time $t\in\left[0,T\right]$ , the value of the net liability of the insurer is given by (16), with $N=1$ , in both the market environment and its model; for its detailed derivations, we defer it to section 4.1, as the model of the market environment, with multiple homogeneous policyholders for effective training, shall be supplied as the training environment. Since the parameters in the model of the market environment (see Table 3) are different from those in the market environment (see Table 2), the net liability evaluated by the insurer using the model is different from that of the market environment. There are two implications. Firstly, the Delta hedging strategy of the insurer using the parameters in Table 3 is incorrect, while the correct Delta hedging strategy should use the parameters in Table 2. Secondly, the asset-value-based fee m and the rider charge $m_e$ given in Table 4, which are determined by the insurer based on the time-0 value of their net liability by Table 3 via the method in section 2.1.3, are mispriced. They would not lead to zero time-0 value of their net liability in the market environment which is based on Table 2.

Table 4.

Fee structures derived from model of market environment.

To evaluate the hedging performance of the incorrect Delta strategy by the insurer in the market environment for the variable annuity of contract characteristics in Table 1, 5,000 market scenarios using the parameters in Table 2 are simulated to realise terminal P&Ls. For comparison, the terminal P&Ls by the correct Delta hedging strategy are also obtained. Figure 1 shows the empirical density and cumulative distribution functions of the 5,000 realised terminal P&Ls by each Delta hedging strategy, while Table 5 outlines the summary statistics of the empirical distributions, in which $\widehat{\textrm{RMSE}}$ is the estimated RMSE of the terminal P&L similar to (2).

Table 5.

Summary statistics of empirical distributions of realised terminal P&Ls by different Delta strategies.

Figure 1

Empirical density and cumulative distribution functions of realised terminal P&Ls by different Delta strategies.

In Figure 1(a), the empirical density function of realised terminal P&Ls by the incorrect Delta hedging strategy is depicted to be more heavy-tailed on the left than that by the correct Delta strategy. In fact, the terminal P&L by the incorrect Delta hedging strategy is stochastically dominated by that by the correct Delta strategy in the first-order; see Figure 1(b). Table 5 shows that the terminal P&L by the incorrect Delta hedging strategy has a mean and a median farther from zero, a higher standard deviation, larger left-tail risks in terms of Value-at-Risk and Tail Value-at-Risk, and a larger RMSE than that by the correct Delta strategy.

These observations conclude that, even in a market environment as simple as the BS and the CFM, the incorrect Delta hedging strategy based on the miscalibrated parameters by the insurer does not perform well when it is being implemented forwardly. In general, the hedging performance of model-based approaches depends crucially on the calibration of parameters for the model of the market environment.

2.5. Two-phase reinforcement learning approach

In an RL approach, at the current time 0, the insurer builds an RL agent to hedge on their behalf in the future. The agent interacts with a market environment, by sequentially observing states, taking, as well as revising, actions, which are the hedging strategies, and collecting rewards. Without possessing any prior knowledge of the market environment, the agent needs to, explore the environment while exploit the collected reward signals, for effective learning.

An intuitive proposition would be allowing an infant RL agent to learn directly from such market environment, like the one in section 2.4, moving forward. However, recall that the insurer actually does not know any exact market dynamics in the environment and thus is not able to provide any theoretical model for the net liability to the RL agent. In turn, the RL agent could not receive any sequential anchor-hedging reward signal in (9) from the environment, but instead receives the single terminal reward signal in (8). Since the rewards, except the terminal one, are all zero, the infant RL agent would learn ineffectively from such sparse rewards, i.e. the RL agent shall take a tremendous amount of time to finally learn a nearly optimal hedging strategy in the environment. Most importantly, while the RL agent is exploring and learning from the environment, which is not a simulated one, the insurer could suffer from huge financial burden due to any sub-optimal hedging performances.

In view of this, we propose that the insurer should first designate the infant RL agent to interact and learn from a training environment, which is constructed by the insurer based on their best knowledge of the market, for example, the model of the market environment in section 2.4. Since the training environment is known to the insurer (but is unknown to the RL agent), the RL agent can be supplied by a net liability theoretical model, and consequently learn from the sequential anchor-hedging reward signal in (9) of the training environment. Therefore, the infant RL agent would be guided by the net liability to learn effectively from the local hedging errors. After interacting and learning from the training environment for a period of time, in order to gauge the effectiveness, the RL agent shall be tested for its hedging performance in simulated scenarios from the same training environment. This first phase is called the training phase.

Training Phase:

1. (i) The insurer constructs the MDP training environment.

2. (ii) The insurer builds the infant RL agent which uses the PPO algorithm.

3. (iii) The insurer assigns the RL agent in the MDP training environment to interact and learn for a period of time, during which the RL agent collects the anchor-hedging reward signal in (9).

4. (iv) The insurer deploys the trained RL agent to hedge in simulated scenarios from the same training environment and documents the baseline hedging performance.

If the hedging performance of the trained RL agent in the training environment is satisfactory, the insurer should then proceed to assign it to interact and learn from the market environment. Since the training and market environments are usually different, such as having different parameters as in section 2.4, the initial hedging performance of the trained RL agent in the market environment is expected to diverge from the fine baseline hedging performance in the training environment. However, different from an infant RL agent, the trained RL agent is experienced so that the sparse reward signal in (8) should be sufficient for the agent to revise the hedging strategy, from the nearly optimal one in the training environment to that in the market environment, within a reasonable amount of time. This second phase is called the online learning phase.

Online Learning Phase:

1. (v) The insurer assigns the RL agent in the market environment to interact and learn in real time, during which the RL agent collects the single terminal reward signal in (8).

These summarise the proposed two-phase RL approach. Figure 2 depicts the above sequence clearly. There are several assumptions underneath this two-phase RL approach in order to apply it effectively to a hedging problem of a contingent claim; as they involve specifics in later sections, we collate their discussions and elaborate their implications in practice in section 7. In the following section, we shall briefly review the training essentials of RL in order to introduce the PPO algorithm. For the details of online learning phase, we defer them until section 5.

Figure 2

The relationship among insurer, RL agent, MDP training environment, and market environment of the two-phase RL approach.

3.1. Stochastic action for exploration

One of the fundamental ideas in RL is that, at any time $t_k$ , where $k=0,1,\dots,{n-1}$ , given the current state $X_{t_k}$ , the RL agent does not take a deterministic action $H_{t_k}$ but extends it to a stochastic action, in order to explore the MDP environment and in turn learn from the reward signals. The stochastic action is sampled through a so-called policy, which is defined below.

Let $\mathcal{P}\!\left(\mathcal{A}\right)$ be a set of probability measures over the action space $\mathcal{A}$ ; each probability measure $\mu\left(\cdot\right)\in\mathcal{P}\!\left(\mathcal{A}\right)$ maps a Borel set $\overline{A}\in\mathcal{B}\!\left(\mathcal{A}\right)$ to $\mu\left(\overline{A}\right)\in\left[0,1\right]$ . The policy $\pi\!\left(\cdot\right)$ is a mapping from the state space $\mathcal{X}$ to the set of probability measures $\mathcal{P}\!\left(\mathcal{A}\right)$ ; that is, for any state $x\in\mathcal{X}$ , $\pi\!\left(x\right)=\mu\left(\cdot\right)\in\mathcal{P}\!\left(\mathcal{A}\right)$ . The value function and the optimal value function, at any time $t_k$ , where $k=0,1,\dots,\tilde{n}-1$ , with the state $x\in\mathcal{X}$ , are then generalised as, for any policy $\pi\!\left(\cdot\right)$ ,

(10) \begin{equation}V\!\left(t_k,x;\;\pi\!\left(\cdot\right)\right)=\mathbb{E}\!\left[\sum_{l=k}^{\tilde{n}-1}R_{t_{l+1}}\Big\vert X_{t_k}=x\right],\quad V^*\left(t_k,x\right)=\sup_{\pi\!\left(\cdot\right)}V\!\left(t_k,x;\;\pi\!\left(\cdot\right)\right);\end{equation}

at any time $t_k$ , where $k=\tilde{n},\tilde{n}+1,\dots,n-1$ , with the state $x\in\mathcal{X}$ , for any policy $\pi\!\left(\cdot\right)$ , $V\!\left(t_k,x;\;\pi\!\left(\cdot\right)\right)=V^*\left(t_k,x\right)=0$ . In particular, if $\mathcal{P}\!\left(\mathcal{A}\right)$ contains only all Dirac measures over the action space $\mathcal{A}$ , which is the case in the DH approach of Bühler et al. (Reference Bühler, Gonon, Teichmann and Wood2019) (see Appendix A for more details), the value function and the optimal value function reduce to (4) and (6). With this relaxed setting, solving the optimal hedging strategy $H^*$ boils down to finding the optimal policy $\pi^*\!\left(\cdot\right)$ .

3.2. Policy approximation and parameterisation

As the hedging problem has the infinite action space $\mathcal{A}$ , tabular solution methods for problems of finite state space and finite action space (such as Q-learning), or value function approximation methods for problems of infinite state space and finite action space (such as deep Q-learning) are not suitable. Instead, a policy gradient method is employed.

To this end, the policy $\pi\!\left(\cdot\right)$ is approximated and parametrised by the weights $\theta_{\textrm{p}}$ in an artificial neural network (ANN); in turn, denote the policy by $\pi\!\left(\cdot;\;\theta_{\textrm{p}}\right)$ . The ANN $\mathcal{N}_{\textrm{p}}\!\left(\cdot;\;\theta_{\textrm{p}}\right)$ (to be defined in (11) below) takes a state $x\in\mathcal{X}$ as the input vector and output parameters of a probability measure in $\mathcal{P}\!\left(\mathcal{A}\right)$ . In the sequel, the set $\mathcal{P}\!\left(\mathcal{A}\right)$ contains all Gaussian measures (see, for example, Wang et al. Reference Wang, Zariphopoulou and Zhou2020 and Wang & Zhou Reference Wang and Zhou2020), in which each has a mean c and a variance $d^2$ , which depend on the state input $x\in\mathcal{X}$ and the ANN weights $\theta_{\textrm{p}}$ . Therefore, for any state $x\in\mathcal{X}$ ,

\begin{equation*}\pi\!\left(x;\;\theta_{\textrm{p}}\right)=\mu\left(\cdot;\;\theta_{\textrm{p}}\right)\sim\textrm{Gaussian}\!\left(c\!\left(x;\;\theta_{\textrm{p}}\right),d^2\left(x;\;\theta_{\textrm{p}}\right)\right),\end{equation*}

where $\left(c\!\left(x;\;\theta_{\textrm{p}}\right),d^2\!\left(x;\;\theta_{\textrm{p}}\right)\right)=\mathcal{N}_{\textrm{p}}\!\left(x;\;\theta_{\textrm{p}}\right)$ .

With such approximation and parameterisation, solving the optimal policy $\pi^*$ further boils down to finding the optimal ANN weights $\theta^*_{\textrm{p}}$ . Hence, denote the value function and the optimal value function in (10) by $V\!\left(t_k,x;\;\theta_{\textrm{p}}\right)$ and $V\!\left(t_k,x;\;\theta^*_{\textrm{p}}\right)$ , for any $t_k$ , where $k=0,1,\dots,\tilde{n}-1$ , with $x\in\mathcal{X}$ . However, the (optimal) value function still depends on the objective probability measure $\mathbb{P}$ , the financial market dynamics, and the mortality model, which are unknown to the RL agent. Before formally introducing the policy gradient methods to tackle this issue, we shall first explicitly construct the ANNs for the approximated policy, as well as for an estimate of the value function (to prepare the algorithm of policy gradient method to be reviewed below).

3.3. Network architecture

As alluded above, in this paper, the ANN involves two parts, which are the policy network and the value function network.

3.3.1. Policy network

Let $N_{\textrm{p}}$ be the number of layers for the policy network. For $l=0,1,\dots,N_{\textrm{p}}$ , let $d_{\textrm{p}}^{\left(l\right)}$ be the dimension of the l-th layer, where the 0-th layer is the input layer; the $1,2,\dots,\left(N_{\textrm{p}}-1\right)$ -th layers are hidden layers; the $N_{\textrm{p}}$ -th layer is the output layer. In particular, $d_{\textrm{p}}^{\left(0\right)}=p$ , which is the number of features in the actuarial and financial parts, and $d_{\textrm{p}}^{\left(N_{\textrm{p}}\right)}=2$ , which outputs the mean c and the variance $d^2$ of the Gaussian measure. The policy network $\mathcal{N}_{\textrm{p}}\;:\;\mathbb{R}^{p}\rightarrow\mathbb{R}^{2}$ is defined as, for any $x\in\mathbb{R}^{p}$ ,

(11) \begin{equation}\mathcal{N}_{\textrm{p}}\!\left(x\right)=\left(W_{\textrm{p}}^{\left(N_{\textrm{p}}\right)}\circ\psi\circ W_{\textrm{p}}^{\left(N_{\textrm{p}}-1\right)}\circ\psi\circ W_{\textrm{p}}^{\left(N_{\textrm{p}}-2\right)}\circ\dots\circ\psi\circ W_{\textrm{p}}^{\left(1\right)}\right)\left(x\right),\end{equation}

where, for $l=1,2,\dots,N_{\textrm{p}}$ , the mapping $W_{\textrm{p}}^{\left(l\right)}\;:\;\mathbb{R}^{d_{\textrm{p}}^{\left(l-1\right)}}\rightarrow\mathbb{R}^{d_{\textrm{p}}^{\left(l\right)}}$ is affine, and the mapping $\psi\;:\;\mathbb{R}^{d_{\textrm{p}}^{\left(l\right)}}\rightarrow\mathbb{R}^{d_{\textrm{p}}^{\left(l\right)}}$ is a componentwise activation function. Let $\theta_{\textrm{p}}$ be the parameter vector of the policy network and in turn denote the policy network in (11) by $\mathcal{N}_{\textrm{p}}\!\left(x;\;\theta_{\textrm{p}}\right)$ , for any $x\in\mathbb{R}^p$ .

3.3.2. Value function network

The value function network is constructed similarly as in the policy network, except that all subscripts p (policy) are replaced by v (value). In particular, the value function network $\mathcal{N}_{\textrm{v}}\;:\;\mathbb{R}^{p}\rightarrow\mathbb{R}$ is defined as, for any $x\in\mathbb{R}^{p}$ ,

(12) \begin{equation}\mathcal{N}_{\textrm{v}}\!\left(x\right)=\left(W_{\textrm{v}}^{\left(N_{\textrm{v}}\right)}\circ\psi\circ W_{\textrm{v}}^{\left(N_{\textrm{v}}-1\right)}\circ\psi\circ W_{\textrm{v}}^{\left(N_{\textrm{v}}-2\right)}\circ\dots\circ\psi\circ W_{\textrm{v}}^{\left(1\right)}\right)\left(x\right),\end{equation}

which models an approximated value function $\hat{V}$ (see section 3.4 below). Let $\theta_{\textrm{v}}$ be the parameter vector of the value function network and in turn denote the value function network in (12) by $\mathcal{N}_{\textrm{v}}\!\left(x;\;\theta_{\textrm{v}}\right)$ , for any $x\in\mathbb{R}^p$ .

3.3.3. Shared layers structure

Since the policy and value function networks should extract features from the input state vector in a similar manner, they are assumed to share the first few layers. More specifically, let $N_{\textrm{s}}\!\left(<\min\left\{N_{\textrm{p}},N_{\textrm{v}}\right\}\right)$ be the number of shared layers for the policy and value function networks; for $l=1,2,\dots,N_{\textrm{s}}$ , $W_{\textrm{p}}^{\left(l\right)}=W_{\textrm{v}}^{\left(l\right)}=W_{\textrm{s}}^{\left(l\right)}$ , and hence, for any $x\in\mathbb{R}^{p}$ ,

\begin{equation*}\mathcal{N}_{\textrm{p}}\!\left(x;\;\theta_{\textrm{p}}\right)=\left(W_{\textrm{p}}^{\left(N_{\textrm{p}}\right)}\circ\psi\circ W_{\textrm{p}}^{\left(N_{\textrm{p}}-1\right)}\circ\dots\circ\psi\circ W_{\textrm{p}}^{\left(N_{\textrm{s}}+1\right)}\circ\psi\circ W_{\textrm{s}}^{\left(N_{\textrm{s}}\right)}\circ\dots\circ\psi\circ W_{\textrm{s}}^{\left(1\right)}\right)\left(x\right),\end{equation*}

\begin{equation*}\mathcal{N}_{\textrm{v}}\!\left(x;\;\theta_{\textrm{v}}\right)=\left(W_{\textrm{v}}^{\left(N_{\textrm{v}}\right)}\circ\psi\circ W_{\textrm{v}}^{\left(N_{\textrm{v}}-1\right)}\circ\dots\circ\psi\circ W_{\textrm{v}}^{\left(N_{\textrm{s}}+1\right)}\circ\psi\circ W_{\textrm{s}}^{\left(N_{\textrm{s}}\right)}\circ\dots\circ\psi\circ W_{\textrm{s}}^{\left(1\right)}\right)\left(x\right).\end{equation*}

Let $\theta$ be the parameter vector of the policy and value function networks. Figure 3 depicts such a shared layers structure.

Figure 3

An example of policy and value function artificial neural networks with a shared hidden layer and a non-shared hidden layer.

3.4. Proximal policy optimisation: a temporal-difference policy gradient method

A policy gradient method entails that, starting from initial ANN weights $\theta^{\left(0\right)}$ , and via interacting with the MDP environment to observe the states and collect the reward signals, the RL agent gradually updates the ANN weights, by the (stochastic) gradient ascent on a certain surrogate performance measure defined for the ANN weights. That is, at each update step $u=1,2,\dots$ ,

(13) \begin{equation}\theta^{\left(u\right)}=\theta^{\left(u-1\right)}+\alpha\widehat{\nabla_{\theta}\mathcal{J}^{\left(u-1\right)}\!\left(\theta^{\left(u-1\right)}\right)},\end{equation}

where the hyperparameter $\alpha\in\left[0,1\right]$ is the learning rate of the RL agent, and, based on the experienced episode(s), $\widehat{\nabla_{\theta}\mathcal{J}^{\left(u-1\right)}\!\left(\theta^{\left(u-1\right)}\right)}$ is the estimated gradient of the surrogate performance measure $\mathcal{J}^{\left(u-1\right)}\!\left(\cdot\right)$ evaluating at $\theta=\theta^{\left(u-1\right)}$ .

REINFORCE, which is pioneered by Williams (Reference Williams1992), is a Monte Carlo policy gradient method, which updates the ANN weights by each episode. As this paper applies a temporal-difference (TD) policy gradient method, we relegate the review of REINFORCE to Appendix B, where the Policy Gradient Theorem, the foundation of any policy gradient methods, is presented.

PPO, which is pioneered by Schulman et al. (Reference Schulman, Wolski, Dhariwal, Radford and Klimov2017), is a TD policy gradient method, which updates the ANN weights by a batch of $K\in\mathbb{N}$ realisations. At each update step $u=1,2,\dots$ , based on the ANN weights $\theta^{\left(u-1\right)}$ , and thus the policy $\pi\!\left(\cdot;\;\theta_{\textrm{p}}^{\left(u-1\right)}\right)$ , the RL agent experiences $E^{\left(u\right)}\in\mathbb{N}$ realised episodes for the K realisations.

• If $E^{\left(u\right)}=1$ , the episode is given by

  \begin{align*}&\;\left\{\dots,x_{t_{K_s^{\left(u\right)}}}^{\left(u-1\right)},h_{t_{K_s^{\left(u\right)}}}^{\left(u-1\right)},x_{t_{K_s^{\left(u\right)}+1}}^{\left(u-1\right)},r_{t_{K_s^{\left(u\right)}+1}}^{\left(u-1\right)},h_{t_{K_s^{\left(u\right)}+1}}^{\left(u-1\right)},\right.\\[5pt] &\;\left.\quad\dots,x_{t_{K_s^{\left(u\right)}+K-1}}^{\left(u-1\right)},r_{t_{K_s^{\left(u\right)}+K-1}}^{\left(u-1\right)},h_{t_{K_s^{\left(u\right)}+K-1}}^{\left(u-1\right)},x_{t_{K_s^{\left(u\right)}+K}}^{\left(u-1\right)},r_{t_{K_s^{\left(u\right)}+K}}^{\left(u-1\right)},\dots\right\},\end{align*}
  where $K_s^{\left(u\right)}=0,1,\dots,\tilde{n}-1$ , such that the time $t_{K_s^{\left(u\right)}}$ is when the episode is initiated in this update, and $h_{t_k}^{\left(u-1\right)}$ , for $k=0,1,\dots,\tilde{n}-1$ , is the time $t_k$ realised hedging strategy being sampled from the Gaussian distribution with the mean $c\!\left(x_{t_k}^{\left(u-1\right)};\;\theta_{\textrm{p}}^{\left(u-1\right)}\right)$ and the variance $d^2\left(x_{t_k}^{\left(u-1\right)};\;\theta_{\textrm{p}}^{\left(u-1\right)}\right)$ ; necessarily, $\tilde{n}-K_s^{\left(u\right)}\geq K$ .

• If $E^{\left(u\right)}=2,3,\dots$ , the episodes are given by

  \begin{align*}&\;\left\{\dots,x_{t_{K_s^{\left(u\right)}}}^{\left(u-1,1\right)},h_{t_{K_s^{\left(u\right)}}}^{\left(u-1,1\right)},x_{t_{K_s^{\left(u\right)}+1}}^{\left(u-1,1\right)},r_{t_{K_s^{\left(u\right)}+1}}^{\left(u-1,1\right)},h_{t_{K_s^{\left(u\right)}+1}}^{\left(u-1,1\right)},\right.\\[5pt] &\;\left.\quad\dots,x_{t_{\tilde{n}^{\left(1\right)}-1}}^{\left(u-1,1\right)},r_{t_{\tilde{n}^{\left(1\right)}-1}}^{\left(u-1,1\right)},h_{t_{\tilde{n}^{\left(1\right)}-1}}^{\left(u-1,1\right)},x_{t_{\tilde{n}^{\left(1\right)}}}^{\left(u-1,1\right)},r_{t_{\tilde{n}^{\left(1\right)}}}^{\left(u-1,1\right)}\right\},\\[5pt] &\;\left\{x_{t_0}^{\left(u-1,2\right)},h_{t_0}^{\left(u-1,2\right)},x_{t_1}^{\left(u-1,2\right)},r_{t_1}^{\left(u-1,2\right)},h_{t_1}^{\left(u-1,2\right)},\right.\\[5pt] &\;\left.\quad\dots,x_{t_{\tilde{n}^{\left(2\right)}-1}}^{\left(u-1,2\right)},r_{t_{\tilde{n}^{\left(2\right)}-1}}^{\left(u-1,2\right)},h_{t_{\tilde{n}^{\left(2\right)}-1}}^{\left(u-1,2\right)},x_{t_{\tilde{n}^{\left(2\right)}}}^{\left(u-1,2\right)},r_{t_{\tilde{n}^{\left(2\right)}}}^{\left(u-1,2\right)}\right\},\\[5pt] &\;\dots,\end{align*}
  \begin{align*}&\;\left\{x_{t_0}^{\left(u-1,E^{\left(u\right)}-1\right)},h_{t_0}^{\left(u-1,E^{\left(u\right)}-1\right)},x_{t_1}^{\left(u-1,E^{\left(u\right)}-1\right)},r_{t_1}^{\left(u-1,E^{\left(u\right)}-1\right)},h_{t_1}^{\left(u-1,E^{\left(u\right)}-1\right)},\right.\\[5pt] &\;\left.\quad\dots,x_{t_{\tilde{n}^{\left(E^{\left(u\right)}-1\right)}-1}}^{\left(u-1,E^{\left(u\right)}-1\right)},r_{t_{\tilde{n}^{\left(E^{\left(u\right)}-1\right)}-1}}^{\left(u-1,E^{\left(u\right)}-1\right)},h_{t_{\tilde{n}^{\left(E^{\left(u\right)}-1\right)}-1}}^{\left(u-1,E^{\left(u\right)}-1\right)},x_{t_{\tilde{n}^{\left(E^{\left(u\right)}-1\right)}}}^{\left(u-1,E^{\left(u\right)}-1\right)},r_{t_{\tilde{n}^{\left(E^{\left(u\right)}-1\right)}}}^{\left(u-1,E^{\left(u\right)}-1\right)}\right\},\\[5pt] &\;\left\{x_{t_0}^{\left(u-1,E^{\left(u\right)}\right)},h_{t_0}^{\left(u-1,E^{\left(u\right)}\right)},x_{t_1}^{\left(u-1,E^{\left(u\right)}\right)},r_{t_1}^{\left(u-1,E^{\left(u\right)}\right)},h_{t_1}^{\left(u-1,E^{\left(u\right)}\right)},\right.\\[5pt] &\;\left.\quad\dots,x_{t_{K_f^{\left(u\right)}-1}}^{\left(u-1,E^{\left(u\right)}\right)},r_{t_{K_f^{\left(u\right)}-1}}^{\left(u-1,E^{\left(u\right)}\right)},h_{t_{K_f^{\left(u\right)}-1}}^{\left(u-1,E^{\left(u\right)}\right)},x_{t_{K_f^{\left(u\right)}}}^{\left(u-1,E^{\left(u\right)}\right)},r_{t_{K_f^{\left(u\right)}}}^{\left(u-1,E^{\left(u\right)}\right)},\dots\right\},\end{align*}
  where $K_f^{\left(u\right)}=1,2,\dots,\tilde{n}^{\left(E^{\left(u\right)}\right)}$ , such that the time $t_{K_f^{\left(u\right)}}$ is when the last episode is finished (but not necessarily terminated) in this update; necessarily, $\tilde{n}^{\left(1\right)}-K_s^{\left(u\right)}+\sum_{e=2}^{E^{\left(u\right)}-1}\tilde{n}^{\left(e\right)}+K_f^{\left(u\right)}=K$ .

The surrogate performance measure of PPO consists of three components. In the following, fix an update step $u=1,2,\dots$ .

Inspired by Schulman et al. (Reference Schulman, Levine, Moritz, Jordan and Abbeel2015), in which the time-0 value function difference between two policies is shown to be equal to the expected advantage, together with importance sampling and KL divergence constraint reformulation, the first component in the surrogate performance measure of PPO is given by:

• if $E^{\left(u\right)}=1$ ,

  \begin{equation*}L_{\textrm{CLIP}}^{\left(u-1\right)}\!\left(\theta_{\textrm{p}}\right)=\mathbb{E}\!\left[\sum_{k=K_s^{\left(u\right)}}^{K_s^{\left(u\right)}+K-1}\min\left\{q_{t_k}^{\left(u-1\right)}\hat{A}^{\left(u-1\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k},\textrm{clip}\!\left(q_{t_k}^{\left(u-1\right)},1-\epsilon,1+\epsilon\right)\hat{A}^{\left(u-1\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k}\right\}\right],\end{equation*}
  where the importance sampling ratio $q_{t_k}^{\left(u-1\right)}=\frac{\phi\left(H^{\left(u-1\right)}_{t_k};\;X^{\left(u-1\right)}_{t_k},\theta_{\textrm{p}}\right)}{\phi\left(H^{\left(u-1\right)}_{t_k};\;X^{\left(u-1\right)}_{t_k},\theta^{\left(u-1\right)}_{\textrm{p}}\right)}$ , in which $\phi\left(\cdot;\;X^{\left(u-1\right)}_{t_k},\theta_{\textrm{p}}\right)$ is the Gaussian density function with mean $c\!\left(X^{\left(u-1\right)}_{t_k};\;\theta_{\textrm{p}}\right)$ and variance $d^2\left(X^{\left(u-1\right)}_{t_k};\;\theta_{\textrm{p}}\right)$ , the estimated advantage is evaluated at $\theta_{\textrm{p}}=\theta^{\left(u-1\right)}_{\textrm{p}}$ and bootstrapped through the approximated value function that
  \begin{equation*}\hat{A}^{\left(u-1\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k}=\begin{cases}\sum_{l=k}^{K_s^{\left(u\right)}+K-1}R_{t_{l+1}}^{\left(u-1\right)}+\hat{V}\!\left(t_{K_s^{\left(u\right)}+K},X^{\left(u-1\right)}_{t_{K_s^{\left(u\right)}+K}};\;\theta_{\textrm{v}}^{\left(u-1\right)}\right)&\\[5pt] \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\;-\hat{V}\!\left(t_k,X^{\left(u-1\right)}_{t_k};\;\theta_{\textrm{v}}^{\left(u-1\right)}\right)&\textrm{if }K_s^{\left(u\right)}+K<\tilde{n},\\[5pt] \sum_{l=k}^{\tilde{n}-1}R_{t_{l+1}}^{\left(u-1\right)}-\hat{V}\!\left(t_k,X^{\left(u-1\right)}_{t_k};\;\theta_{\textrm{v}}^{\left(u-1\right)}\right)&\textrm{if }K_s^{\left(u\right)}+K=\tilde{n},\\[5pt] \end{cases}\end{equation*}
  and the function $\textrm{clip}\!\left(q_{t_k}^{\left(u-1\right)},1-\epsilon,1+\epsilon\right)=\min\left\{\max\left\{q_{t_k}^{\left(u-1\right)},1-\epsilon\right\},1+\epsilon\right\}$ . The approximated value function $\hat{V}$ is given by the output of the value network, i.e. $\hat{V}\!\left(t_k,X^{\left(u-1\right)}_{t_k};\;\theta_{\textrm{v}}^{\left(u-1\right)}\right)=\mathcal{N}_{\textrm{v}}\!\left(X^{\left(u-1\right)}_{t_k};\;\theta_{\textrm{v}}^{\left(u-1\right)}\right)$ as defined in (12) for $k=0,1,\dots,\tilde{n}-1$ .

• if $E^{\left(u\right)}=2,3,\dots$ ,

  \begin{align*}&\;L_{\textrm{CLIP}}^{\left(u-1\right)}\!\left(\theta_{\textrm{p}}\right)=\mathbb{E}\!\left[\sum_{k=K_s^{\left(u\right)}}^{\tilde{n}^{\left(1\right)}-1}\min\left\{q_{t_k}^{\left(u-1,1\right)}\hat{A}^{\left(u-1,1\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k},\textrm{clip}\!\left(q_{t_k}^{\left(u-1,1\right)},1-\epsilon,1+\epsilon\right)\hat{A}^{\left(u-1,1\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k}\right\}\right.\\[5pt] &\;\left.+\sum_{e=2}^{E^{\left(u\right)}-1}\sum_{k=0}^{\tilde{n}^{\left(e\right)}-1}\min\left\{q_{t_k}^{\left(u-1,e\right)}\hat{A}^{\left(u-1,e\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k},\textrm{clip}\!\left(q_{t_k}^{\left(u-1,e\right)},1-\epsilon,1+\epsilon\right)\hat{A}^{\left(u-1,e\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k}\right\}\right.\\[5pt] &\;\left.+\sum_{k=0}^{K_f^{\left(u\right)}-1}\min\left\{q_{t_k}^{\left(u-1,E^{\left(u\right)}\right)}\hat{A}^{\left(u-1,E^{\left(u\right)}\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k},\textrm{clip}\!\left(q_{t_k}^{\left(u-1,E^{\left(u\right)}\right)},1-\epsilon,1+\epsilon\right)\hat{A}^{\left(u-1,E^{\left(u\right)}\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k}\right\}\right].\end{align*}

Similar to REINFORCE in Appendix B, the second component in the surrogate performance measure of PPO minimises the loss between the bootstrapped sum of reward signals and the approximated value function. To this end, define:

• if $E^{\left(u\right)}=1$ ,

  \begin{equation*}L_{\textrm{VF}}^{\left(u-1\right)}\!\left(\theta_{\textrm{v}}\right)=\mathbb{E}\!\left[\sum_{k=K_s^{\left(u\right)}}^{K_s^{\left(u\right)}+K-1}\!\left(\hat{A}^{\left(u-1\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k}+\hat{V}\!\left(t_k,X^{\left(u-1\right)}_{t_k};\;\theta_{\textrm{v}}^{\left(u-1\right)}\right)-\hat{V}\!\left(t_k,X^{\left(u-1\right)}_{t_k};\;\theta_{\textrm{v}}\right)\right)^2\right];\end{equation*}

• if $E^{\left(u\right)}=2,3,\dots$ ,

  \begin{align*}&\;L_{\textrm{VF}}^{\left(u-1\right)}\!\left(\theta_{\textrm{v}}\right)=\mathbb{E}\!\left[\sum_{k=K_s^{\left(u\right)}}^{\tilde{n}^{\left(1\right)}-1}\!\left(\hat{A}^{\left(u-1,1\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k}+\hat{V}\!\left(t_k,X^{\left(u-1,1\right)}_{t_k};\;\theta_{\textrm{v}}^{\left(u-1\right)}\right)-\hat{V}\!\left(t_k,X^{\left(u-1,1\right)}_{t_k};\;\theta_{\textrm{v}}\right)\right)^2\right.\\[5pt] &\;\left.+\sum_{e=2}^{E^{\left(u\right)}-1}\sum_{k=0}^{\tilde{n}^{\left(e\right)}-1}\!\left(\hat{A}^{\left(u-1,e\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k}+\hat{V}\!\left(t_k,X^{\left(u-1,e\right)}_{t_k};\;\theta_{\textrm{v}}^{\left(u-1\right)}\right)-\hat{V}\!\left(t_k,X^{\left(u-1,e\right)}_{t_k};\;\theta_{\textrm{v}}\right)\right)^2\right.\\[5pt] &\;\left.+\sum_{k=0}^{K_f^{\left(u\right)}-1}\!\left(\hat{A}^{\left(u-1,E^{\left(u\right)}\right)}_{\theta^{\left(u-1\right)}_{\textrm{p}},t_k}+\hat{V}\!\left(t_k,X^{\left(u-1,E^{\left(u\right)}\right)}_{t_k};\;\theta_{\textrm{v}}^{\left(u-1\right)}\right)-\hat{V}\!\left(t_k,X^{\left(u-1,E^{\left(u\right)}\right)}_{t_k};\;\theta_{\textrm{v}}\right)\right)^2\right].\end{align*}

Finally, to encourage the RL agent exploring the MDP environment, the third component in the surrogate performance measure of PPO is the entropy bonus. Based on the Gaussian density function, define

• if $E^{\left(u\right)}=1$ ,

  \begin{equation*}L_{\textrm{EN}}^{\left(u-1\right)}\!\left(\theta_{\textrm{p}}\right)=\mathbb{E}\!\left[\sum_{k=K_s^{\left(u\right)}}^{K_s^{\left(u\right)}+K-1}\ln d\left(X^{\left(u-1\right)}_{t_k};\;\theta_{\textrm{p}}\right)\right];\end{equation*}

• if $E^{\left(u\right)}=2,3,\dots$ ,

  \begin{align*}&\;L_{\textrm{EN}}^{\left(u-1\right)}\!\left(\theta_{\textrm{p}}\right)=\mathbb{E}\!\left[\sum_{k=K_s^{\left(u\right)}}^{\tilde{n}^{\left(1\right)}-1}\ln d\left(X^{\left(u-1,1\right)}_{t_k};\;\theta_{\textrm{p}}\right)+\sum_{e=2}^{E^{\left(u\right)}-1}\sum_{k=0}^{\tilde{n}^{\left(e\right)}-1}\ln d\left(X^{\left(u-1,e\right)}_{t_k};\;\theta_{\textrm{p}}\right)\right.\\[5pt] &\;\left.\quad\quad\quad\quad\quad\quad\quad\quad+\sum_{k=0}^{K_f^{\left(u\right)}-1}\ln d\left(X^{\left(u-1,E^{\left(u\right)}\right)}_{t_k};\;\theta_{\textrm{p}}\right)\right].\end{align*}
  Therefore, the surrogate performance measure of PPO is given by:
  (14) \begin{equation}\mathcal{J}^{\left(u-1\right)}\!\left(\theta\right)=L_{\textrm{CLIP}}^{\left(u-1\right)}\!\left(\theta_{\textrm{p}}\right)-c_1L_{\textrm{VF}}^{\left(u-1\right)}\!\left(\theta_{\textrm{v}}\right)+c_2L_{\textrm{EN}}^{\left(u-1\right)}\!\left(\theta_{\textrm{p}}\right),\end{equation}

where the hyperparameters $c_1,c_2\in\left[0,1\right]$ are the loss coefficients of the RL agent. Its estimated gradient, based on the K realisations, is then computed via automatic differentiation; see, for example, Baydin et al. (Reference Baydin, Pearlmutter, Radul and Siskind2018).

4.1. Markov decision process training environment

The model of the market environment is the BS and the CFM in the financial and the actuarial parts. However, unlike the model following the market environment to write a single contract to a single policyholder, for effective training, the insurer writes identical contracts to N homogeneous policyholders in the training environment. Because of the homogeneity of the contracts and the policyholders, for all $i=1,2,\dots,N$ , $x_i=x$ , $\rho^{\left(i\right)}=\rho$ , $m^{\left(i\right)}=m$ , $G_M^{\left(i\right)}=G_M$ , $G_D^{\left(i\right)}=G_D$ , $m_e^{\left(i\right)}=m_e$ , and $F_t^{\left(i\right)}=F_t=\rho S_te^{-mt}$ , for $t\in\left[0,T\right]$ .

At any time $t\in\left[0,T\right]$ , the future gross liability of the insurer accumulated to the maturity is thus $\left(G_M-F_T\right)_+\sum_{i = 1}^{N}J_T^{\left(i\right)} +\sum_{i = 1}^{N} e^{r\left(T - T_{x}^{\left(i\right)} \right)}\!\left(G_D-F_{T_{x}^{\left(i\right)}}\right)_+\unicode{x1d7d9}_{\{T_{x}^{\left(i\right)} \lt T\}}J_{t}^{\left(i\right)},$ and its time-t discounted value is

\begin{align*}V_t^{\textrm{GL}}&=e^{-r\left(T-t\right)}\mathbb{E}^{\mathbb{Q}}\left[\left(G_M-F_T\right)_+\sum_{i=1}^{N}J_T^{\left(i\right)}\Big\vert\mathcal{F}_t\right] \\[5pt] & \quad +\mathbb{E}^{\mathbb{Q}}\left[\sum_{i=1}^{N}e^{-r\left(T_{x}^{\left(i\right)}-t\right)}\!\left(G_D-F_{T_{x}^{\left(i\right)}}\right)_+\unicode{x1d7d9}_{\{T_{x}^{\left(i\right)} \lt T\}}J_{t}^{\left(i\right)}\Big\vert\mathcal{F}_t\right]\\[5pt] \end{align*}

\begin{align*} &=e^{-r\left(T-t\right)}\mathbb{E}^{\mathbb{Q}}\left[\left(G_M-F_T\right)_+\vert\mathcal{F}_t\right]\sum_{i=1}^{N}\mathbb{E}^{\mathbb{Q}}\left[J_T^{\left(i\right)}\big\vert\mathcal{F}_t\right] \\[5pt] &\quad +\sum_{i=1}^{N}J_t^{\left(i\right)}\mathbb{E}^{\mathbb{Q}}\left[e^{-r\left(T_{x}^{\left(i\right)}-t\right)}\!\left(G_D-F_{T_{x}^{\left(i\right)}}\right)_+\unicode{x1d7d9}_{\{T_{x}^{\left(i\right)} \lt T\}}\Big\vert\mathcal{F}_t\right],\end{align*}

where the probability measure $\mathbb{Q}$ defined on $\left(\Omega,\mathcal{F}\right)$ is an equivalent martingale measure with respect to $\mathbb{P}$ . Herein, the probability measure $\mathbb{Q}$ is chosen to be the product measure of each individual equivalent martingale measure in the actuarial or financial part, which implies the independence among the Brownian motion W and the future lifetime $T_x^{\left(1\right)},T_x^{\left(2\right)},\dots,T_x^{\left(N\right)}$ , clarifying the first term in the second equality above. The second term in that equality is due to the fact that, for $i=1,2,\dots,N$ , the single-jump process $J^{\left(i\right)}$ is $\mathbb{F}$ -adapted. Under the probability measure $\mathbb{Q}$ , all future lifetime are identically distributed and have a CFM $\nu>0$ , which are the same as those under the probability measure $\mathbb{P}$ in section 2.4. Therefore, for any $i=1,2,\dots,N$ , and for any $0\leq t\leq s\leq T$ , the conditional survival probability $\mathbb{Q}\!\left(T_x^{\left(i\right)}>s\vert T_x^{\left(i\right)}>t\right)=e^{-\nu\left(s-t\right)}$ . For each policyholder $i=1,2,\dots,N$ , by the independence and the Markov property, for any $0\leq t\leq s\leq T$ ,

(15) \begin{equation}\mathbb{E}^{\mathbb{Q}}\left[J_s^{\left(i\right)}\big\vert\mathcal{F}_t\right]=\mathbb{E}^{\mathbb{Q}}\left[J_s^{\left(i\right)}\big\vert J_t^{\left(i\right)}\right]=\begin{cases}\mathbb{Q}\!\left(T_x^{\left(i\right)}>s\vert T_x^{\left(i\right)}\leq t\right)=0 & \textrm{if}\quad T_x^{\left(i\right)}\!\left(\omega\right)\leq t\\[5pt] \mathbb{Q}\!\left(T_x^{\left(i\right)}>s\vert T_x^{\left(i\right)}>t\right)=e^{-\nu\left(s-t\right)} & \textrm{if}\quad T_x^{\left(i\right)}\!\left(\omega\right)>t\\[5pt] \end{cases}.\end{equation}

Moreover, under the probability measure $\mathbb{Q}$ , for any $t\in\left[0,T\right]$ , $dF_t=\left(r-m\right)F_tdt+\sigma F_tdW_t^{\mathbb{Q}}$ , where $W^{\mathbb{Q}}=\left\{W^{\mathbb{Q}}_t\right\}_{t\in\left[0,T\right]}$ is the standard Brownian motion under the probability measure $\mathbb{Q}$ . Hence, the time-t value of the discounted future gross liability, for $t\in\left[0,T\right]$ , is given by

\begin{align*}V_t^{\textrm{GL}}=&\;e^{-\nu\left(T-t\right)}\!\left(G_Me^{-r\left(T-t\right)}\Phi\left(-d_2\left(t,G_M\right)\right)-F_te^{-m\left(T-t\right)}\Phi\left(-d_1\left(t,G_M\right)\right)\right)\sum_{i=1}^{N}J_t^{\left(i\right)}\\[5pt] &\;+\int_{t}^{T}\!\left(G_De^{-r\left(T-s\right)}\Phi\left(-d_2\left(s,G_D\right)\right)-F_te^{-m\left(T-s\right)}\Phi\left(-d_1\left(s,G_D\right)\right)\right)\nu e^{-\nu\left(s-t\right)}ds\sum_{i=1}^{N}J_t^{\left(i\right)}, \end{align*}

where, for $s \in \left[0,T\right)$ and $G \gt 0$ , $d_1\left(s,G\right)=\frac{\ln\left(\frac{F_s}{G}\right)+\left(r-m+\frac{\sigma^2}{2}\!\left(T-s\right)\right)}{\sigma\sqrt{T-s}}$ , $d_2\left(s,G\right)=d_1\left(s,G\right)-\sigma\sqrt{T-s}$ , $d_1\left(T,G\right) = \lim_{s\rightarrow T^{-}}d_1\left(s,G\right)$ , $d_2\left(T,G\right) = d_1\left(T,G\right)$ , and $\Phi\left(\cdot\right)$ is the standard Gaussian distribution function. Note that $\sum_{i=1}^{N}J_t^{\left(i\right)}$ represents the number of surviving policyholders at time $t\in\left[0,T\right]$ .

As for the cumulative future rider charge to be collected by the insurer from any time $t\in\left[0,T\right]$ onward, it is given by $\sum_{i=1}^{N}\int_{t}^{T}m_eF_sJ_s^{\left(i\right)}e^{r(T-s)}ds$ , and its time-t discounted value is

\begin{equation*}V_t^{\textrm{RC}}=e^{-r\left(T-t\right)}\mathbb{E}^{\mathbb{Q}}\!\left[\!\sum_{i=1}^{N}\int_{t}^{T}m_eF_sJ_s^{\left(i\right)}e^{r(T-s)}ds\Big\vert\mathcal{F}_t\!\right]=\sum_{i=1}^{N}\int_{t}^{T}m_ee^{-r\left(s-t\right)}\mathbb{E}^{\mathbb{Q}}\left[F_s\vert F_t\right]\mathbb{E}^{\mathbb{Q}}\left[J_s^{\left(i\right)}\big\vert J_t^{\left(i\right)}\right]ds,\end{equation*}

where the second equality is again due to the independence and the Markov property. Under the probability measure $\mathbb{Q}$ , $\mathbb{E}^{\mathbb{Q}}\left[F_s\vert F_t\right]=e^{\left(r-m\right)\left(s-t\right)}F_t$ . Together with (15),

\begin{equation*}V_t^{\textrm{RC}}=\frac{1-e^{-\left(m+\nu\right)\left(T-t\right)}}{m+\nu}m_eF_t\sum_{i=1}^{N}J_t^{\left(i\right)}.\end{equation*}

Therefore, the time-t net liability of the insurer, for $t\in\left[0,T\right]$ , is given by

(16) \begin{align}L_t=V_t^{\textrm{GL}}-V_t^{\textrm{RC}}&=\Bigg(e^{-\nu\left(T-t\right)}\!\left(G_Me^{-r\left(T-t\right)}\Phi\left(-d_2\left(t,G_M\right)\right)-F_te^{-m\left(T-t\right)}\Phi\left(-d_1\left(t,G_M\right)\right)\right)\nonumber \\[5pt] &\quad+\int_{t}^{T}\!\left(G_De^{-r\left(T-s\right)}\Phi\left(-d_2\left(s,G_D\right)\right)-F_te^{-m\left(T-s\right)}\Phi\left(-d_1\left(s,G_D\right)\right)\right) \nonumber \\[5pt] &\quad \; \nu e^{-\nu\left(s-t\right)}ds-\frac{1-e^{-\left(m+\nu\right)\left(T-t\right)}}{m+\nu}m_eF_t\Bigg)\sum_{i=1}^{N}J_t^{\left(i\right)}, \end{align}

which contributes parts of the reward signals in (9). The time-t value of the insurer’s hedging portfolio, for $t\in\left[0,T\right]$ , as in (1), is given by: $P_0=0$ , and if $t\in\!\left(t_k,t_{k+1}\right]$ , for some $k=0,1,\dots,n-1$ ,

(17) \begin{align}P_t&=\left(P_{t_k}-H_{t_k}S_{t_k}\right)e^{r\left(t-t_k\right)}+H_{t_k}S_{t}+m_e\int_{t_k}^{t}F_se^{r\left(t-s\right)}\sum_{i=1}^{N}J_s^{\left(i\right)}ds\nonumber \\[5pt] &\quad -\sum_{i = 1}^{N}e^{r\left(t-T_{x}^{\left(i\right)}\right)}\!\left(G_D-F_{T_{x}^{\left(i\right)}}\right)_+\unicode{x1d7d9}_{\{t_k<T_{x}^{\left(i\right)}\leq t<T\}},\end{align}

which is also supplied to the reward signals in (9).

At each time $t_k$ , where $k=0,1,\dots,n$ , the RL agent is given to observe four features from this MDP environment; these four features are summarised in the state vector

(18) \begin{equation}X_{t_k}=\left(\ln{F_{t_k}},\frac{P_{t_k}}{N},\frac{\sum_{i=1}^{N}J^{\left(i\right)}_{t_k}}{N},T-t_k\right).\end{equation}

The first feature is the natural logarithm of the segregated account value of the policyholder. The second feature is the hedging portfolio value of the insurer, being normalised by the initial number of policyholders. The third feature is the ratio of the number of surviving policyholders with respect to the initial number of policyholders. These features are either log-transformed or normalised to prevent the RL agent from exploring and learning from features with high variability. The last feature is the term to maturity. In particular, when either the third or the last feature first hits zero, i.e. at time $t_{\tilde{n}}$ , an episode is terminated. The state space $\mathcal{X}=\mathbb{R}\times\mathbb{R}\times\left[0,1/N, 2/N,\dots, 1\right]\times\left\{0,t_1,t_2,\dots,T\right\}$ .

Recall that, at each time $t_k$ , where $k=0,1,\dots,\tilde{n}-1$ , with the state vector (18) being the input, the output of the policy network in (11) is the mean $c\!\left(X_{t_k};\;\theta_{\textrm{p}}\right)$ and the variance $d^2\left(X_{t_k};\;\theta_{\textrm{p}}\right)$ of a Gaussian measure; herein, the Gaussian measure represents the distribution of the average number of shares of the risky asset being held by the insurer at the time $t_k$ for each surviving policyholder. Hence, for $k=0,1,\dots,\tilde{n}-1$ , the hedging strategy $H_{t_k}$ in (17) is given by $H_{t_k}=\overline{H}_{t_k}\sum_{i=1}^{N}J^{\left(i\right)}_{t_k}$ , where $\overline{H}_{t_k}$ is sampled from the Gaussian measure. Since the hedging strategy is assumed to be Markovian with respect to the state vector, it can be shown, albeit tedious, that the state vector, in (18), and the hedging strategy together satisfy the Markov property in (3).

Also recall that the infant RL agent is trained in the MDP environment with multiple homogeneous policyholders. The RL agent should then effectively update the ANN weights $\theta$ and learn the hedging strategies, via a more direct inference on the force of mortality from the third feature in the state vector. The RL agent hedges daily, so that the difference between the consecutive discrete hedging time is $\delta t_{k}=t_{k+1}-t_{k}=\frac{1}{252}$ , for $k=0,1,\dots,n-1$ . In this MDP training environment, the parameters of the model are given in Table 3, but with $N=500$ .

4.2. Building reinforcement learning agent

After constructing this MDP training environment, the insurer builds the RL agent which implements the PPO, which was reviewed in section 3.4. Table 6(a) summarises all hyperparameters of the implemented PPO, in which three of them are determined via grid searchFootnote ² , while the remaining two are fixed a priori since they alter the surrogate performance measure itself, and thus should not be based on grid search. Table 6(b) outlines the hyperparameters of the ANN architecture in section 3.3, which are all pre-specified, in which ReLU stands for Rectified Linear Unit; that is, the componentwise activation function is given by, for any $z\in\mathbb{R}$ , $\psi\!\left(z\right)=\max\left\{z,0\right\}$ .

Table 6.

Hyperparameters setting of Proximal Policy Optimisation and neural network.

4.3. Training of reinforcement learning agent

With all these being set up, the insurer assigns the RL agent experiencing this MDP training environment, in order to observe the state, decide, as well as revise, the hedging strategy, and collect the anchor-hedging reward signal based on (9), as much as possible. Let $\mathcal{U}\in\mathbb{N}$ be the number of update steps in the training environment on the ANN weights. Hence, the policy of the experienced RL agent is given by $\pi\!\left(\cdot;\;\theta^{\left(\mathcal{U}\right)}\right)=\pi\!\left(\cdot;\;\theta_{\textrm{p}}^{\left(\mathcal{U}\right)}\right)$ .

Figure 4 depicts the training log of the RL agent in terms of bootstrapped sum of rewards and batch entropy. In particular, Figure 4(a) shows that the value function in (2) reduces to almost zero after around $10^8$ training timesteps, which is equivalent to around 48,828 update steps for the ANN weights; within the same number of training timesteps, Figure 4(b) illustrates a gradual depletion on the batch entropy, and hence, the Gaussian measure gently becomes more concentrating around its mean, which implies that the RL agent progressively diminishes the degree of exploration on the MDP training environment, while increases the degree of exploitation on the learned ANN weights.

Figure 4

Training log in terms of bootstrapped sum of rewards and batch entropy.

4.4. Baseline hedging performance

In the final step of the training phase, the trained RL agent is assigned to hedge in simulated scenarios from the same MDP training environment, except that $N=1$ which is in line with hedging in the market environment. The trained RL agent takes the deterministic action $c\!\left(\cdot;\;\theta_{\textrm{p}}^{\left(\mathcal{U}\right)}\right)$ which is the mean of the Gaussian measure.

The number of simulated scenarios is 5,000. For each scenario, the insurer documents the realised terminal P&L, i.e. $P_{t_{\tilde{n}}}-L_{t_{\tilde{n}}}$ . After all scenarios are experienced by the trained RL agent, the insurer examines the baseline hedging performance via the empirical distribution and the summary statistics of the realised terminal P&Ls. The baseline hedging performance of the RL agent is also benchmarked with those by other methods, namely the classical Deltas and the DH; see Appendix C for the implemented hyperparameters of the DH training. The following four classical Deltas are implemented in the simulated scenarios from the training environment, in which the (in)correctness of the Deltas are with respect to the training environment:

• (correct) Delta of the CFM actuarial and BS financial models with the model parameters as in Table 3;

• (incorrect) Delta of the increasing force of mortality (IFM) actuarial and BS financial models, where, for any $i=1,2,\dots,N$ , if $T<\overline{b}$ , the conditional survival probability $\mathbb{Q}\!\left(T_x^{\left(i\right)}>s\vert T_x^{\left(i\right)}>t\right)=\frac{\overline{b}-s}{\overline{b}-t}$ , for any $0\leq t\leq s\leq T<\overline{b}$ , while if $\overline{b}\leq T$ , the conditional survival probability $\mathbb{Q}\!\left(T_x^{\left(i\right)}>s\vert T_x^{\left(i\right)}>t\right)=\frac{\overline{b}-s}{\overline{b}-t}$ , for any $0\leq t\leq s<\overline{b}\leq T$ , and $\mathbb{Q}\!\left(T_x^{\left(i\right)}>s\vert T_x^{\left(i\right)}>t\right)=0$ , for any $0\leq t\leq\overline{b}\leq s\leq T$ or $0\leq\overline{b}\leq t\leq s\leq T$ , with the model parameters as in Tables 3(a) and 7;

• (incorrect) Delta in the CFM actuarial and Heston financial models, where, for any $t\in\left[0,T\right]$ , $dS_t=\mu S_tdt+\sqrt{\Sigma_t}S_tdW_t^{\left(1\right)}$ , $d\Sigma_t=\kappa\left(\overline{\Sigma}-\Sigma_t\right)dt+\eta\sqrt{\Sigma_t}dW_t^{\left(2\right)}$ , and $\left\langle W^{(1)},W^{(2)}\right\rangle_t=\phi t$ , with the model parameters as in Tables 3(b) and 8;

• (incorrect) Delta in the IFM actuarial and Heston financial models with the model parameters as in Tables 7 and 8.

Table 7.

Parameters setting of increasing force of mortality actuarial model for Delta.

Table 8.

Parameters setting of Heston financial model for Delta.

Figure 5 shows the empirical density and cumulative distribution functions via the 5,000 realised terminal P&Ls by each hedging approach, while Table 9 outlines the summary statistics of these empirical distributions. To clearly illustrate the comparisons, Figure 6 depicts the empirical density functions via the 5,000 pathwise differences of the realised terminal P&Ls between the RL agent and each of the other approaches, while Table 10 lists the summary statistics of the empirical distributions; for example, comparing with the DH approach, the pathwise difference of the realised terminal P&Ls for the e-th simulated scenario, for $e=1,2,\dots,5,000$ , is calculated by $\left(P_{t_{\tilde{n}}}^{\textrm{RL}}\!\left(\omega_e\right)-L_{t_{\tilde{n}}}^{\textrm{RL}}\!\left(\omega_e\right)\right)-\left(P_{t_{\tilde{n}}}^{\textrm{DH}}\!\left(\omega_e\right)-L_{t_{\tilde{n}}}^{\textrm{DH}}\!\left(\omega_e\right)\right)$ .

Figure 5

Empirical density and cumulative distribution functions of realised terminal P&Ls by the approaches of reinforcement learning, classical Deltas, and deep hedging.

As expected, the baseline hedging performance of the trained RL agent in this training environment is comparable with those by, the correct CFM and BS Delta, as well as the DH approach. Moreover, the RL agent outperforms all the other three incorrect Deltas, which are based on either incorrect IFM actuarial or Heston financial model, or both.

7.1. Observable, sufficient, relevant, and transformed features in state

One of the crucial components in an MDP environment of the training phase or the online learning phase is the state, in which the features provide information from the environment to the RL agent. First, the features must be observable by the RL agent for learning. For instance, in our proposed state vectors (18) and (19), all the four features, namely the segregated account value, the hedging portfolio value, the number of surviving policyholders, and the term to maturity, are observable. Any unobservable, albeit desirable, features cannot be included in the state, such as insider information which could provide a better inference on the future value of a risky asset, or exact health condition of a policyholder. Second, the observable features in the state should be sufficient for the RL agent to learn. For example, due to the dual-risk bearing nature of the contract in this paper, the proposed state vectors (18) and (19) incorporate both financial and actuarial features; also, the third and the fourth features in the state vectors (18) and (19) would inform the RL agent to halt its hedging at the terminal time. However, incorporating sufficient observable features in the state does not imply that every observable feature in the environment should be included; the observable features in the state need to be relevant for learning efficiently. Since the segregated account value and the term to maturity have already been included in the state vectors (18) and (19) as features, the risky asset value and the hedging time are respective similar information from the environment and thus are redundant features to be contained in the state. Finally, the features in the state which have high variance might be appropriately transformed for reducing the volatility due to exploration. For instance, the segregated account value in the state vectors (18) and (19) is log-transformed in both phases.

7.2. Reward engineering

Another crucial component in an MDP environment is the reward, which supplies signals to the RL agent to evaluate its actions, i.e. the hedging strategy, for learning. First, the reward signals, if available, should suggest the local hedging performance. For example, in this paper, the RL agent is provided by the sequential anchor-hedging reward, given in (9), in the training phase; through the net liability value in the MDP training environment, the RL agent often receives a positive (resp. negative) signal for encouragement (resp. punishment), which is more informative than collecting the zero reward. However, any informative reward signals need to be computable from an MDP environment. In this paper, since the insurer does not know the MDP market environment, the RL agent could not be supplied the sequential anchor-hedging reward signals, which consist of the net liability values, in the online learning phase, even though they are more informative; instead, the RL agent is given the less informative single terminal reward, given in (8), in the online learning phase which can be computed from the market environment.

7.3. Markov property in state and action

In an MDP environment of the training phase or the online learning phase, the state and action pair needs to satisfy the Markov property as in (3). In the training phase, since the MDP training environment is constructed, the Markov property can be verified theoretically for the state, with the included features in line with section 7.1, and the action, which is the hedging strategy. For example, in this paper, with the model of the market environment being the BS and the CFM, the state vector in (18) and the Markovian hedging strategy satisfy the Markov property in the training phase. Since the illustrative example in this paper assumes that the market environment also follows the BS and the CFM, the state vector in (19) and the Markovian hedging strategy satisfy the Markov property in the online learning phase as well. However, in general, as the market environment is unknown, the Markov property for the state and action pair would need to be checked statistically in the online phase as follows.

After the training phase and before an RL agent proceeding to the online learning phase, historical state and action sequences in a time frame are derived by hypothetically writing identical contingent claims and using the historical realisations from the market environment. For instance, historical values of risky assets are publicly available, or an insurer retrieves historical survival status of its policyholders with similar demographic information and medical history as the policyholder being actually written. These historical samples of the state and action pair are then used to conduct hypothesis testing on whether the Markov property in (3) holds for the pair in the market environment, by, for example, the test statistics proposed in Chen & Hong (Reference Chen and Hong2012). If the Markov property holds statistically, the RL agent could begin the online learning phase. Yet, if the property does not hold statistically, the state and action pair should be revised and then the training phase should be revisited; since the hedging strategy is the action in a hedging problem, only the state could be amended by including more features from the environment. Moreover, during the online learning phase, right after each further update step, new historical state and action sequences in a shifted time frame of the same duration are obtained together with the most recent historical realisations from the market environment and using the action samples being drawn from the updated policy. These regularly new samples should be applied to statistically verify the Markov property on a rolling basis. If the property fails to hold at any time, the state needs to be revised and the RL agent must be re-trained before resuming the online learning.

7.4. Re-establishment of contingent claims in online learning phase

Any contingent claims must have a finite terminal time realisation. On one hand, in the training phase, that would be the time when an episode ends and the state is re-initialised so that the RL agent can be trained in the training environment as long as possible. On the other hand, in the online learning phase, the market environment, and hence the state, could not be re-initialised; instead, at each terminal time realisation, the seller re-establishes identical contingent claims of the same contract characteristics and writing on (more or less) the same assets so that the RL agent can be trained in the market environment successively. In this paper, the terms to maturity and the minimum guarantees of all variable annuity contracts in the online learning phase are the same. Moreover, all re-established contracts therein write on the same financial risky asset, though the initial values of the asset are given by the real-time realisations in the market environment. Finally, while a new policyholder is written at each contract inception time, these policyholders have similar, if not identical, distributions of their random future lifetimes via examining their demographic information and medical history.