2.1. Degradation model

Consider a finite collection $\mathcal{N}:=\{1,\ldots,N\}$ of N wind turbines operating in an offshore wind farm. Critical wind turbine components (e.g., generators, transformers, rotors, and turbine blades) degrade over time due to normal usage and the influence of their ambient operating environment. For example, it is well known that ocean air is corrosive to wind turbine blades due to its higher salt content [Reference Dong, Yuan and Ma22]. In what follows, the phrase “turbine degradation” refers to the degradation of a single, critical component whose replacement requires substantial time, effort, and resources. That is, we do not specify which component but refer more generally to the turbine’s degradation. Each turbine degrades until its cumulative level of degradation first reaches, or exceeds, a critical deterministic threshold ξ (ξ > 0), at which time the turbine is said to be failed. In the problem formulation that follows, we consider discrete decision epochs $m\in \mathbb{Z}_+:=\{0,1,2,\ldots\}$ and define a period as the time between two consecutive decision epochs. Let $X_n(m)$ denote the degradation of turbine $n\in\mathcal{N}$ at decision epoch m with state space $[0,\xi]$. Here, level 0 means a turbine is in as-good-as-new condition. Next, define the vector-valued process $\mathbb{X}=\{\left(X_1(m),\ldots,X_N(m)\right):m\in \mathbb{Z}_+\}$ describing the joint degradation process of all N turbines with continuous state space $\mathcal{X}:=[0,\xi]^N$. At decision epoch m, turbine n is said to be working if $X_n(m)\in [0,\xi)$ and failed if $X_n(m)\ge\xi$. Hereafter, and without loss of generality, we set the failure threshold ξ = 1. Stochastic processes are used extensively in the reliability and maintenance optimization communities to model the temporal and/or spatial evolution of degradation in systems. For a single system, $\{X_n(m):m\in \mathbb{Z}_+\}$ is often assumed to evolve as a discrete-time Markov chain (DTMC) on a finite state space, or when a continuous-time, continuous-state process is more appropriate, degradation can be modeled using gamma, Wiener, or other Lévy processes [Reference Andrawus, Watson and Kishk6, Reference Besnard and Bertling10, Reference Liu, Pandey, Wang and Zhao38, Reference Nielsen and Sørensen48].

For our purposes here, it is appropriate to model not only degradation but also the influence of a randomly varying environment on the evolution of degradation. Wind turbines are often located in proximity to one another within a wind farm; thus, they experience not only similar environmental conditions but also wake effects resulting from changes in wind speed [Reference Ding21]. Although the turbines operate independently, their degradation processes exhibit stochastic dependence due to their exposure to a common environment. Monbet et al. [Reference Monbet, Ailliot and Prevosto44] surveyed stochastic models—many of which are Markovian—that can be used to model wind conditions and the sea state from time series data. Although the environment realistically evolves continuously in time and space, researchers have assumed that it evolves over finitely many discrete states. Representative examples include [Reference Abdul-Malak and Kharoufeh2, Reference Bian, Gebraeel and Kharoufeh11, Reference Kharoufeh and Cox31, Reference Kharoufeh, Finkelstein and Mixon32, Reference Ulukuş, Kharoufeh and Maillart71]. In those specific models, each environment state maps to a unique, positive degradation rate. That is, if the environment occupies state i, then the system degrades linearly at a rate $r(i) \gt 0$. Consequently, in the absence of any type of maintenance intervention, the degradation process evolves as a piecewise linear function of time (almost surely). In contrast to those models, our main theoretical results do not impose such restrictive assumptions but only require that future degradation can be predicted using the current level of degradation and the environment state. The discrete environment states are essentially aggregations of several factors, including wind speed, ambient temperature, relative humidity, wave height, and others. Hence, these states can be viewed as approximations of the true (continuous) environment states over time. For example, if the environment evolves as a diffusion model with a randomly evolving drift coefficient, the distinct environment states can be viewed as the average behavior of the diffusion process within an interval of time. Flory et al. [Reference Flory, Kharoufeh and Gebraeel23] employed Markov chain Monte Carlo methods to model such behavior.

Let J(m) denote the state of the environment at decision epoch m and assume $\mathbb{J}:=\{J(m):m\in\mathbb{Z}_+\}$ evolves as a temporally homogeneous DTMC on the finite state space $\mathcal{L}:=\{1,\ldots,L\}$ with $L\in \mathbb{N}$. The environment states are ordered such that state 1 represents the most favorable condition for maintenance activities and state L represents the least favorable condition. To account for accessibility, resource availability, and maintenance feasibility issues, we can append $\mathcal{L}$ with additional states that reflect conditions under which maintenance cannot be performed (e.g., when wind speeds and/or wave heights are excessive, transport vessels are unavailable and/or crews are unavailable). In such cases, the only feasible action is “do nothing” and wait until the next decision epoch. The one-step transition probability matrix (TPM) of $\mathbb{J}$ is denoted by $\mathbf{P}=[p_{\ell k}]$, where $p_{\ell k}$ is the conditional probability that the environment state is k at decision epoch m + 1, given that it was $\ell$ at epoch m for any $m\in \mathbb{Z}_+$. The number of distinct environment states (L) can be estimated using the well-known Bayesian information criterion [Reference Flory, Kharoufeh and Gebraeel23, Reference Schwarz60]. Anastasiou and Tsekos [Reference Anastasiou and Tsekos4] concluded that six-to-eight distinct states are sufficient to model environmental conditions using a first-order, stationary Markov chain and noted that the stationarity assumption has been employed to model wind speed and wave height in other maritime applications [Reference Bern and Houmb8, Reference MacCallum and Bolland39]. It is well-known that weather conditions exhibit strong seasonality effects, as highlighted and modeled by Byon et al. [Reference Byon and Ding12, Reference Byon, Ntaimo and Ding13]. To account for these effects, one can employ a different DTMC model for different seasons of the year. The transition probabilities can be estimated using proportions data [Reference Jones29, Reference McGuire41] or by mining time series data [Reference Song, Geng, Kusiak and Xu66]. A realization of the joint process $(\mathbb{X},\mathbb{J})$ is an $(N+1)$-dimensional vector $(\textbf{x},\ell)$, where $\textbf{x}=(x_1,\ldots,x_N)\in\mathcal{X}$ and $\ell\in \mathcal{L}$.

We assume that future degradation can be predicted via the current state $(\textbf{x},\ell)$. For each turbine $n\in \mathcal{N}$ and a triplet $(x_n,\ell,t)$, let $f_n:[0,1]\times \mathcal{L} \times \mathbb{R}_{+}\to \mathbb{R}_{+}$ be a generic mapping that predicts the degradation of turbine n after t time units when its current degradation level is x_n and the environment state is $\ell$. It can assume any number of functional forms (e.g., linear, piecewise linear, exponential, etc.); however, we impose no such assumption. This mapping satisfies the initial condition $f_n(x_n,\ell,0)=x_n$, and we impose the natural assumption that, for each $n\in \mathcal{N}$, f_n is monotone nondecreasing in $(x_n,\ell,t)$, i.e., $f_n(x_n,\ell,t)\le f_n(x^{\prime}_n,\ell^{\prime},t^{\prime})$ for any $x_n\le x^{\prime}_n$, $\ell\le \ell^{\prime}$, and $t\le t^{\prime}$. That is, future degradation is assumed to be higher than the present level of degradation, and because the state space $\mathcal{L}$ is ordered, higher environment states lead to accelerated degradation of the units; however, we impose no additional assumptions on f_n. Condition monitoring or sensor data (e.g., from a Supervisory Control and Data Acquisition (SCADA) system) can be used to estimate the evolution of degradation and/or the RUL of components using the techniques developed in [Reference Bian, Gebraeel and Kharoufeh11, Reference Gebraeel, Lawley, Li and Ryan24], for example. Papadopoulos et al. [Reference Papadopoulos, Coit and Ezzat53] suggest that when a dynamic environment is considered, the RUL can be estimated using the models developed by Bian et al. [Reference Bian, Gebraeel and Kharoufeh11].

2.2. Problem formulation

Maples et al. [Reference Maples, Saur, Hand, van de Pietermen and Obdam40] note that the majority of wind turbine O&M costs are due to the replacement of critical components (e.g., generators, transformers, rotors, and turbine blades). Thus, optimally timed replacements that exploit economies of scale can reduce O&M costs for wind farms and the overall cost of wind energy to consumers. Here, we present an MDP model to formulate the problem of optimally replacing wind turbines with the objective of minimizing the expected total discounted, setup, replacement and downtime costs over an infinite horizon.

Periodic inspections, which are assumed to be costless and instantaneous, are performed to reveal the state of each turbine perfectly (i.e., the state is completely observable). Such observations can be obtained, for example, by using sensors attached to critical components and tracking important signals of degradation, such as vibration or acoustic signals [Reference Bian, Gebraeel and Kharoufeh11]. A decision to replace or not replace a unit is made by a decision maker (DM) just after an inspection such that $\mathbb{Z}_+=\{0,1,\ldots\}$ is the set of decision epochs, and the time between any two consecutive inspections is a period in the MDP model. The state of the system is the $(N+1)$-dimensional vector $(\textbf{x},\ell)$ whose state space is $\Gamma:=\mathcal{X} \times \mathcal{L}$. For any decision epoch $m\in \mathbb{Z}_+$, let $a_n\in \{0,1\}$ denote the action taken for turbine $n\in \mathcal{N}$, where $a_n=0$ means “do nothing” and $a_n=1$ means “replace.” Therefore, the N-dimensional vector $\textbf{a}=(a_1,\ldots,a_N)$ describes the actions taken by the DM, and the action space is $\mathcal{A}:=\{0,1\}^N$. Whenever replacement is prescribed, the turbine must be shut down for one full period. Replacement activities commence immediately after the inspection and end just prior to the subsequent inspection. We assume that a period is sufficiently long to ensure that replacements are completed during a single period. Because the average time to complete a major replacement is between 2 to 7 days (see Maples et al. [Reference Maples, Saur, Hand, van de Pietermen and Obdam40]), the time between two consecutive decision epochs is 1 week in our model. However, it is important to note that the length of the period can be adjusted to account for different time scales when the model is used to prescribe the optimal timing of maintenance actions for minor components. In such cases, the period length can be set to 1 day. All replacements are assumed to be perfect and reset the degradation to level 0. For the remainder of the paper, vector inequalities are assumed to hold component-wise. That is, $\textbf{a}\le \textbf{a}^{\prime}$ implies $a_n\le a_n^{\prime}$, and $\textbf{x}\le \textbf{x}^{\prime}$ implies $x_n\le x_n^{\prime}$ for each $n\in \mathcal{N}$.

Let ${\hat{\textbf{x}}}_n(x_n,\ell,a_n)$ be the predicted level of degradation of turbine n at the start of the next period when the current degradation and environment states are x_n and $\ell$, respectively, and action a_n is taken. Then,

(1)\begin{equation} {\hat{\textbf{x}}}_n(x_n,\ell,a_n) = \begin{cases} \min\{1,f_n(x_n,\ell,1)\}, & \text{if}\ a_n=0,\\ 0, & \text{if}\ a_n=1. \end{cases} \end{equation}

Following a replacement, degradation is returned to level 0 (an as-good-as-new condition), and without intervention, the turbine either fails or assumes some higher degradation level $f_n(x_n,\ell,1)$.

Next, we describe the setup and other costs associated with replacements in an offshore wind farm. If at least one unit is replaced, a setup cost c_s ($c_s \gt 0$), which comprises significant equipment and crew transfer costs, is incurred. For offshore wind farms, the distance between the base station and the shoreline is usually significant such that helicopters and SOVs and/or JUVs must be used to transport needed equipment and crews to the farm. Additionally, heavy equipment is required to perform replacements; hence, the setup cost c_s can be substantial (in excess of £250,000 per day of operation; see [Reference McMorland, Flannigan, Carroll, Collu, McMillan, Leithead and Coraddu42, Reference Shields, Marsh, Stefek, Oteri, Gould, Rouxel, Diaz, Molinero, Moser and Malvik64]). However, this cost can be shared if multiple turbines are replaced in the same period. The shared setup cost induces economic dependence between the turbines. Second, a per-replacement cost c_r ($c_r \gt 0$) is incurred for every turbine that is replaced. This cost compromises the procurement cost of a new component, as well as the labor hours needed to complete the replacement. Finally, a downtime cost is incurred for any turbine that either (1) fails in the time interval between two inspections or (2) is taken offline so that a replacement can be performed. This cost corresponds to lost power production due to turbine unavailability [Reference Maples, Saur, Hand, van de Pietermen and Obdam40]. As noted by Wan et al. [Reference Wan, Ela and Orwig72], power output is a function of wind speed and is typically characterized by a monotone increasing, S-shaped curve that is convex at low wind speeds, concave after an inflection point, and flat beyond an upper speed threshold [Reference Hwangbo, Johnson and Ding26]; therefore, we assume the downtime cost rate is a function of the environment state $\ell$. Let $c_d(\ell)$ denote the downtime cost per unit time per turbine when the environment is in state $\ell$. For example, if wind speed instantiates the environment, disjoint intervals of speed may correspond to different regions of the power output curve, as depicted in Figure 2(a). Three distinct wind speed ranges (low, medium, and high) result in distinct regions of power output.

Figure 2.

Partition of wind speed into disjoint intervals with distinct downtime cost rates. (a) A typical wind power curve. (b) Downtime cost rate versus environment.

Figure 2(b) illustrates how the wind speed intervals can be mapped to environment states in order to associate a downtime cost rate with each state. Wind speed can be mapped to power production using the well-known method of bins, as prescribed by the International Electrotechnical Commission [27]. In this specific illustration with three environment states, we divide the wind speed range into three intervals of equal length, with the lowest environment state (state 1) corresponding to wind speeds that exceed the cutoff speed; therefore, we assign a positive downtime cost rate for this state. We assume the downtime cost rate $c_d(\ell)$ is strictly increasing in $\ell\in \mathcal{L}$ and bounded from above. That is, $0 \lt c_d(1) \lt \cdots \lt c_d(L) \lt \infty$.

We now describe the degradation dynamics of a particular turbine $n\in \mathcal{N}$ with and without replacement. Figure 3(a) and 3(b) depicts scenarios in which the environment state is fixed at $\ell$ throughout the period. From Figure 3(a), we see that at decision epoch m, the turbine can either be failed (point A) or working (point B).

Figure 3.

Degradation evolution of a turbine within a period with and without replacement. (a) A turbine with replacement. (b) A turbine without replacement.

For either case, if replacement is prescribed at time m, the turbine is down for the entire period and is restored to a new condition at decision epoch m + 1 (point C); hence, the cost $c_d(\ell)+c_r$ is incurred in each of the two scenarios of Figure 3(a). Figure 3(b) depicts the same two scenarios in which the turbine is either failed (point A) or working (point B) at time m. Here, if replacement is not prescribed at decision epoch m and the unit is failed, it remains failed at m + 1 so that the incurred cost is $c_d(\ell)$. On the other hand, if the turbine is working at time m (point B) and replacement is not prescribed, two possible scenarios (represented by the dotted lines) may occur. First, the turbine continues degrading and reaches the critical threshold (ξ = 1) before decision epoch m + 1 (point D). In this case, the turbine is down from the crossing time (point D) until the start of the next period (point E). This downtime is denoted by $\tau_n(x_n,\ell)$ and is obtained via the predictive function $f_n(x_n,\ell,t)$. Specifically, let $\tau_n(x_n,\ell)$ denote the predicted within-period downtime if no replacement is performed ($a_n=0$), x_n is the degradation at the start of the period, and the environment state is $\ell$. Then,

(2)\begin{equation} \tau_n(x_n,\ell)=\max\left\{0,1-\inf\left\{t\ge 0: f_n(x_n,\ell,t)\ge 1\right\}\right\}. \end{equation}

For this first scenario, the downtime cost $c_d(\ell)\cdot \tau_n(x_n,\ell)$ is incurred for turbine n. For the second scenario, the turbine is working at time m (point B), but its level of degradation does not reach the failure threshold before decision epoch m + 1. Rather, it continues operating for the entire period and evolves to a higher level of degradation at decision epoch m + 1 (point F). In this case, $\tau_n(x_n,\ell)=0$ so that no downtime cost is incurred. Finally, we note that if the turbine is failed at decision epoch m (point A) and replacement is not prescribed, it remains failed at decision epoch m + 1 (point E). Since $f_n(1,\ell,t)=1$ for all $\ell\in \mathcal{L}$ and $t\ge 0$, it follows that $\tau_n(1,\ell)=1$. Therefore, the cost $c_d(\ell)\cdot 1=c_d(\ell)$ is incurred.

For notational convenience, define an indicator function $\mathbb{I}(\textbf{a})$, where $\mathbb{I}(\textbf{a})=1$ if $a_n=1$ for some $n\in \mathcal{N}$ (at least one replacement is performed) and $\mathbb{I}(\textbf{a})=0$ if $a_n=0$ for all $n\in \mathcal{N}$ (no replacements are performed). The one-step immediate cost is then

(3)\begin{align} c(\bf{x},\ell,\bf{a})= \mathbb{I}(\bf{a})c_s + \sum_{n\in \mathcal{N}}a_n\left[c_r +c_d(\ell)\right]+ \sum_{n\in \mathcal{N}}(1-a_n)\,c_d(\ell)\,\tau_n(x_n,\ell). \end{align}

The first term on the RHS of Equation (3) represents the setup cost. The second term is the sum of replacement costs and power production losses incurred for all replaced turbines. Finally, the third term accounts for the downtime costs estimated by the predictive functions for turbines that are not replaced.

All immediate costs are discounted at a fixed rate λ $(0 \lt \lambda \lt 1)$. The objective is to minimize the expected total discounted setup, replacement, and downtime costs over an infinite horizon. Let $\mathcal{P}$ denote the set of all nonanticipative replacement policies. Starting in state $(\textbf{x},\ell)$, the DM solves

(4)\begin{equation} V(\textbf{x},\ell)=\inf_{\pi\in \mathcal{P}}\left\{\mathbb{E}\left[\sum_{m=0}^{\infty}\lambda^m c(\textbf{x}_m,\ell_m,\textbf{a}_{\pi}(\textbf{x}_m,\ell_m)|(\textbf{x},\ell))\right]\right\},\qquad (\textbf{x},\ell)\in \Gamma, \end{equation}

where $\textbf{a}_{\pi}(\textbf{x}_m,\ell_m)$ denotes the action prescribed by policy $\pi\in \mathcal{P}$ in state $(\textbf{x}_m,\ell_m)$ at decision epoch m. Equation (4) is equivalent to

(5)\begin{equation} V(\textbf{x},\ell)= \min_{\textbf{a}\in\mathcal{A}}\left\{c(\textbf{x},\ell,\textbf{a})+\lambda \sum_{k\in \mathcal{L}}V(\hat{\textbf{x}}(\textbf{x},\ell,\textbf{a}),k)p_{\ell k}\right\},\qquad (\textbf{x},\ell)\in \Gamma, \end{equation}

where $\hat{\textbf{x}}(\textbf{x},\ell,\textbf{a})=\left({\hat{\textbf{x}}}_1(x_1,\ell,a_1),\ldots,{\hat{\textbf{x}}}_N(x_N,\ell,a_N)\right)$ is the N-dimensional vector of predicted degradation levels at the start of the next period. Note that the state space Γ is Borel-measurable, and the action space $\mathcal{A}$ is finite. Additionally, the immediate costs are strictly positive and bounded, and the problem is discounted. Therefore, by Theorems 6.2 and 6.3 of [Reference Puterman54], there exists an optimal, deterministic stationary replacement policy, and the value iteration (VI) algorithm converges to the optimal value. Next, define the optimal action in state $(\textbf{x},\ell)$ by

(6)\begin{equation} \textbf{a}^*(\textbf{x},\ell):=\min\left\{\textbf{a}^{\prime}\in \mathrm{argmin}_{\textbf{a} \in \mathcal{A}}\left\{c(\textbf{x},\ell,\textbf{a})+\lambda \sum_{k\in \mathcal{L}}V(\hat{\textbf{x}}(\textbf{x},\ell,\textbf{a}),k)p_{\ell k}\right\}\right\}, \qquad (\textbf{x},\ell) \in \Gamma. \end{equation}

That is, if there exist multiple optimal actions in state $(\textbf{x},\ell)$, we take the minimum action; hence, Equation (6) has a unique solution. In Section 3, we establish structural properties and provide some practical insights into this important operational problem.

3.1. Value function properties

We now examine important properties of the value function $V(\textbf{x},\ell)$. Proposition 3.5 posits that the monotonicity and discrete concavity of Lemma 3.3 extend to the value function if the transition matrix P possesses the increasing failure rate (IFR) property.

Definition 3.4. Let $\mathbf{P}=[p_{\ell k}]$ be the one-step TPM of a DTMC with state space $\mathcal{L}=\{1,\ldots,L\}$. Then P is said to be IFR if

\begin{equation*}\eta_k(\ell):=\sum_{\ell'=k}^{L}p_{\ell \ell^{\prime}}\end{equation*}

is nondecreasing in $\ell\in \mathcal{L}$ for each $k\in \mathcal{L}$.

Definition 3.4 implies that, starting from a higher environment state, the DTMC is more likely to transition to even higher states. Next, we state the main result related to $V(\textbf{x},\ell)$.

Proposition 3.5(a) asserts that if P is IFR, then the cost starting from state $(\textbf{x},\ell)$ is nondecreasing over the entire state space. That is, for a fixed environment state $\ell$, the cost function is monotone nondecreasing in degradation x, and for a fixed level of degradation x, the cost is monotone nondecreasing in $\ell$. However, the costs do not increase without bound. Proposition 3.5(b) asserts that the marginal costs diminish as the environment state increases, a property that follows directly from the discrete concavity of the downtime cost rate function $c_d(\ell)$ on $\mathcal{L}$.

Naturally, one would like to examine the sensitivity of the value function to changes in the model’s cost parameters. For example, how sensitive is the value function to an increase in the setup cost c_s in light of the fact that this substantial cost can be shared by performing group replacements? To answer this question, we employ the notion of comparative statics, as described by Milgrom and Shannon [Reference Milgrom and Shannon43]. Let w be any one of the nonnegative cost parameters c_s, c_r, or $c_d(\ell)$. The functions $c(\textbf{x},\ell,\textbf{a},w)$ and $V(\textbf{x},\ell,w)$ denote, respectively, the immediate cost and value functions associated with parameter w. Proposition 3.6 asserts monotonicity in all three cost parameters.

Consider a case in which the DM can reduce the setup cost by decreasing the cost of preparing the JUV and helicopter. Proposition 3.6 asserts that optimally replacing the turbines incurs less cost in this case, that is, $V(\textbf{x},\ell,c_s)\le V(\textbf{x},\ell,c^{\prime}_s)$ for each $(\textbf{x},\ell)\in \Gamma$ and any $c_s\le c^{\prime}_s$. Next, we provide useful structural properties of the optimal wind farm replacement policy.

3.2. Characterizing the optimal replacement policy

Here, we present the main results of this section by establishing structural properties of the optimal replacement policy for a wind farm. For notational convenience, let

\begin{equation*}\mathcal{V}(\textbf{x},\ell):=\{n\in \mathcal{N}:a^*_n(\textbf{x},\ell) = 1\},\end{equation*}

be the set of all turbines for which replacement is optimal in state $(\textbf{x},\ell)$. Similarly, let $\mathcal{N}\setminus \mathcal{V}(\textbf{x},\ell)$ be the set of turbines for which it is optimal to do nothing in state $(\textbf{x},\ell)$.

Theorem 3.7 asserts that, for a given environment state $\ell$, the replacement region extends in each dimension for which it is optimal to replace in degradation level x. The result is elucidated by way of a simple example. Consider a wind farm comprising three turbines (N = 3), and suppose that in state $(\textbf{x},\ell)=(0.5,0.5,0.5,\ell)$, it is optimal to replace turbines 1 and 2 but not to replace turbine 3, that is, $\textbf{a}^*(\textbf{x},\ell)=(1,1,0)$. Figure 4(a) illustrates that these same actions are optimal for any state $(\textbf{y},\ell)$ in the shaded region (extension of the optimal actions in two dimensions).

Figure 4.

Extension of optimal actions at degradation level x to higher degradation levels. (a) $\textbf{a}^*(\textbf{x},\ell)=(1,1,0)$ in shaded region. (b) $\textbf{a}^*(\textbf{x},\ell)=(1,1,1)$ in shaded region.

That is, $\textbf{a}^*(\textbf{y},\ell)=(1,1,0)$ for any degradation level $\textbf{y}\in [0.5,1]\times[0.5,1]\times \{0.5\}$. Figure 4(b) illustrates a different scenario in which the state is again $(\textbf{x},\ell)=(0.5,0.5,0.5,\ell)$, but it is optimal to replace all three turbines, that is, $\textbf{a}^*(\textbf{x},\ell)=(1,1,1)$. For this case, we see that action $(1,1,1)$ is optimal for any degradation level $\textbf{y}\in [0.5,1]\times[0.5,1]\times [0.5,1]$, so the optimal actions extend in three dimensions to states with higher levels of degradation.

Now for a given environment state $\ell$, consider a single turbine $n^{\prime}\in \mathcal{N}$ with degradation level $x_{n^{\prime}}$ and fix the degradation levels of all other turbines $n\in \mathcal{N}\setminus \{n^{\prime}\}$. The following corollary characterizes the behavior of the optimal action $a^*_{n^{\prime}}(\textbf{x},\ell)$ for each environment state.

As a consequence of economic dependence, it is possible that replacement of turbine n ^ʹ is not optimal for any level of degradation, that is, a degradation-based threshold policy does not exist. However, if reactive replacement is required whenever unit n ^ʹ is found to be failed, Corollary 3.8 implies the existence of a degradation threshold $x^*_{n^{\prime}}$ such that

(7)\begin{equation} a^*_{n^{\prime}}(\bf{x},\ell)= \begin{cases} 1, & \text{if}\, x_{n^{\prime}}\ge x^*_{n^{\prime}},\\ 0, & \text{if}\, x_{n^{\prime}} \lt x^*_{n^{\prime}}. \end{cases} \end{equation}

Equation (7) mirrors the optimal replacement policy described in Theorem 2 of [Reference Ulukuş, Kharoufeh and Maillart71] for a single degrading system that is influenced by its random environment; however, their model assumes that an immediate, reactive replacement is performed whenever a failure is revealed. Our model shows that, for some turbines, delaying replacement may be optimal due to the economic dependence induced by the shared setup cost (c_s) and the turbines’ stochastic dependence induced by their common environment. That is, the threshold for replacing a turbine may depend not only that turbine’s degradation but also on the degradation levels of the other turbines in the wind farm. This interesting feature of our model is more transparent in the case of N = 2 turbines.

3.3. Special case: N = 2

Structural properties of the optimal replacement policy over the entire space Γ are not immediately obvious when a wind farm contains three or more turbines. Fortunately, Theorem 3.7 can be used to establish the dependence of an optimal replacement threshold of a particular turbine on the degradation of the other turbine in a two-turbine wind farm (N = 2). Before formalizing this interesting result, it is helpful to consider a graphical depiction of the two-turbine case in Figure 5. For $\ell$ fixed, the black circles in Figure 5 represent degradation levels $(x_1,x_2)\in [0,1]\times [0,1]$. From Theorem 3.7, we can deduce the following. If the optimal action at point A is $(0,1)$, then $(0,1)$ remains optimal for any point along the line segment AD. Specifically, for a fixed degradation level x ₁, if it is optimal to replace turbine 2 at A, then it is optimal to replace turbine 2 for each point along AD. Similarly, if the optimal action at point B is $(1,0)$ (replace turbine 1 but not turbine 2), then $(1,0)$ remains optimal for any point along the line segment BE. Finally, at point C, the optimal action is $(1,1)$, replace both turbines. Then it is also optimal to replace both turbines for every point contained in the shaded region.

Figure 5.

Extension of the optimal actions for two turbines in a fixed environment state.

Next, we seek to provide structure to replacement regions over the degradation space $\mathcal{X}$. For a fixed environment state $\ell\in \mathcal{L}$, let $x^*_2(x_1,\ell)$ denote the optimal replacement threshold for turbine 2 when the degradation of turbine 1 is $x_1\in [0,1]$. Similarly, let $x^*_1(x_2,\ell)$ be the optimal replacement threshold for turbine 1 when the degradation of turbine 2 is $x_2\in [0,1]$. When they exist, let

(8)\begin{align} b_1(x_2,\ell)&:=\inf\{x_1\ge 0:\textbf{a}^*(x_1,x_2,\ell)=(1,1)\}, \end{align}

(9)\begin{align} b_2(x_1,\ell)&:=\inf\{x_2\ge 0:\textbf{a}^*(x_1,x_2,\ell)=(1,1)\}. \end{align}

That is, $b_1(x_2,\ell)$ ($b_2(x_1,\ell)$) is the minimum level of turbine 1 (turbine 2) degradation that makes group replacement optimal when the degradation of turbine 2 (turbine 1) is x ₂ (x ₁). The next proposition asserts that the optimal replacement threshold of one turbine is monotone decreasing in the degradation level of the other turbine beyond these special points $b_1(1,\ell)$ and $b_2(1,\ell)$.

Proposition 3.9 establishes the fact that the replacement threshold of one turbine depends on the degradation state of the other turbine. Figure 6 depicts the region in which it is optimal to replace turbine 2, that is, $\left\{(x_1,x_2):\textbf{a}^*(x_1,x_2,\ell)\in \{(0,1),(1,1)\}\right\}$. The figure illustrates that the replacement threshold for turbine 2 is monotone decreasing in x ₁ for any $x_1\ge b_1(1,\ell)$.

Figure 6.

Monotone threshold policy for turbine 2 in a wind farm with two turbines.

Figure 6 also highlights the fact that it is advisable for the DM to decrease the optimal threshold of turbine 2 as turbine 1 reaches higher degradation levels. The results in Proposition 3.9 confirm that the replacement thresholds for both turbines are monotone decreasing. In Section 4, we provide some numerical examples illustrating the effects of economic and stochastic dependence on the optimal turbine replacement policies.

4.1. A baseline example

Suppose there are two turbines in an offshore wind farm (N = 2), and the turbines are exposed to an environment whose evolution is modeled by a DTMC with state space $\mathcal{L}=\{1,\ldots,6\}$ and TPM

\begin{equation*}\mathbf{P}= \begin{bmatrix} 0.40 & 0.20 & 0.20 & 0.14 & 0.05 & 0.01\\ 0.22 & 0.30 & 0.20 & 0.15 & 0.10 & 0.03\\ 0.20 & 0.14 & 0.30 & 0.22 & 0.11 & 0.03\\ 0.10 & 0.10 & 0.20 & 0.35 & 0.15 & 0.10\\ 0.05 & 0.08 & 0.10 & 0.20 & 0.35 & 0.22\\ 0.03 & 0.05 & 0.10 & 0.15 & 0.40 & 0.27\\ \end{bmatrix}.\end{equation*}

It can be shown that P possesses the IFR property. For this illustration, it is assumed that, for each turbine $n\in \mathcal{N}$, the DM predicts future degradation using the linear predictive function

\begin{equation*}f_n(x_n,\ell,t) = x_n + r_n(\ell)t,\quad t\ge 0,\end{equation*}

where $r_n(\ell)$ is the degradation rate of turbine n in environment state $\ell\in \mathcal{L}$. The form of f_n is immaterial as long as it satisfies the nonnegativity and monotonicity assumptions stated in Section 2. It is difficult to obtain proprietary wind farm O&M records to parameterize our model. Therefore, notional setup costs, downtime cost rates, and degradation rates were chosen relative in scale but are not intended to reflect any real data provided to the authors. The linear degradation rates were chosen such that $r_n(i) \lt r_n(j)$ whenever i < j, and their magnitudes are in proportion to the failure threshold ξ = 1. Because the time scale of one period is on the order of 1 week, the rates of degradation are relatively small as compared to the failure threshold of 1. Similarly, the downtime cost rates are selected such that $c_d(i) \lt c_d(j)$ whenever i < j. The setup and replacement costs are $c_s=5$ and $c_r=8$, respectively. Table 1 lists all other notional parameter values.

Table 1.

Downtime cost and degradation rates for each environment state.

To implement the VI algorithm of [Reference Puterman54], we discretized the continuous degradation space $\mathcal{X}$ (cf. Kushner and Dupuis [Reference Kushner and Dupuis37]). Specifically, for each $n\in \mathcal{N}=\{1,2\}$, the degradation interval $[0,1]$ was uniformly discretized into 101 states so that $x_n\in \{0.00,0.01,\ldots,0.99,1.00\}$, and we let $\mathcal{X}^{\prime}:=\{0.00,0.01,\ldots,0.99,1.00\}^2$. Then for each $(\textbf{x},\ell)$ in $\mathcal{X}^{\prime}\times \mathcal{L}$, let $v^k(\textbf{x},\ell)$ denote the kth iterate of the VI algorithm with $v^0(\textbf{x},\ell)=0$. The algorithm terminates at iteration k $(k\in \mathbb{N}$) if

\begin{equation*}\big\|v^{k} -v^{k-1}\big\|_{\infty} = \max_{(\textbf{x},\ell)}\left\{\big|v^{k}(\textbf{x},\ell)-v^{k-1}(\textbf{x},\ell)\big|\right\}\le \epsilon_v,\end{equation*}

where $\epsilon_v \gt 0$ is a tolerance level. For each implementation of the VI algorithm, we set $\epsilon_v = 0.01$ and λ = 0.99. The VI algorithm was coded in Python 3.9 and executed on a personal computer with a 3.40 GHz processor and 8 GB of RAM. Next, we examine the sensitivity of the optimal replacement policy to each environment state and the setup cost.

4.2. Baseline example sensitivity

Starting in state $(x_1,x_2,\ell)=(0,0,1)$, the VI algorithm yields an optimal total discounted cost of 3,044.20 for the baseline example. For a fixed environment state, replacement regions are the regions in which at least one turbine is replaced (shaded regions). The opportunistic region is the region in which both turbines are replaced (dark shaded region). Figure 7 depicts the replacement regions in different environment states.

Figure 7.

Baseline example optimal replacement policy for each environment state ($c_s=5$): (a) environment state $\ell=1$, (b) environment state $\ell=2$, (c) environment state $\ell=3$, (d) environment state $\ell=4$, (e) environment state $\ell=5$, and (f) environment state $\ell=6$.

Figure 7(b)–(f) shows that higher environment states lead to larger replacement regions, as these states imply accelerated degradation and higher power production rates. Hence, in higher environment states, the DM opts to lower the replacement threshold to avoid high production losses caused by turbine failures. However, in Figure 7(a), the largest replacement region is for environment state $\ell=1$ since the production and degradation rates for this state are least among all environment states; hence, the DM prefers to perform the preventive replacements much sooner. Such a scenario may correspond to the overhaul of large, critical components in a wind farm, as major replacements are typically performed in the summer months when environment conditions are milder. Figure 7(a) also illustrates the result in Proposition 3.9—the optimal replacement threshold of one turbine is decreasing in the degradation state of the other turbine when both are in the opportunistic region. However, we also observe nonintuitive behavior for the optimal replacement policy in Figure 7; the optimal replacement threshold of each turbine is not necessarily decreasing over the entire degradation state space. For example, in Figure 7(a), for a fixed degradation state of turbine 2 ($x_2=0.4$), the DM opts to replace turbine 2 at point B but not to replace at points A and C. Thus, by increasing the degradation state of turbine 1, the optimal replacement threshold for turbine 2 decreases first (moving from point A to B) and then increases (moving from point B to C). One possible explanation for this behavior is that, at point C, turbine 2 is closer to the opportunistic region and will reach this region sooner. Thus, it is best for the DM to postpone replacement of turbine 2 until turbine 1 further degrades so that a group replacement can be performed and setup costs can be shared.

In the first instance of the baseline example, the setup cost ($c_s=5$) was relatively low compared to the downtime cost rates. This scenario corresponds to the case when the required resources for replacement are available (e.g., no cost for chartering a JUV and helicopter) and the majority of the setup cost is fueling. For the sake of comparison, we also examined a scenario with a relatively high setup cost corresponding to the case when a helicopter and JUV must be chartered from a spot market, and the total setup cost exceeds the highest capacity of power production for a turbine ($c_s = 80$). Figure 8 shows how the optimal replacement regions shrink with a substantial increase in the setup cost. Figure 8(c)–(f) shows that for some degradation levels, the DM may allow the other turbine to remain failed and postpone replacement. That is, for higher states of degradation, replacing only one turbine is not optimal, and the DM is more inclined to perform group replacement due to the magnitude of the setup cost.

Figure 8.

Baseline example optimal replacement policy for each environment state ($c_s=80$): (a) environment state $\ell=1$, (b) environment state $\ell=2$, (c) environment state $\ell=3$, (d) environment state $\ell=4$, (e) environment state $\ell=5$, and (f) environment state $\ell=6$.

One would expect the opportunistic region to be larger in higher environment states since the DM opts to perform the group replacement sooner to avoid higher power production losses. However, in Figures 7 and 8, the opportunistic region does not necessarily grow with the environment state. There are two reasons for this nonintuitive behavior. First, if the environment state is 1, then the likelihood of transitioning to higher states is low. Hence, it is expected that, in subsequent periods, the environment remains in state 1 with a higher probability. Second, the downtime cost in environment state 1 is low; hence, the DM prefers to replace both turbines in an environment state that leads to a lower production loss. In the next subsection, we show how models with incorrect assumptions impact the optimal replacement policies.

4.3. Value of correct modeling

Here, our aim is to assess the magnitude of cost increases resulting from incorrect model assumptions. The MDP model incorporates economic dependence via the setup cost c_s and stochastic dependence via the association of degradation processes to the turbines’ common environment. Here, we compare the expected total discounted costs obtained by two distinct models and compare those with the costs obtained by the model of Section 2. In the first model, the DM simply ignores both types of dependence and assumes that replacement decisions are made for each turbine independently. In the second model, the DM includes dependence but assumes the environment can be described by a simple two-state DTMC, thereby ignoring any intermediate environment states.

Let $\pi^*$ be the optimal replacement policy obtained by the MDP model of Section 2, and let π_i be the optimal replacement policy obtained by Model i, for $i=1,2$. Table 2 summarizes how the models differ with regard to their underlying assumptions.

Table 2.

Summary of the three distinct models for the replacement problem.

The full model incorporates both economic and stochastic dependence and all six environment states. After obtaining the optimal policy for each model using the VI algorithm, we simulated the respective costs of the modified policies (π ₁ and π ₂) on the full model using a large number of sample paths. Along each sample path, the total discounted cost was computed, and these values were compared. The simulation run length, which corresponds to the number of decision epochs, is denoted by M ($M\in \mathbb{N})$. Since the expected one-step costs are bounded, and the cost function is discounted, the simulation run length can be determined a priori to ensure that the total discounted cost is accurate to a fixed constant. The simulation run length M was chosen such that $M\ge \left[\ln((1-\lambda)\epsilon_s/C)/\ln(\lambda)\right]-1$, where C is any valid upper bound on the expected one-step costs and λ is the discount factor. For all numerical examples, M was chosen to correspond to $\epsilon_s = 0.01$, λ = 0.99, and $C=c_s+N[c_r+c_d(L)]$. For the baseline example, $M\ge 1428$, so we chose M = 1500.

To compute the average simulated total discounted cost under each policy, we selected the number of replications (R) to achieve a desired margin of error using the well-known relationship

(10)\begin{equation} R = \left(z_{\alpha/2}\,\hat{\sigma}/\Delta\right)^2, \end{equation}

where $z_{\alpha/2}$ is the critical z-value above which lies area $\alpha/2$ under the standard normal density function, $\hat{\sigma}$ is the estimated standard deviation, and Δ is the desired margin of error. For each of the following models, we chose a 95% confidence level (α = 0.05, $z_{\alpha/2}=1.96$) and $\Delta=1$. The simulation models were coded in the MATLAB R2022a computing environment and executed on a personal computer with an 8-core CPU and 8 GB of RAM. We next describe the modified MDP formulations for Models 1 and 2, along with the corresponding numerical results.

Model 1. The first modified model ignores both economic and stochastic dependence; therefore, the problem can be decomposed into two independent, single-turbine problems. For each decomposed problem, the setup and replacement costs are assessed every time a replacement is performed. Although both turbines are assumed to operate in environments governed by the same transition matrix P, their environments (and degradation processes) evolve independently. Thus, for turbine $n\in \{1,2\}$, the state is $(x_n,\ell_n)\in [0,1]\times \mathcal{L}$, $a_n\in \{0,1\}$ is the action, and the immediate cost function is

\begin{equation*}c(x_n,\ell_n,a_n) = a_n\left[c_s +c_r +c_d(\ell_n)\right]+(1-a_n)c_d(\ell_n)\tau_n(x_n,\ell_n).\end{equation*}

The Bellman equation for the decomposed MDP model n is

\begin{equation*}V(x_n,\ell_n)=\min_{a_n\in \{0,1\}}\left\{c(x_n,\ell_n,a_n)+\lambda \sum_{k_n\in \mathcal{L}}V({\hat{\textbf{x}}}_n(\ell_n,a_n),k_n)p_{\ell_n k_n}\right\},\qquad (x_n,\ell_n)\in [0,1]\times\mathcal{L},\end{equation*}

where

\begin{equation*}{\hat{\textbf{x}}}_n(\ell_n,a_n)= \begin{cases} \min\{1,f_n(x_n,\ell_n,1)\}, & \text{if}\ a_n=0,\\ 0, & \text{if}\ a_n=1. \end{cases}\end{equation*}

Figure 9 illustrates the behavior of the Model 1 optimal replacement policy (π ₁) for each environment state. We observe that the optimal replacement threshold of one turbine does not depend on the other’s degradation state in the absence of economic and stochastic dependence. Moreover, the optimal replacement thresholds decrease with the environment state. That is, in environments that induce higher degradation and power production rates, preventive replacements are prescribed at lower levels of degradation for both turbines.

Next, we seek to assess how the Model 1 optimal policy (π ₁) performs under the assumptions of the full model (i.e., when economic and stochastic dependence are present). To this end, we apply policy π ₁ to a simulation of the full model and compare its resulting simulated total discounted cost. We first performed 5,000 replications of the simulation model (each with $M=1,500$ decision epochs) and obtained a sample standard deviation $\hat{\sigma}\approx256$. By Equation (10), the number of replications needed to ensure $\Delta=1$ is $R=251,763$; hence, we selected $R=252,000$. The policy π ₁ was applied in the simulation of the full model, and the simulated total discounted costs for three different initial states are reported in Table 4. The simulated costs are also compared with those obtained using VI on the full model, which includes both economic and stochastic dependence.

Figure 9.

Baseline example optimal replacement policy for each environment state ($c_s=80$): (a) environment state $\ell=1$, (b) environment state $\ell=2$, (c) environment state $\ell=3$, (d) environment state $\ell=4$, (e) environment state $\ell=5$, and (f) environment state $\ell=6$.

Model 2. Like the full model, this second modification incorporates both economic and stochastic dependence (via shared setup costs and a common environment). However, in this case, the DM lacks sufficient information to assess the number of environment states and the degradation rates associated with each state. Recall that the full model assumes that the environment can transition among six distinct states. In Model 2, the DM uses only two states to describe the environment ($\mathcal{L} = \{1,2\}$) and estimates a degradation rate for each state. Table 3 summarizes the downtime cost rates and degradation rates corresponding to each environment state. Note that the values used for states 1 and 2 in Model 2 correspond to those of the extreme states 1 and 6 from Table 1. For this new two-state model, the DM scales the original transition probabilities, p ₁₁, p ₁₆, p ₆₁, and p ₆₆, to construct the new TPM

\begin{equation*}\widehat{\mathbf{P}}=\begin{bmatrix} 0.98 & 0.02\\ 0.10 & 0.90\\ \end{bmatrix},\end{equation*}

Table 3.

Downtime cost and degradation rates for the environment states of Model 2.

where, for example, $\hat{p}_{11}$ was obtained by $\hat{p}_{11}=p_{11}/(p_{11}+p_{16})\approx 0.98$. All other components of this MDP model, including the immediate cost function and the Bellman equation, are identical to the MDP model defined in Section 2. Figure 10 depicts the optimal replacement policy (π ₂) obtained by VI algorithm for each of the two environment states. To compare the costs of this policy to the full model, it was necessary to create a mapping between each of the six states in the full model to the two states of the modified environment process. Specifically, the replacement policy for $\ell=1$ (Figure 10(a)) was used for states 1, 2, and 3 in the full model. The replacement policy for $\ell=2$ (Figure 10(b)) was used for states 4, 5, and 6 in the full model.

Figure 10.

Model 2 optimal replacement policy for each environment state. (a) Environment state $\ell=1$. (b) Environment state $\ell=2$.

We first performed 5,000 replications of the simulation model (each with $M=1,500$ decision epochs) and obtained a sample standard deviation $\hat{\sigma}\approx 251$. By Equation (10), the number of replications needed to ensure $\Delta=1$ is $R=242,024$; hence, we selected $R=243,000$. As before, we apply policy π ₂ to a simulation of the full model and compare its resulting simulated total discounted cost. The simulated total discounted costs for six different initial states are summarized in Table 4 and compared with those obtained using VI on the full model.

Table 4 compares the expected total discounted costs of the full model and the simulated total discounted costs of Models 1 and 2 starting from initial states $(0,0,1)$, $(0,0,2)$, $(0,0,3)$, $(0,0,4)$, $(0,0,5)$, and $(0,0,6)$. This table reveals several interesting results. First, the policies resulting from Model 1, in which both economic and stochastic dependence are ignored, yield average total discounted costs that are just over 8% higher than those generated by the full model, indicating the importance of sharing setup costs and consideration of how the turbines’ degradation processes are related via their shared environment. Second, we note that the policies resulting from Model 2, which includes both types of dependence but fails to adequately characterize the environment process, result in cost increases on the order of 5%. These results provide evidence that accounting for all possible environment scenarios can be important when assessing the lifetime replacement costs of an offshore wind turbine. Finally, we observe a general trend that the total costs are increasing in the initial environment state (assuming both turbines start in an as-good-as-new condition). This observation is in agreement with Proposition 3.5(a).

Table 4.

Total discounted costs for six distinct initial states.

4.4. Comparisons with heuristic policies

Here, we assess the performance of the optimal replacement policy obtained from our model with three distinct and common heuristic policies:

• • Reactive replacement policy. Using this policy, the DM replaces the component reactively, that is, immediately following a failure.

• • Age-replacement policy. The classical age-replacement policy prescribes replacement either at failure or at a chosen deterministic age T, whichever occurs first. The replacement age is taken to be the expected lifetime of the unit, which is typically the manufacturer’s mean time-to-failure.

• • Fixed interval policy. This policy performs a replacement every T units, whether a replacement is warranted or not. Here, the deterministic inter-replacement time T is taken to be the manufacturer’s mean time-to-failure.

We compared the total costs generated by the MDP policies with those of the heuristic policies starting from six distinct initial states (corresponding to the six environment states). The results in Table 5 show that the total discounted costs obtained by the MDP model are approximately 17% lower than those obtained using reactive replacement, 14% lower than those obtained using the age-replacement policy, and over 40% lower than those obtained using a fixed interval policy. These results highlight the value of using the optimal policy as compared to simple heuristic policies.

Table 5.

Total discounted cost comparison: Optimal policy versus three heuristic policies.