Joint optimization of ordering and maintenance with condition monitoring data

Abstract

We study a single-unit deteriorating system under condition monitoring for which collected signals are only stochastically related to the actual level of degradation. Failure replacement is costlier than preventive replacement and there is a delay (lead time) between the initiation of the maintenance setup and the actual maintenance, which is closely related to the process of spare parts inventory and/or maintenance setup activities. We develop a dynamic control policy with a two-dimensional decision space, referred to as a warning-replacement policy, which jointly optimizes the replacement time and replacement setup initiation point (maintenance ordering time) using online condition monitoring data. The optimization criterion is the long-run expected average cost per unit of operation time. We develop the optimal structure of such a dynamic policy using a partially observable semi-Markov decision process and provide some important results with respect to optimality and monotone properties of the optimal policy. We also discuss how to find the optimal values of observation/inspection interval and lead time using historical condition monitoring data. Illustrative numerical examples are provided to show thatour joint policy outperforms conventional suboptimal policies commonly used in theliterature.

Proofs of lemmas and theorems

In this “Appendix”, the proofs of Remark 1, Lemmas 1, 2, 3,4, 5, Theorems 2, 3, 4, and Remark 2 are given.

Proof

(of Remark 1) The lower bound is the expected average cost of an ideal policy under which (i) there is no failure replacement, (ii) there is no early or late warnings, and (iii) the replacement time is as close as possible to the failure time (one decision epoch before failure. Under such a policy, the system pays only for the mandatory replacement cost $c_r$ and the expected replacement cycle is ${\mathbb {E}(\zeta )}$. Achieving this ideal lower bound is very hard in practice due to the stochastic nature of degradation and observation process and not guaranteed, as it requires a 100% perfect policy. The upper bound is calculated based on the fact that if an optimal policy exists, it should be at least better than (or equivalent to) the two trivial policies, (i) warning and replacement at failure and (ii) warning at time 0 and replace at failure. Therefore the cost associated with an optimal policy should be larger than or equal to the minimum of the expected costs of these two policies. In the first case, we have to pay for the cost of replacement, the cost of failure, and the cost of warning ($c_s\times l$) while the duration of a cycle is qual to $\mathbb {E}(\zeta )$. For the 2nd case, we have to pay for replacement and the cost of failure the same as case 1, however, since we issue warning at time zero, the expected cost of ordering becomes

$$\begin{aligned} \mathbb {E}\left( \mathscr {C}(\zeta ,0)\right) =\int \limits _{0}^{l}c_s(l-x)\Pr (\zeta =x) \mathrm {d}x+\int \limits _{l}^{\infty }c_h(x-l)\Pr (\zeta =x)\mathrm {d}x. \end{aligned}$$

Therefore, the optimal average cost is bounded, which completes the proof. $\square$

Before providing the proofs of the theorems, we introduce some important lemmas. Using these lemmas, we provide important results with regards to the form of decision controls, which can be used to compare two actions.

Lemma 1

For $k<d+l$, the following term is nondecreasing in $\varvec{\theta }$ and k:

$$\begin{aligned} \int _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,d\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }) \, +[\mathscr {C}_l(k\delta +\delta ,d\delta )]R(k\delta +\delta |k\delta ,{\varvec{\theta }})-\mathscr {C}_l(k\delta +\delta ,d\delta ). \end{aligned}$$

Lemma 2

The following holds true for all $k,d \in \mathbb {N}_0, d \le k$, and $\varvec{\theta }\in \varTheta$:

$$\begin{aligned} \bigg [V_\gamma ( k+1,\varvec{\theta },d,1) - \mathscr {C}_l(k\delta +\delta ,d\delta ) \bigg ] \ge \bigg [ V_\gamma ( k,\varvec{\theta },d,1 )-\mathscr {C}_l(k\delta ,d\delta ) \bigg ]. \end{aligned}$$

Lemma 3

If $\varvec{\theta }_1 \le _{st} \varvec{\theta }_2$, where $\le _{st}$ means stochastically increasing (see Ohnishi et al. 1994), then the following holds true for all $k,d \in \mathbb {N}_0, d \le k$, and $\varvec{\theta }_1,\varvec{\theta }_2 \in \varTheta :$

$$\begin{aligned} V_\gamma ( k,\varvec{\theta }_2,d,1 ) \ge V_\gamma ( k,\varvec{\theta }_1,d,1 ). \end{aligned}$$

Lemma 4

The value function $V_\gamma ( k,\varvec{\theta },\infty ,1 )$ is nondecreasing in $(k,\varvec{\theta })$ for any $k\in \mathbb {N}_0$, and $\varvec{\theta }\in \varTheta$.

Lemma 5

The following holds true for all $k,d \in \mathbb {N}_0, d \le k$, and $\varvec{\theta }\in \varTheta$:

$$\begin{aligned} \bigg [V_\gamma ( k+1,\varvec{\theta },k,1) - \mathscr {C}_l(k\delta +\delta ,k\delta ) \bigg ] \ge \bigg [ V_\gamma (k,\varvec{\theta },\infty ,1 )-\mathscr {C}_l(k\delta ,k\delta ) \bigg ]. \end{aligned}$$

Proof

(of Lemma 1) This lemma also states that the given term is approximately constant in k and $\varvec{\theta }$ for $k>d+l$. By simplifying $\mathscr {C}_l(k\delta ,d\delta )$ and $\mathscr {C}_l(k\delta +\delta ,d\delta )$ for $k<d+l$, the term in the lemma becomes

$$\begin{aligned} -c_s \delta \int _{k\delta }^{k\delta +\delta } {R}(\mathrm {d}x|k\delta ,\varvec{\theta }) \, -c_s k R(k\delta +\delta |k\delta ,{\varvec{\theta }})+c_s(k+1)\delta , \end{aligned}$$

which is nondecreasing in $\varvec{\theta }$ and k. We can also show that for small $\delta$ and $k>d+l$, if we assume that we issue warning at the decision epochs only, we have

$$\begin{aligned}&\int _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,d\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }) \approx \mathscr {C}_l(k\delta +\delta ,d\delta )\bar{R}(k\delta +\delta |k\delta ,\varvec{\theta }), \text {and thus}\\&\quad \approx \mathscr {C}_l(k\delta +\delta ,d\delta )\bar{R}(k\delta +\delta |k\delta ,\varvec{\theta })\, +[\mathscr {C}_l(k\delta +\delta ,d\delta )]R(k\delta \\&\quad +\delta |k\delta ,{\varvec{\theta }})-\mathscr {C}_l(k\delta +\delta ,d\delta )=0. \end{aligned}$$

$\square$

Proof

(of Lemma 2) Since the minimum of two nondecreasing functions is nondecreasing, we prove this lemma separately for $V^3_\gamma$ and $V^4_\gamma$. It follows from (17) that

$$\begin{aligned} V^4_\gamma ( k +1,{\varvec{\theta }}, d, 1 )- V^4_\gamma ( k ,\varvec{\theta }, d, 1 )=\mathscr {C}_l((k+1)\delta ,d)-\mathscr {C}_l(k\delta ,d\delta ), \end{aligned}$$

which satisfies Lemma 2 if $V_\gamma ( k,\varvec{\theta }, d, 1 )=V^4_\gamma ( k ,\varvec{\theta }, d, 1 )$. Now, it is sufficient to prove that

$$\begin{aligned} \left[ V^3_\gamma ( k +1,{\varvec{\theta }}, d, 1 ) - \mathscr {C}_l(k\delta +\delta ,d\delta ) \right] \ge \left[ V^3_\gamma ( k ,\varvec{\theta },d,1 ) - \mathscr {C}_l(k\delta ,d\delta )\right] . \end{aligned}$$

For a large $\hat{k}$ where $R(\hat{k}\delta +\delta |\hat{k}\delta ,{\varvec{\theta }}) \approx 0$ , that is when we definitely impose a preventive replacement, we have

$$\begin{aligned} V^3_\gamma ( \hat{k}+1,{\varvec{\theta }}, d, 1 )&\approx c_r+\sum \limits _i c_f^i \theta _k^i \, \bar{R}(k\delta +\delta | k\delta ,i)+V_\gamma (\varvec{\pi }_0)+\mathscr {C}_l(\hat{k}\delta +\delta ,d\delta ) \\ V^4_\gamma ( \hat{k}+1,{\varvec{\theta }}, d, 1 )&\approx c_r+V_\gamma (\varvec{\pi }_0)+\mathscr {C}_l(\hat{k}\delta +\delta ,d\delta ). \end{aligned}$$

Therefore, $V_\gamma ^4( \hat{k},\varvec{\theta }, d, 1 )<V^3_\gamma ( \hat{k},\varvec{\theta },d,1)$ is always true for large k, which means $V_\gamma ( \hat{k},\varvec{\theta }, d, 1 )=V_\gamma ^4(\hat{k},{\theta }, d, 1 )$. This implies that Lemma 1 holds true for sufficiently large k. Now, by induction hypothesis, let us assume that $\left[ V_\gamma ( k+1,\varvec{\theta },d,1) - \mathscr {C}_l(k\delta +\delta ,d\delta ) \right]$ is nondecreasing for $k+1$ where $1<k+1<\hat{k}$. It follows by the definition of the value function that

$$\begin{aligned}&\Big ( V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m)- c_r-V_\gamma (\varvec{\pi }_0) - \mathscr {C}_l(k\delta +\delta ,d\delta ) \Big ) \le 0 \rightarrow \\&\quad \left( \sum \limits _{m=1}^M \Pr (y_{k+1}=m|\varvec{\theta },u_k^2)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),d,1)- c_r-V_\gamma (\varvec{\pi }_0)- \mathscr {C}_l(k\delta +\delta ,d\delta ) \right) \le 0, \end{aligned}$$

is also nondecreasing in k. By adding $\pm \, \mathscr {C}_l(k\delta +\delta ,d\delta ) R(k\delta +\delta |k\delta ,{\varvec{\theta }_k})$ to $V^3_\gamma ( k ,\varvec{\theta }_k,d,1 )$, we get

$$\begin{aligned} \begin{aligned}&V^3_\gamma ( k ,\varvec{\theta }_k,d,1 ) = \left[ c_r+V_\gamma (\varvec{\pi }_0)\right] \bar{R}(k\delta +\delta |k,\varvec{\theta }_k)+\sum \limits _i c_f^i \theta _k^i \, \bar{R}(k\delta +\delta | k\delta ,i)\\&\quad + \int \limits _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,d\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }_k) \, \, -g(\gamma )\bar{\tau }(\delta |k,\varvec{\theta }_k) \\&\quad + \left( \sum \limits _{m=1}^M \Pr (y_{k+1}=m|\varvec{\theta }_k,1)\times V( k+1,\hat{\varvec{\theta }}_{k+1}(m),d,1 ) \right) R(k\delta +\delta |k\delta ,{\varvec{\theta }}) \\&\quad \pm \mathscr {C}_l(k\delta +\delta ,d\delta ) R(k\delta +\delta |k\delta ,{\varvec{\theta }_k}) \\&\quad = \left( c_r+V_\gamma (\varvec{\pi }_0)+\right) \bar{R}(k\delta +\delta |k, \varvec{\theta }_k)+\sum \limits _i c_f^i \theta _k^i \, \bar{R}(k\delta +\delta | k\delta ,i)\\&\quad \quad + \int \limits _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,d\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }_k) \, \, -g(\gamma )\bar{\tau }(\delta |k,\varvec{\theta }_k) \\&\quad \quad + \underbrace{\left( \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_k,1)\times V( k+1,\hat{\varvec{\theta }}_{k+1}(m),d,1 )- c_r-V_\gamma (\varvec{\pi }_0)- \mathscr {C}_l(k\delta +\delta ,d\delta ) \right) }_{(*)}\\&\qquad \quad R(k\delta +\delta |k\delta ,{\varvec{\theta }_k}) \\&\quad \quad + \mathscr {C}_l(k\delta +\delta ,d\delta ) R(k\delta +\delta |k\delta ,{\varvec{\theta }_k}). \end{aligned} \end{aligned}$$

As $(*)$ is nondecreasing and negative, and $R(k\delta +\delta |k\delta ,{\varvec{\theta }})$ is nonincreasing and positive in k, we can conclude that $(*) \times R(k\delta +\delta |k\delta ,{\varvec{\theta }})$ is also nondecreasing in k. Recalling Lemma 1 and the fact that R and $\bar{\tau }$ are nondecreasing in k, we conclude that $V^3_\gamma ( k ,\varvec{\theta }_k,d,1)-\mathscr {C}_l(k\delta ,d\delta )$ is nondecreasing in k. This completes the proof. $\square$

Proof

(Proof of Lemma 3) It is clear that $V^4_\gamma ( k ,\varvec{\theta }, d, 1 )$ is constant wrt ${\varvec{\theta }}$ and therefore is nondecreasing in $\varvec{\theta }$. The rest of the proof is to show that $V^3_\gamma ( k ,\varvec{\theta },d,1 )$ is nondecreasing in $\varvec{\theta }$. Following the same steps as we did in Lemma 2 , we get

$$\begin{aligned} V^3_\gamma ( k ,\varvec{\theta }_2,d,1)-V^3_\gamma ( k ,\varvec{\theta }_1,d,1 )\ge & {} \left[ \int _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,d\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }) \right. \\&\left. +[\mathscr {C}_l(k\delta +\delta ,d\delta )]R(k\delta +\delta |k\delta ,{\varvec{\theta }}) \right] \Biggl |^{\varvec{\theta }_2}_{\varvec{\theta }_1} \ge 0. \end{aligned}$$

This completes the proof. This lemma implies that if the decision process starts from a state that is stochastically more deteriorated in the sense of the stochastic ordering, it will be incurred a higher or equal cost in future. $\square$

Proof

(of Lemma 4) We first show the proof for k. The proof simply includes verifying that all elements in the right-hand side of (12) ($V^0_\gamma ( k,\varvec{\theta },+ \infty ,1 )- V^2_\gamma (k,\varvec{\theta },+ \infty ,1 )$) are nondecreasing functions in k given the fact that the minimum of nondecreasing functions is also nondecreasing. From (13)–(15), we can show that for a sufficiently large $\hat{k}$ (where $R(k\delta +\delta |k\delta ,{\varvec{\theta }}) \approx 0)$, we have

$$\begin{aligned} V^i_\gamma (\hat{k},\varvec{\theta },\infty ,1) \approx \left\{ \begin{array}{lll} c_r+\sum \limits _i c_f^i \theta ^i \, \bar{R}(k\delta +\delta | k\delta ,i)+V_\gamma (\varvec{\pi }_0)+\mathscr {C}_l(\hat{k}\delta ,\hat{k}\delta ), &{}\quad i=0,1 \\ c_r+V_\gamma (\varvec{\pi }_0)+\mathscr {C}_l(\hat{k}\delta ,\hat{k}\delta ),&{}\quad i=2 \end{array}. \right. \end{aligned}$$

Therefore, the statement $V^2_\gamma ( \hat{k},\varvec{\theta },+ \infty ,1 )<\min \{V^0_\gamma ( \hat{k},\varvec{\theta },+ \infty ,1 ),V^1_\gamma (k,\varvec{\theta },+ \infty ,1 )\}$ is always true regardless of $\varvec{\theta }$. This implies that when the age of the device is very high in the sense that it is very likely to fail during the next observation interval (given that the warning has not been generated), the best decision is to replace immediately. It is also obvious that $[c_r+ c_sl+V_\gamma (\varvec{\pi }_0)]$ is constant in k and therefore $V^2_\gamma (k,\varvec{\theta },+ \infty ,1)$ is nondecreasing in k. Therefore $V_\gamma ( k,\varvec{\theta },\infty ,1)$ is constant and nondecreasing for sufficiently large k. The rest of the proof is to show that $V^0_\gamma ( k,\varvec{\theta },+ \infty ,1 )$ and $V^1_\gamma (k,\varvec{\theta },+ \infty ,1 )$ are both nondecreasing in k for all $k<\hat{k}$. Let us first assume by induction hypothesis that $V_\gamma ( k+1,{\varvec{\theta }},+ \infty ,1 )$ is nondecreasing in k. It follows by definition that

$$\begin{aligned} \left[ \sum \limits _{m=1}^M \Pr (y_{k+1}=m|\varvec{\theta },1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 ) \right] - \left[ c_r+c_sl+V_\gamma (\varvec{\pi }_0) \right] \le 0. \end{aligned}$$

Since $R(k\delta +\delta |k\delta ,{\varvec{\theta }})$ and $V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 )$ are nondecreasing in k, then

$$\begin{aligned} (**)= & {} \left[ \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta },1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 ) \right. \\&\left. -\,\left[ c_r+c_sl+V_\gamma (\varvec{\pi }_0) \right] \right] R(k\delta +\delta |k\delta ,{\varvec{\theta }}) , \end{aligned}$$

is also nondecreasing in k. Also, from (13), we know that

$$\begin{aligned} V^0_\gamma ( k,\varvec{\theta },\infty ,1)= & {} (**)+ c_r+V_\gamma (\varvec{\pi }_0)+ \sum \limits _i c_f^i \theta ^i \, \bar{R}(k\delta +\delta | k\delta ,i)\\&\quad -g(\gamma )\bar{\tau }(\delta |k,\varvec{\theta },1) + \mathscr {C}_l(k\delta ,k\delta ). \end{aligned}$$

Since all of the above elements are nondecreasing in k, therefore $V^0_\gamma ( k,\varvec{\theta },+ \infty ,1 )$ is nondecreasing in k. Implementing the same procedures as we did above and considering Lemma 2, which states that $V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k, 1 )-\mathscr {C}_l(k\delta +\delta ,k\delta )$ is nondecreasing in k, we have

$$\begin{aligned} (***)= & {} \left[ \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta },1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k, 1 )\right. \\&\left. -\,\left[ c_r+\mathscr {C}_l(k\delta +\delta ,k\delta )+V_\gamma (\varvec{\pi }_0) \right] \right] \le 0, \end{aligned}$$

which is also nondecreasing in k. From (13), we have

$$\begin{aligned} V^1_\gamma (k,\varvec{\theta },+ \infty ,1 )= & {} (***) R(k\delta +\delta |k\delta ,\varvec{\theta }_k)+ \sum \limits _i c_f^i \theta ^i \, \bar{R}(k\delta +\delta | k\delta ,i)-g(\gamma )\bar{\tau }(\delta |k,\varvec{\theta }) \\&+ \,\mathscr {C}_l(k\delta +\delta ,k\delta ) R(k\delta +\delta |k\delta ,{\varvec{\theta }}) + \int \limits _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,k\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }_k)\\&+\,c_r+V(\varvec{\pi }_0). \end{aligned}$$

As all of the above terms are nondecreasing in k, then $V^1_\gamma (k,\varvec{\theta },+ \infty ,1 )$ is also nondecreasing in k. Considering that $V^i_\gamma ( k,\varvec{\theta },+ \infty ,1 )$ is a nondecreasing function in k for $i \in \{0,1,2\}$, we can conclude that $V_\gamma ( k,\varvec{\theta },\infty ,1 )$ is also nondecreasing in k. This completes the proof for k. To prove that $V_\gamma ( k,\varvec{\theta },\infty ,1 )$ is nondecreasing in $\varvec{\theta }$, we can implement the same steps as we did for the case of k, except that we use Lemma 3 to show that $(***)$ and $(***)R(k\delta +\delta |k\delta ,\varvec{\theta }_k)$ are nondecreasing in $\varvec{\theta }$. We have removed the rest of the proof for its simplicity. This completes the proof. $\square$

Proof

(of Lemma 5) From (14) and (17), we have $V^1_\gamma (k,\varvec{\theta }_k,\infty ,1) = V^3_\gamma ( k ,\varvec{\theta }_k,k,1)$. From the results given in the proof of Lemma 2, we know that

$$\begin{aligned} V_\gamma ( k,\varvec{\theta }_k,\infty ,1) \le V^3_\gamma ( k ,\varvec{\theta }_k,k,1) \le V^3_\gamma ( k +1,\varvec{\theta }_k,k,1) - \left[ \mathscr {C}_l(k\delta +\delta ,k\delta )-\mathscr {C}_l(k\delta ,k\delta ) \right] . \end{aligned}$$

Recall from (15) that

$V_\gamma ( k,\varvec{\theta }_k,\infty ,1) \le c_r+\mathscr {C}_l(k\delta ,k\delta )+V_\gamma (\varvec{\pi }_0)=V^4_\gamma ( k1,\varvec{\theta }_k,k,1).$ By adding $\pm \big [\mathscr {C}_l(k\delta +\delta ,k\delta )-\mathscr {C}_l(k\delta ,k\delta ) \big ]$ to the right-hand side of the above inequality, we have

$$\begin{aligned} V_\gamma ( k,\varvec{\theta }_k,\infty ,1) \le V^4_\gamma ( k +1,\varvec{\theta }_k,k,1) - \left[ \mathscr {C}_l(k\delta +\delta ,k\delta )-\mathscr {C}_l(k\delta ,k\delta ) \right] . \end{aligned}$$

Recalling that $V_\gamma ( k+1,\varvec{\theta }_k,k,1)=\min \{V^3_\gamma ( k +1,\varvec{\theta }_k,k,1),V^4_\gamma ( k +1,\varvec{\theta }_k,k,1)\}$, we get

$$\begin{aligned} \left[ V_\gamma ( k+1,\varvec{\theta },k,1) - V_\gamma (k,\varvec{\theta },\infty ,1 ) \right] \ge \left[ \mathscr {C}_l(k\delta +\delta ,k\delta )-\mathscr {C}_l(k\delta ,k\delta ) \right] , \end{aligned}$$

which completes the proof. $\square$

Proof

(of Theorem 1) From (14)–(15), we have

$$\begin{aligned} V^1_\gamma (\varvec{\pi }_{k}) - V^2_\gamma \left( \varvec{\pi }_{k}\right)&= \sum \limits _i c_f^i \theta _k^i \, \bar{R}(k\delta +\delta | k\delta ,i)-\mathscr {C}_l(k\delta ,k\delta )\bar{R}(k\delta +\delta |k\delta ,{\varvec{\theta }_{k}})\\&\quad +\, \int \limits _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,k\delta )\bar{R}(\mathrm {d}x| k\delta ,\varvec{\theta }_k)- g(\gamma )\bar{\tau }(\delta | k,\varvec{\theta }_k) \\&\quad +\, \left[ \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_k,1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k, 1 )\right. \\&\quad \left. -\,V^2_\gamma (k,\varvec{\theta }_k,\infty ,1) \right] R(k\delta +\delta |k\delta ,\varvec{\theta }_k) . \end{aligned}$$

Replacing $V^2_\gamma (k+1,\varvec{\theta }_{k+1},\infty ,1 )$ by $V^4_\gamma ( k+1,\varvec{\theta }_{k+1},k,1)+ c_s$ from (19) and adding $\mathscr {C}_l(k\delta +\delta ,k\delta )-\mathscr {C}_l(k\delta ,k\delta )$, we get

$$\begin{aligned} V^1_\gamma (\varvec{\pi }_{k})-V^2_\gamma (\varvec{\pi }_{k})&=\sum \limits _i c_f^i \theta _k^i \, \bar{R}(k\delta +\delta | k\delta ,i)\nonumber \\&\quad -\,\mathscr {C}_l(k\delta +\delta ,k\delta )\bar{R}(k\delta +\delta |k\delta ,{\varvec{\theta }_{k}})\nonumber \\&\quad -\, g(\gamma )\bar{\tau }(\delta | k,\varvec{\theta }_k) {- c_s + \int \limits _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,k\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }_k)} \end{aligned}$$

(31)

$$\begin{aligned}&\quad +\,\left[ \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_k,1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k, 1 )\right. \nonumber \\&\quad \left. -\,V^4_\gamma ( k +1,\hat{\varvec{\theta }}_{k+1}(m),k,1 ) \right] R(k\delta +\delta |k\delta ,\varvec{\theta }_k). \end{aligned}$$

(32)

Recall from (16) that $V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k,1 ) \le V^4_\gamma ( k +1,\hat{\varvec{\theta }}_{k+1}(m),k,1 )$, and thus

$$\begin{aligned} \sum \limits _{m=1}^M\Pr (y_{k+1}&=m|\varvec{\theta }_k,1)V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k,1 )\\&\le \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_k,1)V^4_\gamma ( k +1,\hat{\varvec{\theta }}_{k+1}(m),k,1 ) \\&\le V^4_\gamma ( k +1,\hat{\varvec{\theta }}_{k+1}(m),k,1 ), \end{aligned}$$

which follows that the last term in (31) is non-positive. Therefore if ${\varPhi _\gamma ^1( \varvec{\pi }_k)}$ is non-positive, then $V^1_\gamma (\varvec{\pi }_k) \le V^2_\gamma (\varvec{\pi }_k)$, that is warning immediately and no replacement is better than immediate warning and immediate replacement. Now we show that if ${\varPhi _\gamma ^1( k,\varvec{\theta }_k,\infty ,1)} > 0$, then $V_\gamma ( \varvec{\pi }_k)=V^1_\gamma (\varvec{\pi }_k)$ cannot be true. Let us assume that $\varPhi _\gamma ^1(\varvec{\pi }_k)>0$ and $V_\gamma (\varvec{\pi }_k)=V^1_\gamma (\varvec{\pi }_k)$. Then we have

$$\begin{aligned}&{V_\gamma ( k+1,{\varvec{\theta }_k},k,1 )-V_\gamma (\varvec{\pi }_k)} = V(k+1,\varvec{\theta }_k,k,1)-V^1_\gamma (\varvec{\pi }_k)\\&\quad = -\varPhi ^1_\gamma (\varvec{\pi }_k) + \bigg [V_\gamma ( k+1,{\varvec{\theta }_k},k,1 )-c_r-\mathscr {C}_l(k\delta +\delta ,k\delta )-V_\gamma (\varvec{\pi }_0)\bigg ]\\&\quad (1-{R}(k\delta +\delta |k\delta ,\varvec{\theta }_{k}))\\&\qquad + \left[ V_\gamma ( k+1,{\varvec{\theta }_k},k,1 ) -\sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_k,1)\right. \\&\qquad \left. \times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k, 1 ) \right] R(k\delta +\delta |k\delta ,{\varvec{\theta }_k})- c_s. \end{aligned}$$

From Lemma 5, we know that the left-hand side is greater than $\mathscr {C}_l(k\delta +\delta ,k\delta )-\mathscr {C}_l(k\delta ,k\delta )$. Therefore, since $-\varPhi _\gamma ^1( \varvec{\pi }_k)<0$, the second term on the right-hand side (see Equation (16)), and the third term on the right-hand side (see Lemma 3) are less than zero, then the right-hand side of the above equation is less than $\mathscr {C}_l(k\delta +\delta ,k\delta )-\mathscr {C}_l(k\delta ,k\delta )=-c_s$. Thus, the above equation cannot be true as the sign of the two sides of the equation does not match. This means that if $\varPhi _\gamma ^1(\varvec{\pi }_k)>0$, then $V_\gamma (\varvec{\pi }_k) < V^1_\gamma (\varvec{\pi }_k)$ cannot be true. $\square$

Proof

(of Theorem 2) From (13)–(14), we have

$$\begin{aligned}&V^0_\gamma ( k,\varvec{\theta }_k,\infty ,1)-V^2_\gamma (k,\varvec{\theta }_k,\infty ,1)= \underbrace{\sum \limits _i c_f^i \theta _k^i \, \bar{R}(k\delta +\delta | k\delta ,i)- g(\gamma )\bar{\tau }(\delta | k,\varvec{\theta }_k)}_{\varPhi ^2_\gamma ( t, \varvec{\theta }_k, \infty , 1 )}\\&\quad + \left[ \sum \limits _{m}\Pr (y_{k+1}=m|\varvec{\theta }_k,1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 ) - V^2_\gamma (k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1) \right] \\&\quad {R}(k\delta +\delta |k\delta ,\varvec{\theta }_{k}). \end{aligned}$$

As the second term of the above is nonpositive, it is obvious that if $\varPhi ^2_\gamma (k, \varvec{\theta }_k, \infty , 1 ) \le 0$, then $V^0_\gamma ( k,\varvec{\theta }_k,\infty ,1) \le V^2_\gamma (k,\varvec{\theta }_k,\infty ,1)$, that is, the do nothing is a better action. Now, we can show that if $\varPhi ^2_\gamma (k, \varvec{\theta }_k, \infty , 1 )>0$, then $V_\gamma ( k,\varvec{\theta }_k,\infty ,1)=V^0_\gamma (k,\varvec{\theta }_k,\infty ,1)$ cannot be true. Thus, if $\varvec{\pi }_{k}=(k,{\varvec{\theta }_k},+ \infty ,1)$ and $\varvec{\pi }_{k+1}=(k+1,{\varvec{\theta }_k},+ \infty ,1)$, then

$$\begin{aligned}&{V_\gamma (\varvec{\pi }_{k+1} )-V_\gamma ( \varvec{\pi }_{k})} ={V_\gamma ( \varvec{\pi }_{k+1})-V_\gamma ^0( \varvec{\pi }_{k} )}>0 \\&\quad = V_\gamma ( \varvec{\pi }_{k+1}) -\varPhi ^2_\gamma (\varvec{\pi }_{k}) - [c_r+\mathscr {C}_l(k\delta ,k\delta )+V_\gamma (\varvec{\pi }_0)](1-{R} (k\delta +\delta |k\delta ,\varvec{\theta }_{k}))\\&\qquad - \left[ \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_k,1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 )\right] R(k\delta +\delta |k\delta ,\varvec{\theta }_k) \\&\qquad \times V_\gamma ( k+1,{\varvec{\theta }_k},+ \infty ,1 )\\&\quad = -\varPhi ^2_\gamma (\varvec{\pi }_{k}) + {[V_\gamma ( \varvec{\pi }_{k+1} )-c_r-\mathscr {C}_l(k\delta ,k\delta )-V_\gamma (\varvec{\pi }_0)] (1-{R}(k\delta +\delta |k\delta ,\varvec{\theta }_{k}))}\\&\qquad + {\left[ V_\gamma ( \varvec{\pi }_{k}) -\sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_k,1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 ) \right] R(k\delta +\delta |k\delta ,\varvec{\theta }_k)}. \end{aligned}$$

As $-\varPhi ^2_\gamma (\varvec{\pi }_{k} )<0$, and the second and the third terms are also nonpositive, we can conclude that the above equation cannot be true. Therefore, if $\varPhi ^2_\gamma ( \varvec{\pi }_{k})>0$ , then $V_\gamma ( \varvec{\pi }_{k})=V^0_\gamma ( \varvec{\pi }_{k})$ cannot be true. $\square$

Proof

(of Theorem 3) Using (13)–(14), we have

$$\begin{aligned} V^0_\gamma ( \varvec{\pi }_{k})-V^1_\gamma (\varvec{\pi }_{k})&={ \mathscr {C}_l(k\delta ,k\delta )(1-{R}(k\delta +\delta |k\delta ,\varvec{\theta }_{k}))-\int \limits _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,k\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }_k)}\nonumber \\&\quad + \left[ \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_k,1)\times [ V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 )\right. \nonumber \\&\quad \left. -V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k, 1 ) ] \right] R(k\delta +\delta |k\delta ,\varvec{\theta }_k). \end{aligned}$$

(33)

The comparison between the value function of these two options requires evaluation of the last term of the above equation, which can be rewritten as

$$\begin{aligned} \bigg [\mathbb {E}^m\left[ V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 )\right] - \mathbb {E}^m\left[ V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k, 1 )\right] \bigg ] R(k\delta +\delta |k\delta ,\varvec{\theta }_k), \end{aligned}$$

where $\mathbb {E}^m$ is the expectation operation wrt to m, given that the replacement control is to do nothing. To calculate the above term, we use the property of the value function as

$$\begin{aligned}&V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 ,\gamma ) - V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k, 1 )\\&\quad = \sum \limits _i c_f^i \times \Big (\bar{Q}_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1|i)-\bar{Q}_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k,1|i) \Big ) \\&\qquad + \Big ( \bar{C}_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 ) - \bar{C}_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k,1 ) \Big ) \\&\qquad -g(\gamma ) \Big ( \bar{W}_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty ,1 ) - \bar{W}_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),k,1 ) \Big ). \end{aligned}$$

where $\bar{W}_\gamma ( \varvec{\pi }_k)$, $\bar{Q}_\gamma (\varvec{\pi }_k|i)$, and $\bar{C}_\gamma ( \varvec{\pi }_k)$ (see Sect. 5.4.1) are the expected remaining time to replacement, the expected probability of failure at state i, and the expected cost of the warning process, given the state $\varvec{\pi }_k$ and a policy $\gamma$ with cost $g(\gamma )$. Now, the statement given in Theorem 3 follows. $\square$

Proof

(of Theorem 4) As shown earlier, when the warning has already been issued, there are only two maintenance options to compare at the policy improvement step. Thus, if we assume $\varvec{\pi }_k=( k ,\varvec{\theta }_k,d,1)$, we get

$$\begin{aligned} V^3_\gamma (\varvec{\pi }_k)-V^4_\gamma (\varvec{\pi }_k)= & {} \left[ c_r+V_\gamma (\varvec{\pi }_0)\right] (\bar{R}(k\delta +\delta |k\delta ,\varvec{\theta }_k) )+\sum \limits _i c_f^i \theta _k^i \, \bar{R}(k\delta +\delta | k\delta ,i) \\&+\int \limits _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,d\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }_k) \\&-g(\gamma )\bar{\tau }(\delta |k,\varvec{\theta }_k)+ \left( \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_k,1)\right. \\&\left. \times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),d, 1 )\right) {R}(k\delta +\delta |k\delta ,\varvec{\theta }_k) \\&- \left[ c_r +\mathscr {C}_l(k\delta ,d\delta )+ V_\gamma (\varvec{\pi }_0)\right] . \end{aligned}$$

By adding $\pm \mathscr {C}_l(k\delta +\delta ,d\delta )R(k\delta +\delta |k\delta ,\varvec{\theta }_k)$ to the above, we get

$$\begin{aligned}&V^3_\gamma (\varvec{\pi }_k)-V^4_\gamma (\varvec{\pi }_k) \\&\quad =\underbrace{ \sum \limits _i c_f^i \theta _k^i \, \bar{R}(k\delta +\delta | k\delta ,i)- g(\gamma )\bar{\tau }(\delta | k,\varvec{\theta }_k)+ [\mathscr {C}_l(k\delta +\delta ,d\delta ){R}(k\delta +\delta |k\delta ,\varvec{\theta }_{k})-\mathscr {C}_l(k\delta ,d\delta )]+\int \limits _{k\delta }^{k\delta +\delta } \mathscr {C}_l(x,d\delta )\bar{R}(\mathrm {d}x|k\delta ,\varvec{\theta }_k)}_{\varPhi ^4_\gamma ( k,\varvec{\theta }_k,d,1)}\\&\qquad + \underbrace{\left[ \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_{k},1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),d, 1 ) - V^4_\gamma ( k+1,\varvec{\theta }_{k+1},d,1) \right] {R}(k\delta +\delta |k\delta ,\varvec{\theta }_{k}) }_{(****)}. \end{aligned}$$

It follows from (16) that $V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty , 1 ) \le V^4_\gamma ( k+1,\varvec{\theta }_{k+1},d,1)$, and thus

$$\begin{aligned} \sum \limits _{m=1}^M\Pr (y_{k+1}= & {} m|\varvec{\theta }_{k},1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),\infty , 1 ) \\\le & {} { \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta }_{k},1)} V^4_\gamma ( k+1,\varvec{\theta }_{k+1},d,1), \end{aligned}$$

which means $(****)<0$. Therefore, if $\varPhi ^4_\gamma (\varvec{\pi }_k) \le 0$, then $V^3_\gamma ( \varvec{\pi }_k)-V^4_\gamma (\varvec{\pi }_k)<0$, that is the optimal policy is do nothing. The rest of the proof is devoted to show that if $\varPhi ^4_\gamma (\varvec{\pi }_k)>0$, then $V^3_\gamma ( \varvec{\pi }_k)-V^4_\gamma ( \varvec{\pi }_k)<0$ cannot be true and therefore $V^3_\gamma (\varvec{\pi }_k)>V^4_\gamma (\varvec{\pi }_k)$, that is the optimal policy is to replace immediately. It follows from Lemma 2 that $V_\gamma ( k+1,{\varvec{\theta }},d,1 )-V(\varvec{\pi }_k)> [\mathscr {C}_l(k\delta +\delta ,d\delta )-\mathscr {C}_l(k\delta ,d\delta )]$ and then

$$\begin{aligned}&{V_\gamma ( k+1,{\varvec{\theta }},d,1 )-V_\gamma (\varvec{\pi }_k)}=V_\gamma ( k+1,{\varvec{\theta }},d,1 )-V_\gamma ^3(\varvec{\pi }_k)&\\&\quad = V_\gamma ( k+1,{\varvec{\theta }},k,1 )-V^3_\gamma (\varvec{\pi }_k) + \, \left[ V_\gamma ( k+1,{\varvec{\theta }},d,1 ){R}(k\delta +\delta |k\delta ,\varvec{\theta })\right. \\&\qquad \left. -V_\gamma ( k+1,{\varvec{\theta }},d,1 ){R}(k\delta +\delta |k\delta ,\varvec{\theta })\right] \\&\quad = -\varPhi _\gamma ^4(k,\varvec{\theta },d,1)+ \left[ \mathscr {C}_l(k\delta +\delta ,d\delta )R({k\delta +\delta }|k\delta ,\varvec{\theta })-\mathscr {C}_l(k\delta ,d\delta )\right] \\&\qquad + [V_\gamma ( k+1,{\varvec{\theta }},d,1 )-c_r-V_\gamma (\varvec{\pi }_0)](1-{R}(k\delta +\delta |k\delta ,\varvec{\theta })) \\&\qquad + V_\gamma ( k+1,{\varvec{\theta }},d,1 ){R}(\delta |k,\theta ) - \left[ \sum \limits _{m=1}^M\Pr (y_{k+1}=m|\varvec{\theta },1)\times V_\gamma ( k+1,\hat{\varvec{\theta }}_{k+1}(m),d, 1 ) \right] \\&\qquad {R}(k\delta +\delta |k\delta ,\varvec{\theta }_k). \end{aligned}$$

By adding $\mathscr {C}_l(k\delta +\delta ,d\delta )-\mathscr {C}_l(k\delta +\delta ,d\delta )$ to the above, we get

$$\begin{aligned}&{V_\gamma ( k+1,{\varvec{\theta }},d,1)-V_\gamma (\varvec{\pi }_k)}=\underbrace{-\varPhi ^4(k,\varvec{\theta },d,1)}_{<0}+ {\left[ \mathscr {C}_l(k\delta +\delta ,d\delta )-\mathscr {C}_l(k\delta ,d\delta )\right] } \\&\quad + {[V_\gamma ( k+1,{\varvec{\theta }},d,1 )-c_r-\mathscr {C}_l(k\delta +\delta ,d\delta )-V_\gamma (\varvec{\pi }_0)]}(1-{R}(k\delta +\delta |k\delta ,\varvec{\theta })) \\&\quad + \left[ V_\gamma ( k+1,{\varvec{\theta }},d,1 ) - \sum \limits _{m=1}^M \Pr (y_{k+1}=m|\varvec{\theta },1)\times V_\gamma (k+1,\hat{\varvec{\theta }}_{k+1}(m),d, 1) \right] \\&\qquad {R}(k\delta +\delta |k\delta ,\varvec{\theta }) . \end{aligned}$$

Therefore, as the last two terms of the above are nonpositive, if $\varPhi ^4_\gamma (\varvec{\pi }_k)>0$, we have sharp inequality, which is a contradiction. From the above we can conclude that $V^3_\gamma (\varvec{\pi }_k)>V^4_\gamma (\varvec{\pi }_k)$, that is the optimal decision is to replace immediately. This completes the proof. $\square$

Proof

(of Remark 2) By taking the difference between $\varPhi _\gamma ^1$ and $\varPhi _\gamma ^2$ , we get

$$\begin{aligned} \varPhi _\gamma ^1( k,\varvec{\theta }_k,\infty ,1)- \varPhi _\gamma ^2( k,\varvec{\theta }_k,\infty ,1)= \mathscr {C}_l(k\delta +\delta ,k\delta )-\mathscr {C}_l(k\delta ,k\delta ) \le 0, \end{aligned}$$

which completes the proof. $\square$

Algorithm 1 The backward recursive process to find $\bar{Q}, \, \bar{W}, \,\text {and} \, \bar{C}$ for a given policy $\gamma$

• Step 1 Using the unconditional reliability function, find $T_{max}$ as a time point at which the reliability is approximately 0. This makes $\varPhi _1$-$\varPhi _4$ greater than zero with probability 1. Set $k:=\lceil {T_{max}/\delta }\rceil$, and move to Step 2.

• Step 2 For $d:=0:k$, and each $\varvec{\theta }_k \in \tilde{\varTheta }_\sigma$, update $\varvec{\pi _k}$. If $\gamma _2(\varvec{\pi }_k)=1$, calculate $\bar{W}_\gamma (\varvec{\pi }_k)$, $\bar{Q}_\gamma (\varvec{\pi }_k,i), \forall i$, and $\bar{C}_\gamma (\varvec{\pi }_k)$, and if $\gamma _2(\varvec{\pi }_k)=0$, calculate $\bar{C}_\gamma (\varvec{\pi }_k)$ only.

• Step 3 Set $k:=k-1$. If $k \ge 0$, move back to Step 2, otherwise go to Step 4.

• Step 4 Let $\lceil k=T_{max}/\delta \rceil$. For $d=\infty$ and $\varvec{\theta }_k \in \tilde{\varTheta }_\epsilon$, update $\varvec{\pi _k}$. If $\gamma _2(\varvec{\pi }_k)=1$, calculate $\bar{W}_\gamma (\varvec{\pi }_k)$, $\bar{Q}_\gamma (\varvec{\pi }_k,i)$, and $\bar{C}_\gamma (\varvec{\pi }_k)$, and if $\gamma _2(\varvec{\pi }_k)=0$, calculate $\bar{C}_\gamma (\varvec{\pi }_k)$ only.

• Step 5 Set $k:=k-1$. If $k \ge 0$, move to Step 4, otherwise terminate the algorithm and compute $g(\gamma )$ from (9).