Optimal sequencing of heterogeneous, non-instantaneous interventions

Abstract

We consider a stochastically deteriorating system that generates reward at a rate that depends on its condition. There are three heterogeneous interventions/actions that can be performed. Two of these are one-time interventions, each of which is either ineffective or effective with some known probability after a specific delay period. The third intervention is palliative in nature (i.e., does not affect the deterioration process), and has no effectiveness delay. To maximize the total expected reward generated by the system, we examine the problem of optimally sequencing these interventions to simultaneously balance three inherent trade-offs: the time until (potential) effectiveness is revealed, probability of effectiveness, and cost. This problem is motivated by decision-making in the treatment of chronic diseases where providers must determine the order in which to prescribe treatment options (e.g., medications) with uncertain outcomes. We provide both theoretical conditions and numerical examples that indicate when it is optimal to reserve the palliative intervention as a last resort and in what order to implement the other two interventions.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. Alagoz, O., Maillart, L. M., Schaefer, A. J., & Roberts, M. S. (2004). The optimal timing of living-donor liver transplantation. Management Science, 50(10), 1420–1430.

    Article  Google Scholar 

  2. Cho, D. I., & Parlar, M. (1991). A survey of maintenance models for multi-unit systems. European Journal of Operational Research, 51(1), 1–23.

    Article  Google Scholar 

  3. Derman, C., Lieberman, G. J., & Ross, S. M. (1984). On the use of replacements to extend system life. Operations Research, 32(3), 616–627.

    Article  Google Scholar 

  4. Dohi, T., & Nakagawa, T. (2013). Stochastic reliability and maintenance modelling. London: Springer.

    Google Scholar 

  5. Frosch, D. L., & Kaplan, R. M. (1999). Shared decision making in clinical medicine: Past research and future directions. American Journal of Preventive Medicine, 17(4), 285–294.

    Article  Google Scholar 

  6. Garg, A., & Deshmukh, S. G. (2006). Maintenance management: Literature review and directions. Journal of Quality in Maintenance Engineering, 12(3), 205–238.

    Article  Google Scholar 

  7. Hunink, M. G. M., Weinstein, M. C., et al. (2014). Decision making in health and medicine: Integrating evidence and values. Cambridge: Cambridge University Press.

    Google Scholar 

  8. Icten, Z. G., Shechter, S. M., Maillart, L. M., & Nagarajan, M. (2013). Optimal management of a limited number of replacements under Markovian deterioration. IIE Transactions, 45(2), 206–214.

    Article  Google Scholar 

  9. Jardine, A. K. S., Lin, D., & Banjevic, D. (2006). A review on machinery diagnostics and prognostics implementing condition-based maintenance. Mechanical Systems and Signal Processing, 20(7), 1483–1510.

    Article  Google Scholar 

  10. Kurt, M., & Kharoufeh, J. P. (2010). Optimally maintaining a Markovian deteriorating system with limited imperfect repairs. European Journal of Operational Research, 205(2), 368–380.

    Article  Google Scholar 

  11. McCall, J. J. (1965). Maintenance policies for stochastically failing equipment: A survey. Management Science, 11(5), 493–524. https://doi.org/10.1287/mnsc.11.5.493.

    Article  Google Scholar 

  12. Ohnishi, M., Morioka, T., & Ibaraki, T. (1994). Optimal minimal-repair and replacement problem of discrete-time Markovian deterioration system under incomplete state information. Computers & industrial engineering, 27(1–4), 409–412.

    Article  Google Scholar 

  13. Serfozo, R. (2009). Basics of Applied Stochastic Processes. Berlin: Springer.

    Google Scholar 

  14. Sharma, A., Yadava, G. S., & Deshmukh, S. G. (2011). A literature review and future perspectives on maintenance optimization. Journal of Quality in Maintenance Engineering, 17(1), 5–25.

    Article  Google Scholar 

  15. Shechter, S. M., Bailey, M. D., & Schaefer, A. J. (2008). Replacing nonidentical vital components to extend system life. Naval Research Logistics (NRL), 55(7), 700–703.

    Article  Google Scholar 

  16. Shi, J., Alagoz, O., Erenay, F. S., & Su, Q. (2014). A survey of optimization models on cancer chemotherapy treatment planning. Annals of Operations Research, 221(1), 331–356.

    Article  Google Scholar 

  17. Singh, J. A., Saag, K. G., Bridges, S. L., et al. (2016). 2015 American College of Rheumatology guideline for the treatment of rheumatoid arthritis. Arthritis & Rheumatology, 68(1), 1–26.

    Article  Google Scholar 

  18. Valdez-Flores, C., & Feldman, R. M. (1989). A survey of preventive maintenance models for stochastically deteriorating single-unit systems. Naval Research Logistics, 36(4), 419–446.

    Article  Google Scholar 

  19. Wang, H. (2002). A survey of maintenance policies of deteriorating systems. European Journal of Operational Research, 139(3), 469–489.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Oleg A. Prokopyev.

Appendix

Appendix

First, we provide an important technical result that is exploited in some of the proofs below.

Lemma 5

(Puterman 1994) Let \(\{x_j\}\), \(\{x_j'\}\) be real-valued non-negative sequences satisfying

$$\begin{aligned} \sum _{j=k}^{\infty } x_j \ge \sum _{j=k}^{\infty } x_j' \end{aligned}$$
(15)

for all k, with equality holding in (15) for \(k = 0\). Suppose \(v_{j+1} \ge v_j\) for \(j = 0,1,\dots ,\) then

$$\begin{aligned} \sum _{j=0}^{\infty } v_j x_j \ge \sum _{j=0}^{\infty } v_j x_j', \end{aligned}$$
(16)

where limits in (16) exist but may be infinite.

Proof of Lemma 1

Assume \(\delta _1 \ge \delta _2\). By Assumption A4, we have

$$\begin{aligned}&\sum _{\delta '=k}^{D} P^{(d)}(\delta '|\delta _1) \ge \sum _{\delta '=k}^{D} P^{(d)}(\delta '|\delta _2) \text{, } \text{ and } \\&{\sum _{\delta '=k}^{D} P_\theta ^{(d)}(\delta '|\delta _1) \ge \sum _{\delta '=k}^{D} P_\theta ^{(d)}(\delta '|\delta _2)} \end{aligned}$$

for all \(k \in \Delta \), \(\theta \in \Theta \) and \(d \in \mathbb {Z}_+\). Recall that by Assumption A3, \(q^\theta (\delta )\) is nonincreasing in \(\delta \). Then, by Lemma 5, it follows that

$$\begin{aligned}&\sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _1) (-q^{\theta }(\delta ')) \ge \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _2) (-q^{\theta }(\delta ')),\quad \forall j \in \{1,2,\dots ,d^{\theta }\} \text{, } \text{ and } \\&{\sum _{\delta '=1}^{D} P_\theta ^{(j)}(\delta '|\delta _1) (-q^{\theta }(\delta ')) \ge \sum _{\delta '=1}^{D} P_\theta ^{(j)}(\delta '|\delta _2) (-q^{\theta }(\delta ')),\quad \forall j \in \{1,2,\dots ,d^{\theta }\}.} \end{aligned}$$

Therefore, \(r_0(\theta ,\delta _1) \le r_0(\theta ,\delta _2)\) and \(r_1(\theta ,\delta _1) \le r_1(\theta ,\delta _2)\). \(\square \)

Proof of Theorem 1

Next, we establish conditions under which it is optimal to reserve the palliative intervention as a last resort. Let us first construct a general sequence as follows: there is a delay x units of time before the system starts intervention A (i.e., the system is on palliative intervention for x units of time). If A fails, then there is another delay y units of time before the system switches to intervention B. Here, we omit the part of a sequence after both A and B fail, because the system is put on the palliative intervention indefinitely then. Note that x and y are both nonnegative integers. Denote the corresponding sequence as \(S_1\). We also construct two additional sequences \(S_2\) and \(S_3\) in a similar manner, see Fig. 5. Our goal is to show that the rewards generated by those three sequences are ordered as \(R^{\langle S_1 \rangle } \le R^{\langle S_2 \rangle } \le R^{\langle S_3 \rangle }\).

Fig. 5
figure5

Three possible intervention sequences \(S_1\), \(S_2\) and \(S_3\)

First, observe that if intervention A is ineffective, then when A reveals this fact (i.e., when the system reaches age \(t_0 + d^A + x\)), the distribution of the deterioration level is the same for both sequences \(S_1\) and \(S_2\), since the system transitions according to P during this time. As a result, the expected reward generated from this point forward is the same for both sequences. To order \(R^{\langle S_1 \rangle }\) and \(R^{\langle S_2 \rangle }\), we need to consider two cases: (i) the reward generated if intervention A is effective; (ii) the reward generated between \((t_0,t_0 + d^A + x)\) if A is ineffective. The expected reward for cases (i) and (ii) in sequence \(S_1\) is given by:

$$\begin{aligned}&\sum _{j=1}^{x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j) + \rho ^A\cdot \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_1(A,\delta ') \nonumber \\&\quad +\, \rho ^A Q^A_E\left( t_0+d^A+x\right) +\,{(1-\rho ^A)\cdot \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_0(A,\delta ').} \end{aligned}$$
(17)

Similarly, the expected reward for cases (i) and (ii) in sequence \(S_2\) is:

$$\begin{aligned}&{\rho ^A\cdot r_1(A,\delta _{t_0}) + (1-\rho ^A)\cdot r_0(A,\delta _{t_0})} \nonumber \\&\quad {+\, (1-\rho ^A)\sum _{j=d^A+1}^{d^A+x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j) + \rho ^A Q^A_E\left( t_0+d^A\right) .} \end{aligned}$$
(18)

Subtracting (17) from (18), we have

$$\begin{aligned}&{\rho ^A \bigg (Q^A_E\left( t_0+d^A\right) -Q^A_E\left( t_0+d^A+x\right) \bigg ) - \sum _{j=1}^{x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j) + (1-\rho ^A)} \nonumber \\&\quad \times {\sum _{j=d^A+1}^{d^A+x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j) + \rho ^A\cdot \bigg (r_1(A,\delta _{t_0}) - \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_1(A,\delta ')\bigg ) } \nonumber \\&\quad + {(1-\rho ^A)\cdot \bigg (r_0(A,\delta _{t_0}) - \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_0(A,\delta ')\bigg )} \end{aligned}$$
(19)

that represents the obtained difference. Based on Assumption A4, Lemma 1 and Lemma 5, observe that

$$\begin{aligned}&{r_1(A,\delta _{t_0}) - \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_1(A,\delta ') \ge 0,\ \text{ and }} \nonumber \\&{r_0(A,\delta _{t_0}) - \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_0(A,\delta ') \ge 0.} \end{aligned}$$
(20)

As a result, if

$$\begin{aligned} \rho ^A \bigg (Q^A_E\left( t_0+d^A\right) -Q^A_E\left( t_0+d^A+x\right) \bigg ) \ge \sum _{j=1}^{x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j), \end{aligned}$$
(21)

then (19) is nonnegative, i.e., the reward generated in sequence \(S_2\) is no less than that in \(S_1\). We conclude that under condition (21), sequence \(S_1\) results in a lower expected reward than \(S_2\).

Now, let us prove that the reward generated by \(S_2\) is lower than that by \(S_3\). Observe that both sequences start with intervention A for \(d^A\) units of time. Therefore, we simply omit this period from consideration. The distribution of the deterioration level of the system is the same in both new sequences \(S_2\) and \(S_3\). We immediately identify a similar situation as in the previous step (i.e., the system is in the absence of one-time intervention for \(x+y\) units of time before starting intervention B). By applying the same argument as in (17)–(21), we conclude that if

$$\begin{aligned} \rho ^B \bigg (Q^B_E\left( t_0+d^B\right) -Q^B_E\left( t_0+d^B+x+y\right) \bigg ) \ge \sum _{j=1}^{x+y} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j), \end{aligned}$$
(22)

then sequence \(S_2\) leads to a lower expected reward than \(S_3\). \(\square \)

Proof of Theorem 2

Without loss of generality, assume that \(\langle AB \rangle \) is an optimal sequence. By Lemma 1, \(f_1^{\langle AB \rangle }(\delta )\) is nonincreasing in \(\delta \). By applying Lemma 5 and mimicking the proof of Lemma 1, we can also prove that \(f_2^{\langle AB \rangle }(\delta )\) is nonincreasing in \(\delta \). Similarly, we can show that \(f_3^{\langle AB \rangle }(\delta )\) is nonincreasing in \(\delta \) by exploiting Assumption A5. Since \(V(\delta )\) is a linear combination of \(f_1^{\langle AB \rangle }(\delta )\), \(f_2^{\langle AB \rangle }(\delta )\) and \(f_3^{\langle AB \rangle }(\delta )\) with nonnegative coefficients, the nonincreasing property of \(V(\delta )\) in \(\delta \) follows. \(\square \)

Proof of Lemma 2

First, we prove that

$$\begin{aligned} f_1^{\langle AB \rangle }(\delta _{t_0}) \ge f_1^{\langle BA \rangle }(\delta _{t_0}). \end{aligned}$$
(23)

Based on Assumption A3 and the fact that \(q^A(\delta ) \ge q^B(\delta )\) for all \(\delta \in \Delta \), and \(Q^A_E(t) \ge Q^B_E(t)\) for all t, we observe that

$$\begin{aligned} Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^B\right) \ge \max _\delta q^A(\delta )\cdot (d^B-d^A) \ge \max _\delta q^B(\delta )\cdot (d^B-d^A). \end{aligned}$$
(24)

As \(d^B \ge d^A\) and \(q^A(\delta ) \ge q^B(\delta )\) for all \(\delta \in \Delta \), from (2) we have

$$\begin{aligned} r(B,\delta _{t_0}) =&\sum _{j=1}^{d^B} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') \nonumber \\ =&\sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + \sum _{j=d^A+1}^{d^B} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') \nonumber \\ \le&\sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{A}(\delta ') + \max _\delta q^B(\delta ) \cdot \sum _{j=d^A+1}^{d^B} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) \nonumber \\ =&\sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{A}(\delta ') + \max _\delta q^B(\delta )\cdot (d^B-d^A) \end{aligned}$$
(25)
$$\begin{aligned} \le&r(A,\delta _{t_0}) + Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^B\right) , \end{aligned}$$
(26)

where the last inequality follows from (24). As \(Q^A_E\left( t_0+d^B\right) \ge Q^B_E\left( t_0+d^B\right) \), then (26) implies that

$$\begin{aligned} r(A,\delta _0) + Q^A_E\left( t_0+d^A\right)&= r(A,\delta _0) + Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^B\right) + Q^A_E\left( t_0+d^B\right) \\&\quad \,\ge r(B,\delta _{t_0}) + Q^B_E\left( t_0+d^B\right) , \end{aligned}$$

and therefore, (23) holds.

Next, we show that

$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) \ge f_2^{\langle AB \rangle }(\delta _{t_0}). \end{aligned}$$

From Lemma 1 and Assumption A4, we observe that

$$\begin{aligned} r(B,\delta _{t_0}) \ge \sum _{\delta =1}^D P^{(d^A)}(\delta |\delta _{t_0})\cdot r(B,\delta ) . \end{aligned}$$
(27)

Furthermore, we have

$$\begin{aligned}&Q^B_E\left( t_0+d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) - r(A,\delta _{t_0}) \nonumber \\&\qquad \qquad \ge \max _\delta q^A(\delta ) \cdot d^A - r(A,\delta _{t_0}) \ge 0, \end{aligned}$$
(28)

where the second inequality is by Assumption A3. Therefore, by applying (27) and (28) we obtain that

$$\begin{aligned}&f_1^{\langle BA \rangle }(\delta _{t_0}) - f_2^{\langle AB \rangle }(\delta _{t_0})\\&\quad = \bigg (r(B,\delta _{t_0}) - \sum _{\delta =1}^D P^{(d^A)}(\delta |\delta _{t_0})\cdot r(B,\delta )\bigg )\\&\qquad +\, \bigg (Q^B_E\left( t_0+d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) - r(A,\delta _{t_0}) \bigg ) \ge 0, \end{aligned}$$

which completes the proof. \(\square \)

Proof of Lemma 3

Define

$$\begin{aligned} L(u)=\sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{A}(\delta ') + \sum _{\delta '=1}^{D} P^{(d^A)}(\delta '|\delta _{t_0}) \sum _{j=1}^{u} \sum _{\delta ''=1}^{D} P^{(j)}(\delta ''|\delta ') q^{B}(\delta ''). \end{aligned}$$
(29)

Similarly, define

$$\begin{aligned} R(u)=\sum _{j=1}^{u} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + \sum _{\delta '=1}^{D} P^{(u)}(\delta '|\delta _{t_0}) \sum _{j=1}^{d^A} \sum _{\delta ''=1}^{D} P^{(j)}(\delta ''|\delta ') q^{A}(\delta ''). \end{aligned}$$
(30)

Observe that (9) is equivalent to

$$\begin{aligned} L(d^B) - R(d^B) \ge 0. \end{aligned}$$
(31)

Next, we prove (31). For any \(k,j \in \mathbb {Z}_+\) and \(\delta '' \in \Delta \), we have (e.g., see Serfozo (2009))

$$\begin{aligned} \sum _{\delta '=1}^{D} P^{(k)}(\delta '|\delta _{t_0}) P^{(j)}(\delta ''|\delta ') = P^{(k+j)}(\delta ''|\delta _{t_0}), \end{aligned}$$

which implies that

$$\begin{aligned} \sum _{\delta '=1}^{D} P^{(d^A)}(\delta '|\delta _{t_0}) \sum _{j=1}^{u} \sum _{\delta ''=1}^{D} P^{(j)}(\delta ''|\delta ') q^{B}(\delta '') = \sum _{j=d^A+1}^{d^A+u} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{B}(\delta ), \end{aligned}$$

and

$$\begin{aligned} \sum _{\delta '=1}^{D} P^{(u)}(\delta '|\delta _{t_0}) \sum _{j=1}^{d^A} \sum _{\delta ''=1}^{D} P^{(j)}(\delta ''|\delta ') q^{A}(\delta '') = \sum _{j=u+1}^{d^A+u} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{A}(\delta ). \end{aligned}$$

Therefore,

$$\begin{aligned} L(d^A) - R(d^A)&= \sum _{j=1}^{d^A} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) (q^{A}(\delta ) - q^{B}(\delta ))\nonumber \\&\quad \,-\sum _{j=d^A+1}^{d^A+d^A} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) (q^{A}(\delta ) - q^{B}(\delta )) \ge 0, \end{aligned}$$
(32)

where the inequality holds by Assumption A4. Because both \(d^A\) and \(d^B\) are integers, and \(d^B \ge d^A\), we can now increase u from \(d_A\) to \(d_B\) and check if a similar inequality still holds as in (32). If \(u = d^A + 1\), then the increase in (29) is given by

$$\begin{aligned} L(d^A+1) - L(d^A)&= \sum _{\delta =1}^{D} \sum _{j=d^A+1}^{d^A+d^A+1} P^{(j)}(\delta |\delta _{t_0}) q^{B}(\delta ) \\&\quad - \sum _{\delta =1}^{D} \sum _{j=d^A+1}^{d^A+d^A} P^{(j)}(\delta |\delta _{t_0}) q^{B}(\delta ) \nonumber \\&= \sum _{\delta =1}^{D} P^{(d^A+d^A+1)}(\delta |\delta _{t_0}) q^{B}(\delta ), \end{aligned}$$

and the increase in (30) is given by

$$\begin{aligned}&R(d^A+1) - R(d^A) \\&\quad = \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{B}(\delta ) + \sum _{j=d^A+2}^{d^A+d^A+1} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{A}(\delta ) \\&\qquad -\, \sum _{j=d^A+1}^{d^A+d^A} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{A}(\delta ) \\&\quad = \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{B}(\delta ) + \sum _{\delta =1}^{D} P^{(d^A+d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) - \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ). \end{aligned}$$

Then by Lemma 5 we have that

$$\begin{aligned}&L(d^A+1) - L(d^A) - \bigg (R(d^A+1) - R(d^A)\bigg ) \nonumber \\&\quad = \sum _{\delta =1}^{D} P^{(d^A+d^A+1)}(\delta |\delta _{t_0}) (q^B(\delta )-q^A(\delta )) - \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) (q^B(\delta )-q^A(\delta )) \ge 0. \end{aligned}$$
(33)

Therefore,

$$\begin{aligned}&L(d^A+1)- R(d^A+1) \\&\quad = \bigg (L(d^A+1) - L(d^A)+ L(d^A)\bigg ) - \bigg (R(d^A+1) - R(d^A)+R(d^A)\bigg ) \\&\quad = \bigg (L(d^A+1) - L(d^A) - R(d^A+1) + R(d^A) \bigg ) + \bigg (L(d^A)-R(d^A)\bigg ) \ge 0. \end{aligned}$$

The last inequality holds due to (32) and (33). It implies that increasing one unit of u in both (29) and (30) does not change the inequality in (32). We can apply the same approach to u until it reaches \(d^B\), and the same result holds. Thus, (31) holds, which completes the proof. \(\square \)

Proof of Lemma 4

By taking the difference between the left-hand and right-hand sides of (10), we have

$$\begin{aligned}&r(A,\delta _{t_0}) + Q^A_E\left( t_0 + d^A\right) - \bigg (r(B,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta ) + Q^A_E\left( t_0 + d^A + d^B\right) \bigg ) \nonumber \\&\quad = \bigg (r(A,\delta _{t_0}) - \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta )\bigg ) \nonumber \\&\qquad +\, \bigg (Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^A+d^B\right) - r(B,\delta _{t_0}) \bigg ) . \end{aligned}$$
(34)

Observe that \(r(A,\delta _{t_0}) \ge \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta )\), because P is upper triangular by Assumption A4. In addition, by Assumption A3

$$\begin{aligned} Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^A+d^B\right) \ge \max _{\delta } q^A(\delta ) d^B \ge \max _{\delta } q^B(\delta ) d^B \ge r(B,\delta _{t_0}). \end{aligned}$$

Therefore, (34) is nonnegative, which completes the proof. \(\square \)

Proof of Theorem 3

First, from Assumption A3 and similar to (24) we observe that

$$\begin{aligned}&r(A,\delta _{t_0}) + \bigg (Q^B_E\left( t_0 + d^A\right) - Q^B_E\left( t_0+d^B\right) \bigg ) \nonumber \\&\ge r(A,\delta _{t_0}) + \max _\delta q^B(\delta )\cdot (d^B-d^A) \nonumber \\&\ge \sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta ) q^{B}(\delta ') + \max _\delta q^B(\delta )\cdot (d^B-d^A) \nonumber \\&\ge r(B,\delta _{t_0}) . \end{aligned}$$
(35)

Next, we have

$$\begin{aligned}&r(A,\delta _{t_0}) + Q^A_E\left( t_0 + d^A\right) - Q^A_E\left( t_0+d^A+d^B\right) \nonumber \\&\quad = r(A,\delta _{t_0}) + \bigg (Q^A_E\left( t_0 + d^A\right) - Q^A_E\left( t_0+d^A+d^B\right) \bigg )\nonumber \\&\qquad + \bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\qquad -\, \bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\ge r(A,\delta _{t_0}) + \bigg (Q^B_E\left( t_0 + d^A\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\qquad + \bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\qquad - \,\bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&= r(A,\delta _{t_0}) + \bigg (Q^B_E\left( t_0 + d^A\right) - Q^B_E\left( t_0+d^B\right) \bigg )\nonumber \\&\qquad + \bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\ge r(B,\delta _{t_0}) + Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) , \end{aligned}$$
(36)

where the first inequality holds by the assumption that \(Q^A_E(t) - Q^A_E(t+k) \ge Q^B_E(t) - Q^B_E(t+k)\) for all tk, and the second inequality holds by (35). Therefore, (36) implies that

$$\begin{aligned}&r(A,\delta _{t_0}) + Q^A_E\big (t_0 + d^A\big ) + Q^B_E\big (t_0+d^A+d^B\big ) \nonumber \\&\quad \ge r(B,\delta _{t_0}) + Q^B_E\big (t_0 + d^B\big ) + Q^A_E\big (t_0+d^A+d^B\big ). \end{aligned}$$
(37)

Next, observe that

$$\begin{aligned}&f_1^{\langle AB \rangle }(\delta _{t_0}) + f_2^{\langle AB \rangle }(\delta _{t_0}) - \bigg ( f_1^{\langle BA \rangle }(\delta _{t_0}) + f_2^{\langle BA \rangle }(\delta _{t_0}) \bigg ) \nonumber \\&= r(A,\delta _{t_0}) + Q^A_E\big (t_0+d^A\big ) + r(A,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) + Q^B_E\big (t_0+d^A+d^B\big ) \nonumber \\&\quad -\, \bigg (r(B,\delta _{t_0}) + Q^B_E\big (t_0+d^B\big ) + r(B,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta ) + Q^A_E\big (t_0+d^A+d^B\big ) \bigg ) \nonumber \\&= \bigg (r(A,\delta _{t_0}) + Q^A_E\big (t_0+d^A\big )+Q^B_E\big (t_0+d^A+d^B\big ) - r(B,\delta _{t_0}) \nonumber \\&\qquad - Q^B_E\big (t_0+d^B\big ) - Q^A_E\big (t_0+d^A+d^B\big ) \bigg ) \nonumber \\&\quad +\, \bigg (r(A,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) - r(B,\delta _{t_0}) - \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta ) \bigg ) \ge 0, \end{aligned}$$
(38)

where the last inequality holds by (37) and Lemma 3. Also, from Lemma 3, we have that

$$\begin{aligned}&f_3^{\langle AB \rangle }(\delta _{t_0})\\&\quad = r(A,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) + \sum _{\delta =1}^{D}P^{(d^A+d^B)} (\delta |\delta _{t_0}) Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ) \\&\quad \ge r(B,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta ) + \sum _{\delta =1}^{D}P^{(d^A+d^B)} (\delta |\delta _{t_0}) Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ) \\&\quad = f_3^{\langle BA \rangle }(\delta _{t_0}). \end{aligned}$$

Therefore, from (7), we observe that a sufficient condition under which it is optimal to prescribe intervention A first is given by

$$\begin{aligned} \rho ^A f_1^{\langle AB \rangle }(\delta _{t_0}) + (1-\rho ^A)\rho ^B f_2^{\langle AB \rangle }(\delta _{t_0}) \ge \rho ^B f_1^{\langle BA \rangle }(\delta _{t_0}) + (1-\rho ^B)\rho ^A f_2^{\langle BA \rangle }(\delta _{t_0}), \end{aligned}$$

which can be re-written as

$$\begin{aligned}&\rho ^A \ge {\bar{\rho }} = \frac{\rho ^B \cdot \bigg (f_1^{\langle BA \rangle }(\delta _{t_0})-f_2^{\langle AB \rangle }(\delta _{t_0})\bigg )}{f_1^{\langle AB \rangle }(\delta _{t_0})-\rho ^B f_2^{\langle AB \rangle }(\delta _{t_0}) - (1-\rho ^B) f_2^{\langle BA \rangle }(\delta _{t_0})}. \end{aligned}$$
(39)

Now, let us verify that \({\bar{\rho }} \le \rho ^B\). The difference between \(f_1^{\langle BA \rangle }(\delta _{t_0})-f_2^{\langle AB \rangle }(\delta _{t_0})\) and \(f_1^{\langle AB \rangle }(\delta _{t_0})-\rho ^B f_2^{\langle AB \rangle }(\delta _{t_0}) - (1-\rho ^B) f_2^{\langle BA \rangle }(\delta _{t_0})\) in (39) is given by

$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) + (1-\rho ^B)f_2^{\langle BA \rangle }(\delta _{t_0}) - \bigg (f_1^{\langle AB \rangle }(\delta _{t_0}) + (1-\rho ^B)f_2^{\langle AB \rangle }(\delta _{t_0})\bigg ). \end{aligned}$$
(40)

From (38) we have

$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) + f_2^{\langle BA \rangle }(\delta _{t_0}) \le f_1^{\langle AB \rangle }(\delta _{t_0}) + f_2^{\langle AB \rangle }(\delta _{t_0}), \end{aligned}$$

and by multiplying the both sides in the above inequality by \((1-\rho ^B)\), we have

$$\begin{aligned} (1-\rho ^B) \bigg (f_1^{\langle BA \rangle }(\delta _{t_0}) + f_2^{\langle BA \rangle }(\delta _{t_0})\bigg ) \le (1-\rho ^B) \bigg (f_1^{\langle AB \rangle }(\delta _{t_0}) + f_2^{\langle AB \rangle }(\delta _{t_0})\bigg ). \end{aligned}$$
(41)

From Lemma 2, we have

$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) \le f_1^{\langle AB \rangle }(\delta _{t_0}). \end{aligned}$$
(42)

Multiplying (42) by \(\rho ^B\) and adding to (41), we obtain

$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) + (1-\rho ^B)f_2^{\langle BA \rangle }(\delta _{t_0}) \le f_1^{\langle AB \rangle }(\delta _{t_0}) + (1-\rho ^B)f_2^{\langle AB \rangle }(\delta _{t_0}), \end{aligned}$$

which implies the term in (40) is nonpositive, and therefore \({\bar{\rho }} \le \rho ^B \cdot 1 = \rho ^B.\) This completes the proof. \(\square \)

Proof of Theorem 4

The proof of this result follows similar arguments as in the proof of Theorem 3. In particular, see inequality (39). \(\square \)

Proof of Theorem 5

From (7), the necessary and sufficient condition to prefer intervention sequence \(\langle AB \rangle \) rather than \(\langle BA \rangle \) is given by

$$\begin{aligned}&\rho ^A Q^A_E\big (t_0+d^A\big ) + (1-\rho ^A) \rho ^B Q^B_E\big (t_0+d^A+d^B\big ) \nonumber \\&\quad +\, r(A,\delta _{t_0}) + (1-\rho ^A)\sum _{\delta '=1}^{D} P^{(d^A)}(\delta '|\delta _{t_0})r(B,\delta ') \nonumber \\&\ge \rho ^B Q^B_E\big (t_0+d^B\big ) + (1-\rho ^B) \rho ^A Q^A_E\big (t_0+d^A+d^B\big )\nonumber \\&\quad +\, r(B,\delta _{t_0}) + (1-\rho ^B)\sum _{\delta '=1}^{D} P^{(d^B)}(\delta '|\delta _{t_0})r(A,\delta '). \end{aligned}$$
(43)

Since \(q^A(\delta ) = \mu q^B(\delta )\) for all \(\delta \in \Delta \), and \(Q^A_E(t) = \mu Q^B_E(t)\) for all t, we can rearrange the terms in (43), and have

$$\begin{aligned} \mu \ge \frac{\rho ^B Q^B_E\big (t_0+d^B\big ) - (1-\rho ^A)\rho ^B Q^B_E(t_1) + r(B,\delta _{t_0}) - (1-\rho ^A) \sum \nolimits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) }{\rho ^A Q^B_E\big (t_0+d^A\big ) + \sum \nolimits _{j=1}^{d^A} \sum \nolimits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - (1-\rho ^B)\bigg (\rho ^A Q^B_E(t_1) + \sum \nolimits _{j=d^B+1}^{d^A+d^B} \sum \nolimits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ')\bigg )}, \end{aligned}$$
(44)

where \(t_1 = t_0 + d^A + d^B\).

Next, we show that \(\mu \le 1\) (otherwise, this result is equivalent to Theorem 3). First, we can verify that both numerator and denominator in (44) are positive. First, define

$$\begin{aligned} g_1&= \rho _A Q^B_E\big (t_0+d^A\big ) - \rho _B Q^B_E\big (t_0+d^B\big ) - (\rho ^A-\rho ^B)Q^B_E\big (t_0 + d^A+d^B\big ), \\ g_2&= - \rho ^A \sum \limits _{j=d^A+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + \rho ^B \sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta '). \end{aligned}$$

Next, we verify that \(g_1 + g_2\) is the difference between the denominator and numerator of (44). Observe that the part corresponding to \(g_1\) is straightforward, while \(g_2\) is derived in the following way

$$\begin{aligned}&\sum \limits _{j=1}^{d^A} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - r(B,\delta _{t_0}) - (1-\rho ^B)\sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') \nonumber \\&\quad + \,(1-\rho ^A)\sum \limits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) \nonumber \\&= \sum \limits _{j=1}^{d^A} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - r(B,\delta _{t_0}) - \sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ')\nonumber \\&\quad +\, \sum \limits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) \nonumber \\&\quad +\, \rho ^B\sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - \rho ^A\sum \limits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) \nonumber \\&= \bigg (\sum \limits _{j=1}^{d^A} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - \sum \limits _{j=1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ')\bigg ) \nonumber \\&\quad - \bigg (\sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - \sum \limits _{j=d^A+1}^{d^A+d^B} \sum \limits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta )\bigg ) + g_2 \nonumber \\&= -\sum \limits _{j=d^A+1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + \sum \limits _{j=d^A+1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + g_2 = g_2. \end{aligned}$$
(45)

To prove \(g_1 + g_2 \ge 0\), note first

$$\begin{aligned} Q^B_E\big (t_0+d^A\big ) - Q^B_E\big (t_0+d^B\big ) \ge \max _\delta q^B(\delta )\cdot (d^B-d^A) \ge \sum \limits _{j=d^A+1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta '), \end{aligned}$$
(46)
$$\begin{aligned} Q^B_E\big (t_0+d^B\big )-Q^B_E\big (t_0 + d^A+d^B\big ) \ge \max _\delta q^B(\delta )\cdot d^A \ge \sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta '). \end{aligned}$$
(47)

By rearranging the terms in \(g_1\), we have

$$\begin{aligned} g_1&= \rho _A Q^B_E\big (t_0+d^A\big ) - \rho _B Q^B_E\big (t_0+d^B\big ) - (\rho ^A-\rho ^B)Q^B_E\big (t_0 + d^A+d^B\big ) \nonumber \\&= \rho _A \bigg (Q^B_E\big (t_0+d^A\big ) - Q^B_E\big (t_0+d^B\big )\bigg ) \nonumber \\&\quad +\, (\rho ^A-\rho ^B)\bigg (Q^B_E\big (t_0+d^B\big )-Q^B_E\big (t_0 + d^A+d^B\big )\bigg ). \end{aligned}$$
(48)

We also have

$$\begin{aligned} g_2&= - \rho ^A \sum \limits _{j=d^A+1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - (\rho ^A-\rho ^B) \sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta '). \end{aligned}$$
(49)

From (46)–(49), we can verify that \(g_1 + g_2 \ge 0\). Therefore, \(\mu \le 1\). \(\square \)

Proof of Proposition 1

Define \(W^{\langle AB \rangle }(d^A)\) and \(W^{\langle BA \rangle }(d^A)\) to be the reward gained by implementing sequences \(\langle AB \rangle \) and \(\langle BA \rangle \), respectively. We first show that both of the reward is nonincreasing in \(d^A\). Note that

$$\begin{aligned}&W^{\langle AB \rangle }(d^A) \\&= r(A,\delta _{t_0})+\rho ^A Q^A_E\big (t_0+d^A\big )+(1-\rho ^A) \sum _{\delta =1}^{D} P^{(d^A)}(\delta |\delta _{t_0})\cdot r(B,\delta ) \\&\quad +\, (1-\rho ^A)\rho ^B Q^B_E\big (t_0+d^A+d^B\big ) \\&\quad + (1-\rho ^A)(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\cdot Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ) \\&= \sum _{j=1}^{d^A} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{A}(\delta )+\rho ^A Q^A_E\big (t_0+d^A\big )+(1-\rho ^A) \sum _{j=d^A+1}^{d^A+d^B} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^B(\delta ) \\&\quad +\, (1-\rho ^A)\rho ^B Q^B_E\big (t_0+d^A+d^B\big ) + (1-\rho ^A)(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\\&\quad \times Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ). \end{aligned}$$

Hence,

$$\begin{aligned}&W^{\langle AB \rangle }(d^A) - W^{\langle AB \rangle }(d^A+1 ) \\&= -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \\&\quad +\, (1-\rho ^A) \bigg (\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^B(\delta ) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^B(\delta )\bigg ) \\&\quad +\, (1-\rho ^A)\rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg ) + (1-\rho ^A)(1-\rho ^B) \\&\quad \times \,\bigg (\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\cdot Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0})\\&\quad \times \, Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B+1\big )\bigg ). \end{aligned}$$

Because \(\rho ^A \ge \rho ^B\) and \(Q^A_E(t) - Q^A_E(t+k) \ge Q^B_E(t) - Q^B_E(t+k)\), we have

$$\begin{aligned} \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \ge \rho ^B \bigg (Q^B_E\big (t_0+d^A\big ) - Q^B_E\big (t_0+d^A+1\big )\bigg ). \end{aligned}$$
(50)

From (13), Assumption A3 and (50), we can show that

$$\begin{aligned} -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \ge 0. \end{aligned}$$
(51)

From Assumptions A3A4 and Lemma 5, we have

$$\begin{aligned} \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^B(\delta ) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^B(\delta ) \ge 0. \end{aligned}$$
(52)

Based on Assumption A2, we have that \(Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big ) \ge 0\). Furthermore, Assumptions A4A5 and Lemma 5 imply that

$$\begin{aligned}&\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\cdot Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B\right) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0})\nonumber \\&\quad \times Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B+1\right) \ge 0. \end{aligned}$$
(53)

From (51)–(53) we conclude that \(W^{\langle AB \rangle }(d^A) - W^{\langle AB \rangle }(d^A+1 )\) is nonnegative.

Similarly, for intervention sequence \(\langle BA \rangle \), we have

$$\begin{aligned}&W^{\langle BA \rangle }(d^A) \\&= r(B,\delta _{t_0})+\rho ^B Q^B_E\left( t_0+d^B\right) +(1-\rho ^B) \sum _{\delta =1}^{D} P^{(d^B)}(\delta |\delta _{t_0})\cdot r(A,\delta ) \\&\quad +\, (1-\rho ^B)\rho ^A Q^A_E\left( t_0+d^A+d^B\right) + (1-\rho ^A)(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\\&\quad \times Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B\right) \\&= \sum _{j=1}^{d^B} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{B}(\delta )+\rho ^B Q^B_E\left( t_0+d^B\right) +(1-\rho ^B) \sum _{j=d^B+1}^{d^A+d^B} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^A(\delta ) \\&\quad +\, (1-\rho ^B)\rho ^A Q^A_E\left( t_0+d^A+d^B\right) + (1-\rho ^A)(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\\&\quad \times Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B\right) . \end{aligned}$$

Therefore,

$$\begin{aligned}&W^{\langle BA \rangle }(d^A) - W^{\langle BA \rangle }(d^A+1 ) \\&\quad = -(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) \\&\qquad + (1-\rho ^B)\rho ^A \bigg (Q^A_E\left( t_0+d^A+d^B\right) -Q^A_E\left( t_0+d^A+d^B+1\right) \bigg ) + (1-\rho ^A)(1-\rho ^B)\\&\qquad \times \bigg (\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\cdot Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B\right) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0})\\&\qquad \times Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B+1\right) \bigg ). \end{aligned}$$

Similar to (51), we have

$$\begin{aligned} -\sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A+d^B\big )-Q^A_E\big (t_0+d^A+d^B+1\big )\bigg ) \ge 0. \end{aligned}$$
(54)

Based on (53), we conclude that \(W^{\langle BA \rangle }(d^A) - W^{\langle BA \rangle }(d^A+1 )\) is also nonnegative. Therefore, both \(W^{\langle AB \rangle }(d^A)\) and \(W^{\langle BA \rangle }(d^A)\) are nonincreasing in \(d^A\).

Next, we prove that for any \(d^A\)

$$\begin{aligned} W^{\langle AB \rangle }(d^A) - W^{\langle AB \rangle }(d^A+1 ) \ge W^{\langle BA \rangle }(d^A) - W^{\langle BA \rangle }(d^A+1). \end{aligned}$$

In order to show this fact, we use the following derivations:

$$\begin{aligned}&W^{\langle AB \rangle }(d^A) - W^{\langle AB \rangle }(d^A+1 ) - \bigg (W^{\langle BA \rangle }(d^A) - W^{\langle BA \rangle }(d^A+1)\bigg )\\&\quad = -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \\&\qquad +\, (1-\rho ^A) \bigg (\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^B(\delta ) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^B(\delta )\bigg )\\&\qquad +\, (1-\rho ^A)\rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg ) \\&\qquad +\, (1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) \\&\qquad - (1-\rho ^B)\rho ^A \bigg (Q^A_E\big (t_0+d^A+d^B\big )-Q^A_E\big (t_0+d^A+d^B+1\big )\bigg ) \\&\quad \ge -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \\&\qquad +\, (1-\rho ^A)\rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg ) \\&\quad \quad - (1-\rho ^B)\rho ^A \bigg (Q^A_E\big (t_0+d^A+d^B\big )-Q^A_E\big (t_0+d^A+d^B+1\big )\bigg ) \\&\quad \ge -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \rho ^B \bigg (Q^A_E\big (t_0+d^A+d^B\big )-Q^A_E\big (t_0+d^A+d^B+1\big )\bigg )\\&\quad \quad + \,(1-\rho ^A)\rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg )\\&\quad \ge -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg ) \ge 0. \end{aligned}$$

The first inequality follows by (52) and the nonnegativity of \(q^A(\delta )\). The second inequality is based on (14). The third inequality is due to our assumption \(Q^A_E(t) - Q^A_E(t+k) \ge Q^B_E(t) - Q^B_E(t+k)\) for all t and \(k > 0\) as well as Assumption A3. Finally, the last inequality holds by (13).

As a result, if \(d^A\) increases by one unit, then the decrease in value function \(W^{\langle AB \rangle }(d^A)\) is no smaller than that in \(W^{\langle BA \rangle }(d^A)\). Recall from Theorem 3 that \(W^{\langle AB \rangle }(1) \ge W^{\langle BA \rangle }(1)\) because \(d^B \ge 1\). Therefore, there exists at most one \(\bar{d^A}\), such that we prefer sequence \(\langle AB \rangle \) if \(d^A \le \bar{d^A}\), and prefer sequence \(\langle BA \rangle \), otherwise. \(\square \)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

He, K., Maillart, L.M. & Prokopyev, O.A. Optimal sequencing of heterogeneous, non-instantaneous interventions. Ann Oper Res 276, 109–135 (2019). https://doi.org/10.1007/s10479-018-2813-3

Download citation

Keywords

  • Maintenance planning
  • Heterogeneous non-instantaneous interventions
  • Sequencing
  • Optimization