Skip to main content
SpringerLink
Log in

Optimal Stopping Time on Semi-Markov Processes with Finite Horizon

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

In this paper, we consider the optimal stopping problems on semi-Markov processes (sMPs) with finite horizon and aim to establish the existence and algorithm of optimal stopping times. The key method is the equivalence between optimal stopping problems on sMPs and a special class of semi-Markov decision processes (sMDPs). We first introduce the optimality equation and show the existence of the optimal policies of finite-horizon sMDPs with additional terminal costs. Based on the optimal stopping problems on sMPs, we give an explicit construction of sMDPs such that the optimal stopping times of sMPs are equivalent to the optimal policies of the constructed sMDPs. Then, using the results of sMDPs developed here, we not only prove the existence of the optimal stopping times of sMPs, but also provide an algorithm for computing the optimal stopping times of sMPs. Moreover, we show that the optimal and \(\varepsilon \)-optimal stopping time can be characterized by the hitting time of some special sets. Finally, we give an example to illustrate the effectiveness of our conclusions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Arkin, V.I., Slastnikov, A.D.: On optimal threshold stopping times for Itô diffusions. Stochastics. 93(5), 655-681 (2021) https://doi.org/10.1080/17442508.2020.1783263

  2. Bäuerle, N., Popp, A.: Risk-sensitive stopping problems for continuous-time Markov chains. Stochastics 90(3), 411–431 (2018). https://doi.org/10.1080/17442508.2017.1357724

    Article  MathSciNet  MATH  Google Scholar 

  3. Bäuerle, N., Rieder, U.: Markov Decision Processes with Applications to Finance. Springer, Heidelberg (2011)

    Book  Google Scholar 

  4. Belomestny, D., Krätschmer, V.: Optimal stopping under model uncertainty: randomized stopping times approach. Ann. Appl. Probab. 26(2), 1260–1295 (2016). https://doi.org/10.1214/15-AAP1116

    Article  MathSciNet  MATH  Google Scholar 

  5. Boshuizen, F.A., Gouweleeuw, J.M.: General optimal stopping theorems for semi-Markov processes. Adv. Appl. Probab. 25(4), 825–846 (1993). https://doi.org/10.1017/S0001867800025775

    Article  MathSciNet  MATH  Google Scholar 

  6. Brémaud, P.: Point Processes and Queues: Martingale Dynamics. Springer, New York (1981)

    Book  Google Scholar 

  7. Çekyay, B.: Customizing exponential semi-Markov decision processes under the discounted cost criterion. Eur. J. Oper. Res. 266(1), 168–178 (2017). https://doi.org/10.1016/j.ejor.2017.09.016

    Article  MathSciNet  MATH  Google Scholar 

  8. Chen, F., Guo, X.P., Liao, Z.-W.: Optimal stopping time on discounted semi-Markov processes. Front. Math. China. 16(2), 303–324 (2021). https://doi.org/10.1007/s11464-021-0919-4

    Article  MathSciNet  MATH  Google Scholar 

  9. Chow, Y.S., Robbins, H., Siegmund, D.: Great Expectations: The Theory of Optimal Stopping. Houghton Mifflin Company, Boston (1991)

    MATH  Google Scholar 

  10. Christensen, S., Lindensjö, K.: On time-inconsistent stopping problems and mixed strategy stopping times. Stochastic Process. Appl. 130(5), 2886–2917 (2020). https://doi.org/10.1016/j.spa.2019.08.010

    Article  MathSciNet  MATH  Google Scholar 

  11. Davis, M.H.A.: Markov Models and Optimization. Chapman and Hall, London (1993)

    Book  Google Scholar 

  12. Dufour, F., Piunovskiy, A.B.: Multiobjective stopping problem for discrete-time Markov processes: convex analytic approach. J. Appl. Probab. 47(4), 947–966 (2010). https://doi.org/10.1017/S0021900200007282

    Article  MathSciNet  MATH  Google Scholar 

  13. Feinberg, E.A.: Reduction of discounted continuous-time MDPs with unbounded jump and reward rates to discrete-time total-reward MDPs. Optimization, Control, and Applications of Stochastic Systems, Systems Control Found. Birkhuser, New York, 77–97 (2012) https://doi.org/10.1007/978-0-8176-8337-5_5

  14. Gapeev, P.V., Kort, P.M., Lavrutich, M.N.: Discounted optimal stopping problems for maxima of geometric Brownian motions with switching payoffs. Adv. Appl. Probab. 53(1), 189–219 (2021). https://doi.org/10.1017/apr.2020.57

    Article  MathSciNet  MATH  Google Scholar 

  15. Henderson, V., Hobson, D., Zeng, M.: Optimal stopping and the sufficiency of randomized threshold strategies. Electron. Commun. Probab. 23(30), 11 (2018). https://doi.org/10.1214/18-ECP125

    Article  MathSciNet  MATH  Google Scholar 

  16. Hernández-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Springer, New York (1996)

    Book  Google Scholar 

  17. Huang, Y.H., Guo, X.P.: Finite horizon semi-Markov decision processes with application to maintenance systems. Eur. J. Oper. Res. 212(1), 131–140 (2011). https://doi.org/10.1016/j.ejor.2011.01.027

    Article  MathSciNet  MATH  Google Scholar 

  18. Huang, Y.J., Zhou, Z.: The optimal equilibrium for time-inconsistent stopping problems: the discrete-time case. SIAM J. Control. Optim. 57(1), 590–609 (2019). https://doi.org/10.1137/17M1139187

    Article  MathSciNet  MATH  Google Scholar 

  19. Jaśkiewicz, A., Nowak, A.S.: Optimality in Feller semi-Markov control processes. Oper. Res. Lett. 34(6), 713–718 (2006). https://doi.org/10.1016/j.orl.2005.11.005

    Article  MathSciNet  MATH  Google Scholar 

  20. Karpowicz, A.: Double optimal stopping in the fishing problem. J. Appl. Probab. 46(2), 415–428 (2009). https://doi.org/10.1239/jap/1245676097

    Article  MathSciNet  MATH  Google Scholar 

  21. Karpowicz, A., Szajowski, K.: Double optimal stopping of a risk process. Stochastics 79(1–2), 155–167 (2007). https://doi.org/10.1080/17442500601084204

    Article  MathSciNet  MATH  Google Scholar 

  22. Kitaev, M.Y.: Semi-Markov and jump Markov controlled models: average cost criterion. Theory Probab. Appl. 30(2), 272–288 (1986). https://doi.org/10.1137/1130036

    Article  MATH  Google Scholar 

  23. Kitaev, M.Y., Rykov, V.: Controlled Queueing Systems. CRC Press, Boca Raton (1995)

    MATH  Google Scholar 

  24. Limnios, N., Oprisan, G.: Semi-Markov Processes and Reliability. Birkhäuser, Boston (2001)

    Book  Google Scholar 

  25. Nutz, M., Zhang, Y.: Conditional optimal stopping: A time-inconsistent optimization. Ann. Appl. Probab. 30(4), 1669–1692 (2020). https://doi.org/10.2139/ssrn.3409585

    Article  MathSciNet  MATH  Google Scholar 

  26. Ohtsubo, Y.: Optimal stopping in generalized semi-Markov jump processes. Mem. Fac. Sci. A 5, 63–71 (1984)

    MathSciNet  MATH  Google Scholar 

  27. Rosenberg, D., Solan, E., Vieille, N.: Stopping games with randomized strategies. Probab. Theory Related Fields 119(3), 433–451 (2001). https://doi.org/10.1007/PL00008766

    Article  MathSciNet  MATH  Google Scholar 

  28. de Saporta, B., Dufour, F., Nivot, C.: Partially observed optimal stopping problem for discrete-time Markov processes. 4OR. 15(3), 277-302 (2017) https://doi.org/10.1007/s10288-016-0337-8

  29. Shao, J.H., Tian, T.R.: Optimal stopping problem for jump-diffusion processes with regime-switching. Nonlinear Anal. Hybrid Syst. 41: No.101029 (2021) https://doi.org/10.1016/j.nahs.2021.101029

  30. Shiryaev, A.N.: Optimal Stopping Rules. Springer, New York (1978)

    MATH  Google Scholar 

  31. Sinha, S., Mondal, P.: Semi-Markov decision processes with limiting ratio average rewards. J. Math. Anal. Appl. 455(1), 864–871 (2017). https://doi.org/10.1016/j.jmaa.2017.06.017

    Article  MathSciNet  MATH  Google Scholar 

  32. Szajowski, K.: Optimal stopping of a 2-vector risk process. Banach Center Publ. 90, 179-191 (2010) https://doi.org/10.4064/bc90-0-12

  33. Zuckerman, D.: Optimal stopping in a semi-Markov shock model. J. Appl. Probab. 15(3), 629–634 (1978). https://doi.org/10.2307/3213126

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was partly supported by the National Natural Science Foundation of China (No. 11931018, 11701588). We would like to thank the referee for helpful comments and suggestions which led to the improved version of the paper.

Author information

Authors and Affiliations

  1. School of Mathematics, Sun Yat-Sen University, Guangzhou, 510275, China

    Fang Chen & Xianping Guo

  2. Beijing Normal University, Zhuhai, 519087, China

    Zhong-Wei Liao

Authors
  1. Fang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xianping Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhong-Wei Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhong-Wei Liao.

Additional information

Communicated by Jörg Rambau.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, F., Guo, X. & Liao, ZW. Optimal Stopping Time on Semi-Markov Processes with Finite Horizon. J Optim Theory Appl 194, 408–439 (2022). https://doi.org/10.1007/s10957-022-02026-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-022-02026-x

Keywords

Mathematics Subject Classification

Search

Navigation