Mean-Payoff Optimization in Continuous-Time Markov Chains with Parametric Alarms

  • Christel Baier
  • Clemens Dubslaff
  • Ľuboš KorenčiakEmail author
  • Antonín Kučera
  • Vojtěch Řehák
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10503)


Continuous-time Markov chains with alarms (ACTMCs) allow for alarm events that can be non-exponentially distributed. Within parametric ACTMCs, the parameters of alarm-event distributions are not given explicitly and can be subject of parameter synthesis. An algorithm solving the \(\varepsilon \)-optimal parameter synthesis problem for parametric ACTMCs with long-run average optimization objectives is presented. Our approach is based on reduction of the problem to finding long-run average optimal strategies in semi-Markov decision processes (semi-MDPs) and sufficient discretization of parameter (i.e., action) space. Since the set of actions in the discretized semi-MDP can be very large, a straightforward approach based on explicit action-space construction fails to solve even simple instances of the problem. The presented algorithm uses an enhanced policy iteration on symbolic representations of the action space. The soundness of the algorithm is established for parametric ACTMCs with alarm-event distributions satisfying four mild assumptions that are shown to hold for uniform, Dirac, exponential, and Weibull distributions in particular, but are satisfied for many other distributions as well. An experimental implementation shows that the symbolic technique substantially improves the efficiency of the synthesis algorithm and allows to solve instances of realistic size.


  1. 1.
    Alur, R., Henzinger, T.A., Vardi, M.Y.: Parametric real-time reasoning. In: STOC, pp. 592–601. ACM (1993)Google Scholar
  2. 2.
    Amparore, E.G., Buchholz, P., Donatelli, S.: A structured solution approach for Markov regenerative processes. In: Norman, G., Sanders, W. (eds.) QEST 2014. LNCS, vol. 8657, pp. 9–24. Springer, Cham (2014). doi: 10.1007/978-3-319-10696-0_3 Google Scholar
  3. 3.
    Baier, C., Dubslaff, C., Korenčiak, Ľ., Kučera, A., Řehák, V.: Mean-payoff optimization in continuous-time Markov chains with parametric alarms. CoRR, abs/1706.06486 (2017)Google Scholar
  4. 4.
    Bernardin, L., et al.: Maple 16 Programming Guide (2012)Google Scholar
  5. 5.
    Bertsekas, D.P., Gallager, R.G.: Data Networks, 2nd edn. Prentice-Hall International, Upper Saddle River (1992)zbMATHGoogle Scholar
  6. 6.
    Brázdil, T., Korenčiak, Ľ., Krčál, J., Novotný, P., Řehák, V.: Optimizing performance of continuous-time stochastic systems using timeout synthesis. In: Campos, J., Haverkort, B.R. (eds.) QEST 2015. LNCS, vol. 9259, pp. 141–159. Springer, Cham (2015). doi: 10.1007/978-3-319-22264-6_10 CrossRefGoogle Scholar
  7. 7.
    Brázdil, T., Krčál, J., Křetínský, J., Řehák, V.: Fixed-delay events in generalized semi-Markov processes revisited. In: Katoen, J.-P., König, B. (eds.) CONCUR 2011. LNCS, vol. 6901, pp. 140–155. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-23217-6_10 CrossRefGoogle Scholar
  8. 8.
    Češka, M., Dannenberg, F., Kwiatkowska, M., Paoletti, N.: Precise parameter synthesis for stochastic biochemical systems. In: Mendes, P., Dada, J.O., Smallbone, K. (eds.) CMSB 2014. LNCS, vol. 8859, pp. 86–98. Springer, Cham (2014). doi: 10.1007/978-3-319-12982-2_7 Google Scholar
  9. 9.
    Choi, H., Kulkarni, V.G., Trivedi, K.S.: Markov regenerative stochastic Petri nets. Perform. Eval. 20(1–3), 337–357 (1994)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Alfaro, L.: Stochastic transition systems. In: Sangiorgi, D., Simone, R. (eds.) CONCUR 1998. LNCS, vol. 1466, pp. 423–438. Springer, Heidelberg (1998). doi: 10.1007/BFb0055639 Google Scholar
  11. 11.
    Diciolla, M., Kim, C.H.P., Kwiatkowska, M., Mereacre, A.: Synthesising optimal timing delays for timed I/O automata. In: EMSOFT, pp. 1–10. ACM (2014)Google Scholar
  12. 12.
    German, R.: Performance Analysis of Communication Systems with Non-Markovian Stochastic Petri Nets. Wiley, Hoboken (2000)zbMATHGoogle Scholar
  13. 13.
    Haas, P.J.: Stochastic Petri Nets: Modelling, Stability, Simulation. Springer, New York (2010)zbMATHGoogle Scholar
  14. 14.
    Haase, C., Kreutzer, S., Ouaknine, J., Worrell, J.: Reachability in succinct and parametric one-counter automata. In: Bravetti, M., Zavattaro, G. (eds.) CONCUR 2009. LNCS, vol. 5710, pp. 369–383. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-04081-8_25 CrossRefGoogle Scholar
  15. 15.
    Hahn, E.M., Hermanns, H., Zhang, L.: Probabilistic reachability for parametric Markov models. STTT 13(1), 3–19 (2011)CrossRefGoogle Scholar
  16. 16.
    Han, T., Katoen, J.P., Mereacre, A.: Approximate parameter synthesis for probabilistic time-bounded reachability. In: RTSS, pp. 173–182. IEEE (2008)Google Scholar
  17. 17.
    Jha, S.K., Langmead, C.J.: Synthesis and infeasibility analysis for stochastic models of biochemical systems using statistical model checking and abstraction refinement. TCS 412(21), 2162–2187 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Jovanović, A., Kwiatkowska, M.: Parameter synthesis for probabilistic timed automata using stochastic game abstractions. In: Ouaknine, J., Potapov, I., Worrell, J. (eds.) RP 2014. LNCS, vol. 8762, pp. 176–189. Springer, Cham (2014). doi: 10.1007/978-3-319-11439-2_14 Google Scholar
  19. 19.
    Jovanovic, A., Lime, D., Roux, O.H.: Integer parameter synthesis for real-time systems. IEEE Trans. Softw. Eng. 41(5), 445–461 (2015)CrossRefGoogle Scholar
  20. 20.
    Korenčiak, Ľ., Kučera, A., Řehák, V.: Efficient timeout synthesis in fixed-delay CTMC using policy iteration. In: MASCOTS, pp. 367–372. IEEE (2016)Google Scholar
  21. 21.
    Korenčiak, Ľ., Řehák, V., Farmadin, A.: Extension of PRISM by synthesis of optimal timeouts in fixed-delay CTMC. In: iFM, pp. 130–138 (2016)Google Scholar
  22. 22.
    Lindemann, C.: An improved numerical algorithm for calculating steady-state solutions of deterministic and stochastic Petri net models. Perform. Eval. 18(1), 79–95 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Martina, S., Paolieri, M., Papini, T., Vicario, E.: Performance evaluation of Fischer’s protocol through steady-state analysis of Markov regenerative processes. In: MASCOTS, pp. 355–360. IEEE (2016)Google Scholar
  24. 24.
    Minh, D.L.P., Minh, D.D.L., Nguyen, A.L.: Regenerative Markov chain Monte Carlo for any distribution. Commun. Stat.-Simul. C. 41(9), 1745–1760 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Nelson, W.: Weibull analysis of reliability data with few or no failures. J. Qual. Technol. 17, 140–146 (1985)Google Scholar
  26. 26.
    Norris, J.R.: Markov Chains. Cambridge University Press, Cambridge (1998)zbMATHGoogle Scholar
  27. 27.
    Puterman, M.L.: Markov Decision Processes. Wiley, Hoboken (1994)CrossRefzbMATHGoogle Scholar
  28. 28.
    Qiu, Q., Wu, Q., Pedram, M.: Stochastic modeling of a power-managed system: construction and optimization. In: ISLPED, pp. 194–199. ACM Press (1999)Google Scholar
  29. 29.
    Traonouez, L.-M., Lime, D., Roux, O.H.: Parametric model-checking of time Petri nets with stopwatches using the state-class graph. In: Cassez, F., Jard, C. (eds.) FORMATS 2008. LNCS, vol. 5215, pp. 280–294. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-85778-5_20 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Christel Baier
    • 1
  • Clemens Dubslaff
    • 1
  • Ľuboš Korenčiak
    • 2
    Email author
  • Antonín Kučera
    • 2
  • Vojtěch Řehák
    • 2
  1. 1.TU DresdenDresdenGermany
  2. 2.Masaryk UniversityBrnoCzech Republic

Personalised recommendations