Bisimulations and Logical Characterizations on Continuous-Time Markov Decision Processes

  • Lei Song
  • Lijun Zhang
  • Jens Chr. Godskesen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8318)


In this paper we study strong and weak bisimulation equivalences for continuous-time Markov decision processes (CTMDPs) and the logical characterizations of these relations with respect to the continuous-time stochastic logic (CSL). For strong bisimulation, it is well known that it is strictly finer than the CSL equivalence. In this paper we propose strong and weak bisimulations for CTMDPs and show that for a subclass of CTMDPs, strong and weak bisimulations are both sound and complete with respect to the equivalences induced by CSL and the sub-logic of CSL without next operator respectively. We then consider a standard extension of CSL, and show that it and its sub-logic without X can be fully characterized by strong and weak bisimulations respectively over arbitrary CTMDPs.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alur, R., Henzinger, T.A.: A really temporal logic. J. ACM 41(1), 181–203 (1994)CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    Aziz, A., Sanwal, K., Singhal, V., Brayton, R.K.: Verifying continuous time Markov chains. In: Alur, R., Henzinger, T.A. (eds.) CAV 1996. LNCS, vol. 1102, pp. 269–276. Springer, Heidelberg (1996)CrossRefGoogle Scholar
  3. 3.
    Aziz, A., Singhal, V., Balarin, F., Brayton, R.K., Sangiovanni-Vincentelli, A.L.: It usually works: The temporal logic of stochastic systems. In: Wolper, P. (ed.) CAV 1995. LNCS, vol. 939, pp. 155–165. Springer, Heidelberg (1995)CrossRefGoogle Scholar
  4. 4.
    Baier, C., Haverkort, B., Hermanns, H., Katoen, J.-P.: Model-checking algorithms for continuous-time Markov chains. IEEE Trans. Softw. Eng. 29(6), 524–541 (2003)CrossRefGoogle Scholar
  5. 5.
    Baier, C., Hermanns, H., Katoen, J.-P., Haverkort, B.R.: Efficient computation of time-bounded reachability probabilities in uniform continuous-time Markov decision processes. Theor. Comput. Sci. 345(1), 2–26 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Baier, C., Katoen, J.-P., Hermanns, H., Wolf, V.: Comparative branching-time semantics for Markov chains. Inf. Comput. 200(2), 149–214 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  7. 7.
    Böde, E., Herbstritt, M., Hermanns, H., Johr, S., Peikenkamp, T., Pulungan, R., Wimmer, R., Becker, B.: Compositional performability evaluation for STATEMATE. In: QEST, pp. 167–178. IEEE (2006)Google Scholar
  8. 8.
    Bouyer, P., Markey, N., Ouaknine, J., Worrell, J.: The cost of punctuality. In: LICS, pp. 109–120. IEEE (2007)Google Scholar
  9. 9.
    Browne, M.C., Clarke, E.M., Grümberg, O.: Characterizing finite Kripke structures in propositional temporal logic. Theor. Comput. Sci. 59(1-2), 115–131 (1988)CrossRefzbMATHGoogle Scholar
  10. 10.
    Bruno, J., Downey, P., Frederickson, G.N.: Sequencing tasks with exponential service times to minimize the expected flow time or makespan. J. ACM 28(1), 100–113 (1981)CrossRefzbMATHMathSciNetGoogle Scholar
  11. 11.
    Buchholz, P., Hahn, E.M., Hermanns, H., Zhang, L.: Model checking algorithms for CTMDPs. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 225–242. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  12. 12.
    Buchholz, P., Schulz, I.: Numerical analysis of continuous time Markov decision processes over finite horizons. Computers & Operations Research 38(3), 651–659 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  13. 13.
    Chen, T., Diciolla, M., Kwiatkowska, M., Mereacre, A.: Time-bounded verification of CTMCs against real-time specifications. In: Fahrenberg, U., Tripakis, S. (eds.) FORMATS 2011. LNCS, vol. 6919, pp. 26–42. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  14. 14.
    D’Argenio, P.R., Wolovick, N., Terraf, P.S., Celayes, P.: Nondeterministic labeled Markov processes: Bisimulations and logical characterization. In: QEST, pp. 11–20. IEEE (2009)Google Scholar
  15. 15.
    Desharnais, J., Gupta, V., Jagadeesan, R., Panangaden, P.: Metrics for labelled Markov processes. Theor. Comput. Sci. 318(3), 323–354 (2004)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Desharnais, J., Gupta, V., Jagadeesan, R., Panangaden, P.: Weak bisimulation is sound and complete for pCTL*. Inf. Comput. 208(2), 203–219 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Desharnais, J., Panangaden, P.: Continuous stochastic logic characterizes bisimulation of continuous-time Markov processes. J. Log. Algebr. Program. 56(1-2), 99–115 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  18. 18.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects of Computing 6(5), 512–535 (1994)CrossRefzbMATHGoogle Scholar
  19. 19.
    Haverkort, B.R., Hermanns, H., Katoen, J.-P.: On the use of model checking techniques for dependability evaluation. In: SRDS, pp. 228–237 (2000)Google Scholar
  20. 20.
    Hermanns, H., Parma, A., Segala, R., Wachter, B., Zhang, L.: Probabilistic logical characterization. Inf. Comput. 209(2), 154–172 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  21. 21.
    Jenkins, M., Ouaknine, J., Rabinovich, A., Worrell, J.: Alternating timed automata over bounded time. In: LICS, pp. 60–69. IEEE (2010)Google Scholar
  22. 22.
    Jonsson, B., Larsen, K., Wang, Y.: Probabilistic extensions of process algebras. In: Bergstra, J., Ponse, A., Smolka, S. (eds.) Handbook of Process Algebra, pp. 685–710. Elsevier (2001)Google Scholar
  23. 23.
    Katoen, J.-P., Zapreev, I.S., Hahn, E.M., Hermanns, H., Jansen, D.N.: The ins and outs of the probabilistic model checker MRMC. In: QEST, pp. 167–176 (2009)Google Scholar
  24. 24.
    Koymans, R.: Specifying real-time properties with metric temporal logic. Real-Time Syst. 2(4), 255–299 (1990)CrossRefGoogle Scholar
  25. 25.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: Verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  26. 26.
    Neuhäußer, M.R., Katoen, J.-P.: Bisimulation and logical preservation for continuous-time Markov decision processes. In: Caires, L., Vasconcelos, V.T. (eds.) CONCUR 2007. LNCS, vol. 4703, pp. 412–427. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  27. 27.
    Neuhäußer, M.R., Stoelinga, M., Katoen, J.-P.: Delayed nondeterminism in continuous-time Markov decision processes. In: de Alfaro, L. (ed.) FOSSACS 2009. LNCS, vol. 5504, pp. 364–379. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  28. 28.
    Ouaknine, J., Worrell, J.: On the decidability of metric temporal logic. In: LICS, pp. 188–197. IEEE (2005)Google Scholar
  29. 29.
    Pnueli, A.: The temporal logic of programs. In: FOCS, pp. 46–57. IEEE (1977)Google Scholar
  30. 30.
    Rabe, M.N., Schewe, S.: Finite optimal control for time-bounded reachability in CTMDPs and continuous-time Markov games. Acta. Inf. 48(5-6), 291–315 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  31. 31.
    Segala, R., Lynch, N.A.: Probabilistic simulations for probabilistic processes. Nord. J. Comput. 2(2), 250–273 (1995)zbMATHMathSciNetGoogle Scholar
  32. 32.
    Sharma, A., Katoen, J.-P.: Weighted lumpability on Markov chains. In: Clarke, E., Virbitskaite, I., Voronkov, A. (eds.) PSI 2011. LNCS, vol. 7162, pp. 322–339. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  33. 33.
    Song, L., Zhang, L., Godskesen, J.C.: Bisimulations meet PCTL equivalences for probabilistic automata. In: Katoen, J.-P., König, B. (eds.) CONCUR 2011. LNCS, vol. 6901, pp. 108–123. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  34. 34.
    Song, L., Zhang, L., Godskesen, J.C.: The branching time spectrum for continuous-time mdps. CoRR, abs/1204.1848 (2012)Google Scholar
  35. 35.
    van Glabbeek, R.J.: The linear time - branching time spectrum ii. In: Best, E. (ed.) CONCUR 1993. LNCS, vol. 715, pp. 66–81. Springer, Heidelberg (1993)Google Scholar
  36. 36.
    van Glabbeek, R.J.: The linear time - branching time spectrum i. In: Bergstra, J., Ponse, A., Smolka, S. (eds.) Handbook of Process Algebra, pp. 3–99. Elsevier (2001)Google Scholar
  37. 37.
    Wolovick, N., Johr, S.: A characterization of meaningful schedulers for continuous-time Markov decision processes. In: Asarin, E., Bouyer, P. (eds.) FORMATS 2006. LNCS, vol. 4202, pp. 352–367. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  38. 38.
    Zhang, L., Hermanns, H., Eisenbrand, F., Jansen, D.N.: Flow faster: Efficient decision algorithms for probabilistic simulations. Logical Methods in Computer Science 4(4) (2008)Google Scholar
  39. 39.
    Zhang, L., Neuhäußer, M.R.: Model Checking Interactive Markov Chains. In: Esparza, J., Majumdar, R. (eds.) TACAS 2010. LNCS, vol. 6015, pp. 53–68. Springer, Heidelberg (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Lei Song
    • 1
  • Lijun Zhang
    • 2
  • Jens Chr. Godskesen
    • 3
  1. 1.Max-Planck-Institut für InformatikSaarland UniversitySaarbrückenGermany
  2. 2.State Key Laboratory of Computer Science, Institute of SoftwareChinese Academy of SciencesChina
  3. 3.Programming, Logic, and Semantics GroupIT University of CopenhagenDenmark

Personalised recommendations