Probabilistic Model Checking of Labelled Markov Processes via Finite Approximate Bisimulations

  • Alessandro Abate
  • Marta Kwiatkowska
  • Gethin Norman
  • David Parker
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8464)


This paper concerns labelled Markov processes (LMPs), probabilistic models over uncountable state spaces originally introduced by Prakash Panangaden and colleagues. Motivated by the practical application of the LMP framework, we study its formal semantics and the relationship to similar models formulated in control theory. We consider notions of (exact and approximate) probabilistic bisimulation over LMPs and, drawing on methods from both formal verification and control theory, propose a simple technique to compute an approximate probabilistic bisimulation of a given LMP, where the resulting abstraction is characterised as a finite-state labelled Markov chain (LMC). This construction enables the application of automated quantitative verification and policy synthesis techniques over the obtained abstract model, which can be used to perform approximate analysis of the concrete LMP. We illustrate this process through a case study of a multi-room heating system that employs the probabilistic model checker PRISM.


Model Check Markov Decision Process Atomic Proposition Probabilistic Model Check Default Action 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abate, A.: A contractivity approach for probabilistic bisimulations of diffusion processes. In: Proc. 48th IEEE Conf. Decision and Control, Shanghai, China, pp. 2230–2235 (December 2009)Google Scholar
  2. 2.
    Abate, A., D’Innocenzo, A., Di Benedetto, M.D.: Approximate abstractions of stochastic hybrid systems. IEEE Transactions on Automatic Control 56(11), 2688–2694 (2011), doi:10.1109/TAC.2011.2160595MathSciNetCrossRefGoogle Scholar
  3. 3.
    Abate, A., Katoen, J.-P., Lygeros, J., Prandini, M.: Approximate model checking of stochastic hybrid systems. European Journal of Control 16, 624–641 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Abate, A., Prandini, M.: Approximate abstractions of stochastic systems: A randomized method. In: 2011 50th IEEE Conference on Decision and Control and European Control Conference (CDC-ECC), pp. 4861–4866 (2011)Google Scholar
  5. 5.
    Abate, A., Prandini, M., Lygeros, J., Sastry, S.: Probabilistic reachability and safety for controlled discrete time stochastic hybrid systems. Automatica 44(11), 2724–2734 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Baier, C., Katoen, J.-P.: Principles of model checking. The MIT Press (2008)Google Scholar
  7. 7.
    Bertsekas, D.P., Shreve, S.E.: Stochastic optimal control: The discrete time case, vol. 139. Academic Press (1978)Google Scholar
  8. 8.
    Bianco, A., de Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Thiagarajan, P.S. (ed.) FSTTCS 1995. LNCS, vol. 1026, pp. 499–513. Springer, Heidelberg (1995)CrossRefGoogle Scholar
  9. 9.
    Bouchard-Cote, A., Ferns, N., Panangaden, P., Precup, D.: An approximation algorithm for labelled Markov processes: towards realistic approximation. In: Proc. 2nd Int. Conf. Quantitative Evaluation of Systems, QEST 2005 (2005)Google Scholar
  10. 10.
    Cattani, S., Segala, R., Kwiatkowska, M., Norman, G.: Stochastic transition systems for continuous state spaces and non-determinism. In: Sassone, V. (ed.) FOSSACS 2005. LNCS, vol. 3441, pp. 125–139. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  11. 11.
    Chaput, P., Danos, V., Panangaden, P., Plotkin, G.: Approximating markov processes by averaging. In: Albers, S., Marchetti-Spaccamela, A., Matias, Y., Nikoletseas, S., Thomas, W. (eds.) ICALP 2009, Part II. LNCS, vol. 5556, pp. 127–138. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  12. 12.
    Comanici, G., Panangaden, P., Precup, D.: On-the-fly algorithms for bisimulation metrics. In: Proc. 9th Int. Conf. Quantitative Evaluation of Systems (QEST 2012), pp. 681–692 (2012)Google Scholar
  13. 13.
    Danos, V., Desharnais, J., Panangaden, P.: Labelled markov processes: Stronger and faster approximations. Electr. Notes Theor. Comput. Sci. 87, 157–203 (2004)CrossRefzbMATHGoogle Scholar
  14. 14.
    Desharnais, J., Edalat, A., Panangaden, P.: Bisimulation for labelled Markov processes. Information and Computation 179(2), 163–193 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Desharnais, J., Gupta, V., Jagadeesan, R., Panangaden, P.: Metrics for labelled Markov processes. Theoretical Computer Science 318(3), 323–354 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Desharnais, J., Jagadeesan, R., Gupta, V., Panangaden, P.: The metric analogue of weak bisimulation for probabilistic processes. In: Proc. 17th Annual IEEE Symp. Logic in Computer Science (LICS 2002), pp. 413–422 (2002)Google Scholar
  17. 17.
    Desharnais, J., Laviolette, F., Tracol, M.: Approximate analysis of probabilistic processes: logic, simulation and games. In: Proc. 5th Int. Conf. Quantitative Evaluation of SysTems (QEST 2008), pp. 264–273 (September 2008)Google Scholar
  18. 18.
    Desharnais, J., Panangaden, P.: Continuous stochastic logic characterizes bisimulation of continuous-time Markov processes. The Journal of Logic and Algebraic Programming 56(1-2), 99–115 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Desharnais, J., Panangaden, P., Jagadeesan, R., Gupta, V.: Approximating labeled Markov processes. In: Proc. 15th Annual IEEE Symp. Logic in Computer Science (LICS 2000), pp. 95–105 (2000)Google Scholar
  20. 20.
    D’Innocenzo, A., Abate, A., Katoen, J.-P.: Robust PCTL model checking. In: Proc. 15th ACM Int. Conf. Hybrid Systems: computation and control, Beijing, PRC, pp. 275–285 (April 2012)Google Scholar
  21. 21.
    Esmaeil Zadeh Soudjani, S., Abate, A.: Adaptive and sequential gridding procedures for the abstraction and verification of stochastic processes. SIAM Journal on Applied Dynamical Systems 12(2), 921–956 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Fehnker, A., Ivančić, F.: Benchmarks for hybrid systems verification. In: Alur, R., Pappas, G.J. (eds.) HSCC 2004. LNCS, vol. 2993, pp. 326–341. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  23. 23.
    Ferns, N., Panangaden, P., Precup, D.: Metrics for finite Markov decision processes. In: Proc. 20th Conf. Uncertainty in Artificial Intelligence (UAI 2004), Banff, Canada, pp. 162–169 (2004)Google Scholar
  24. 24.
    Ferns, N., Panangaden, P., Precup, D.: Metrics for Markov decision processes with infinite state spaces. In: Proc. 21st Conf. Uncertainty in Artificial Intelligence, UAI 2005 (2005)Google Scholar
  25. 25.
    Ferns, N., Panangaden, P., Precup, D.: Bisimulation metrics for continuous Markov decision processes. SIAM Journal of Computing 60(4), 1662–1724 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Forejt, V., Kwiatkowska, M., Norman, G., Parker, D.: Automated verification techniques for probabilistic systems. In: Bernardo, M., Issarny, V. (eds.) SFM 2011. LNCS, vol. 6659, pp. 53–113. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  27. 27.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects of Computing 6(5), 512–535 (1994)CrossRefzbMATHGoogle Scholar
  28. 28.
    Hernández-Lerma, O., Lasserre, J.B.: Discrete-time Markov control processes, Applications of Mathematics, vol. 30. Springer, New York (1996)CrossRefzbMATHGoogle Scholar
  29. 29.
    Huth, M.: On finite-state approximants for probabilistic computation tree logic. Theoretical Computer Science 346(1), 113–134 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Huth, M., Kwiatkowska, M.Z.: Quantitative analysis and model checking. In: 12th Annual IEEE Symp. Logic in Computer Science, pp. 111–122. IEEE Computer Society (1997)Google Scholar
  31. 31.
    Julius, A.A., Pappas, G.J.: Approximations of stochastic hybrid systems. IEEE Transactions on Automatic Control 54(6), 1193–1203 (2009)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: Verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  33. 33.
    Larsen, K.G., Mardare, R., Panangaden, P.: Taking it to the limit: Approximate reasoning for markov processes. In: Rovan, B., Sassone, V., Widmayer, P. (eds.) MFCS 2012. LNCS, vol. 7464, pp. 681–692. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  34. 34.
    Larsen, K.G., Skou, A.: Bisimulation through probabilistic testing. Information and Computation 94, 1–28 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Malhame, R., Chong, C.-Y.: Electric load model synthesis by diffusion approximation of a high-order hybrid-state stochastic system. IEEE Transactions on Automatic Control 30(9), 854–860 (1985)CrossRefzbMATHGoogle Scholar
  36. 36.
    Meyn, S.P., Tweedie, R.L.: Markov chains and stochastic stability. Communications and Control Engineering Series. Springer, London (1993)CrossRefzbMATHGoogle Scholar
  37. 37.
    Panangaden, P.: Labelled Markov Processes. Imperial College Press (2009)Google Scholar
  38. 38.
    Puterman, M.: Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, Inc. (1994)Google Scholar
  39. 39.
    Tkachev, I., Abate, A.: On infinite-horizon probabilistic properties and stochastic bisimulation functions. In: 2011 50th IEEE Conf. Decision and Control and European Control Conference (CDC-ECC), pp. 526–531 (2011)Google Scholar
  40. 40.
    Tkachev, I., Abate, A.: Characterization and computation of infinite horizon specifications over Markov processes. Theoretical Computer Science 515, 1–18 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Tkachev, I., Abate, A.: Formula-free finite abstractions for linear temporal verification of stochastic hybrid systems. In: Proc. 16th ACM Int. Conf. Hybrid Systems: Computation and Control, pp. 283–292 (2013)Google Scholar
  42. 42.
    Tkachev, I., Abate, A.: On approximation metrics for linear temporal model-checking of stochastic systems. In: Proc. 17th ACM Int. Conf. Hybrid Systems: Computation and Control (2014)Google Scholar
  43. 43.
    van Breugel, F., Worrell, J.: An algorithm for quantitative verification of probabilistic transition systems. In: Larsen, K.G., Nielsen, M. (eds.) CONCUR 2001. LNCS, vol. 2154, pp. 336–350. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  44. 44.
    van Breugel, F., Worrell, J.: Towards quantitative verification of probabilistic transition systems. In: Orejas, F., Spirakis, P.G., van Leeuwen, J. (eds.) ICALP 2001. LNCS, vol. 2076, pp. 421–432. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  45. 45.
    Wimmer, R., Becker, B.: Correctness issues of symbolic bisimulation computation for Markov chains. In: Proc. 15th Int. GI/ITG Conf. Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance (MMB-DFT), pp. 287–301. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  46. 46.
    Zamani, M., Esfahani, P.M., Majumdar, R., Abate, A., Lygeros, J.: Symbolic control of stochastic systems via approximately bisimilar finite abstractions. arXiv: 1302.3868 (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Alessandro Abate
    • 1
  • Marta Kwiatkowska
    • 1
  • Gethin Norman
    • 2
  • David Parker
    • 3
  1. 1.Department of Computer ScienceUniversity of OxfordUK
  2. 2.School of Computing ScienceUniversity of GlasgowUK
  3. 3.School of Computer ScienceUniversity of BirminghamUK

Personalised recommendations