Journal of Automated Reasoning

, Volume 59, Issue 3, pp 345–387 | Cite as

Markov Chains and Markov Decision Processes in Isabelle/HOL

  • Johannes HölzlEmail author


This paper presents an extensive formalization of Markov chains (MCs) and Markov decision processes (MDPs), with discrete time and (possibly infinite) discrete state-spaces. The formalization takes a coalgebraic view on the transition systems representing MCs and constructs their trace spaces. On these trace spaces properties like fairness, reachability, and stationary distributions are formalized. Similar to MCs, MDPs are represented as transition systems with a construction for trace spaces. These trace spaces provide maximal and minimal expectation over all possible non-deterministic decisions. As applications we provide a certifier for finite reachability problems and we relate the denotational semantics and operational semantics of the probabilistic guarded command language. A distinctive feature of our formalization is the order-theoretic and coalgebraic view on our concepts: we view transition systems as coalgebras, we view traces as coinductive streams, we provide iterative computation rules for expectations, and we define many properties on traces as least or greatest fixed points.


Markov chains Markov decision processes Probabilistic guarded command language Probabilistic model checking Isabelle/HOL 



The author wants to thank Tobias Nipkow, Dmitriy Traytel, and Fabian Immler and the anonymous reviewers for suggesting many textual improvements.


  1. 1.
    Affeldt, R., Hagiwara, M., Sénizergues, J.: Formalization of Shannon’s theorems. J. Autom. Reason. 53(1), 63–103 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Audebaud, P., Paulin-Mohring, C.: Proofs of randomized algorithms in Coq. Sci. Comput. Program. 74(8), 568–589 (2009). (Special Issue on Mathematics of Program Construction (MPC 2006)) MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Avigad, J., Hölzl, J., Serafin, L.: A formally verified proof of the central limit theorem. CoRR arxiv:1405.7012 (2014)
  4. 4.
    Backhouse, R.C.: Galois connections and fixed point calculus. In: Backhouse, R.C., Crole, R.L., Gibbons, J. (eds.) Algebraic and Coalgebraic Methods in the Mathematics of Program Construction, LNCS, vol. 2297, pp. 89–148 (2000)Google Scholar
  5. 5.
    Baier, C.: On the algorithmic verification of probabilistic systems. Habilitation, Universität Mannheim (1998)Google Scholar
  6. 6.
    Baier, C., Katoen, J.P.: Principles of Model Checking. MIT Press, Cambridge (2008)zbMATHGoogle Scholar
  7. 7.
    Berg, M.: Formal verification of cryptographic security proofs. Ph.D. thesis, Saarland University (2013)Google Scholar
  8. 8.
    Blanchette, J.C., Hölzl, J., Lochbihler, A., Panny, L., Popescu, A., Traytel, D.: Truly modular (co)datatypes for Isabelle/HOL. In: Klein, G., Gamboa, R. (eds.) Interactive Theorem Proving (ITP 2014), LNCS, vol. 8558, pp. 93–110. Springer (2014)Google Scholar
  9. 9.
    Blanchette, J.C., Popescu, A., Traytel, D.: Unified classical logic completeness. In: Demri, S., Kapur, D., Weidenbach, C. (eds.) Automated Reasoning (IJCAR 2014), LNCS, vol. 8562, pp. 46–60. Springer (2014)Google Scholar
  10. 10.
    Blazy, S., Paulin-Mohring, C., Pichardie, D. (eds.): Interactive Theorem Proving (ITP 2013), LNCS, vol. 7998. Springer (2013)Google Scholar
  11. 11.
    Celiku, O., McIver, A.: Cost-based analysis of probabilistic programs mechanised in HOL. Nord. J. Comput. 11(2), 102–128 (2004)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Cock, D.: Verifying probabilistic correctness in Isabelle with pGCL. In: Cassez, F., Huuck, R., Klein, G., Schlich, B. (eds.) Systems Software Verification (SSV 2012), EPTCS, vol. 102, pp. 167–178 (2012)Google Scholar
  13. 13.
    Davey, B.A., Priestley, H.A.: Introduction to Lattices and Order, 2nd edn. Cambridge University Press, Cambridge (2002)CrossRefzbMATHGoogle Scholar
  14. 14.
    Daws, C.: Symbolic and parametric model checking of discrete-time Markov chains. In: Liu, Z., Araki, K. (eds.) Theoretical Aspects of Computing (ICTAC 2004), LNCS, vol. 3407, pp. 280–294 (2004)Google Scholar
  15. 15.
    de Alfaro, L.: Formal verification of probabilistic systems. Ph.D. thesis, Stanford University. Technical report STAN-CS-TR-98-1601 (1997)Google Scholar
  16. 16.
    Eberl, M., Hölzl, J., Nipkow, T.: A verified compiler for probability density functions. In: European Symposium on Programming (ESOP 2015), LNCS (2015)Google Scholar
  17. 17.
    Esparza, J., Kučera, A., Mayr, R.: Model checking probabilistic pushdown automata. In: Logic in Computer Science (LICS 2004), pp. 12–21 (2004)Google Scholar
  18. 18.
    Esparza, J., Lammich, P., Neumann, R., Nipkow, T., Schimpf, A., Smaus, J.: A fully verified executable LTL model checker. In: Sharygina, N., Veith, H. (eds.) Computer Aided Verification (CAV 2013), LNCS, vol. 8044, pp. 463–478. Springer (2013)Google Scholar
  19. 19.
    Etessami, K., Yannakakis, M.: Recursive Markov chains, stochastic grammars, and monotone systems of nonlinear equations. J. ACM 56(1), 1–66 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Giry, M.: A categorical approach to probability theory. In: Categorical Aspects of Topology and Analysis, Lecture Notes in Mathematics, vol. 915, pp. 68–85 (1982)Google Scholar
  21. 21.
    Gonthier, G., Norrish, M. (eds.): CPP 2013, LNCS, vol. 8307. Springer (2013)Google Scholar
  22. 22.
    Gouezel, S.: Ergodic theory. The Archive of Formal Proofs (Formal Proof Development). (2015)
  23. 23.
    Gretz, F., Katoen, J., McIver, A.: Operational versus weakest pre-expectation semantics for the probabilistic guarded command language. Perform. Eval. 73, 110–132 (2014)CrossRefGoogle Scholar
  24. 24.
    Haddad, S., Monmege, B.: Reachability in MDPS: refining convergence of value iteration. In: Reachability Problems (RP 2014), LNCS, vol. 8762, pp. 125–137 Springer (2014)Google Scholar
  25. 25.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Technical report SICS/R90013, Swedish Institute of Computer Science (1994)Google Scholar
  26. 26.
    Hölzl, J.: Construction and stochastic applications of measure spaces in higher-order logic. Ph.D. thesis, Technische Universität München (2013)Google Scholar
  27. 27.
    Hölzl, J.: Formalising semantics for expected running time of probabilistic programs. In: Blanchette, C.J., Merz, S. (eds.) Interactive Theorem Proving (ITP 2016), LNCS, vol. 9807, pp. 475–482. Springer (2016)Google Scholar
  28. 28.
    Hölzl, J., Heller, A.: Three chapters of measure theory in Isabelle/HOL. In: van Eekelen, M.C.J.D., Geuvers, H., Schmaltz, J., Wiedijk, F. (eds.) Interactive Theorem Proving (ITP 2011), LNCS, vol. 6898, pp. 135–151. Springer (2011)Google Scholar
  29. 29.
    Hölzl, J., Immler, F., Huffman, B.: Type classes and filters for mathematical analysis in Isabelle/HOL. In: Blazy et al. [10], pp. 279–294Google Scholar
  30. 30.
    Hölzl, J., Lochbihler, A., Traytel, D.: A formalized hierarchy of probabilistic system types (proof pearl). In: Urban, C., Zhang, X. (eds.) Interactive Theorem Proving (ITP 2015), LNCS, vol. 9236, pp. 203–220 (2015)Google Scholar
  31. 31.
    Hölzl, J., Nipkow, T.: Interactive verification of Markov chains: two distributed protocol case studies. In: Fahrenberg, U., Legay, A., Thrane, C. (eds.) Quantities in Formal Methods (QFM 2012), EPTCS, vol. 103(2012)Google Scholar
  32. 32.
    Hölzl, J., Nipkow, T.: Markov models. The Archive of Formal Proofs (Formal Proof Development). (2012)
  33. 33.
    Hölzl, J., Nipkow, T.: Verifying pCTL model checking. In: Flanagan, C., König, B. (eds.) Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2012), LNCS, vol. 7214, pp. 347–361 (2012)Google Scholar
  34. 34.
    Huffman, B., Kunčar, O.: Lifting and transfer: a modular design for quotients in Isabelle/HOL. In: Gonthier and Norrish [21], pp. 131–146Google Scholar
  35. 35.
    Hurd, J.: Formal verification of probabilistic algorithms. Ph.D. thesis, University of Cambridge (2002)Google Scholar
  36. 36.
    Hurd, J., McIver, A., Morgan, C.: Probabilistic guarded commands mechanized in HOL. Theor. Comput. Sci. 346(1), 96–112 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Katoen, J.P., Zapreev, I.S., Hahn, E.M., Hermanns, H., Jansen, D.N.: The ins and outs of the probabilistic model checker MRMC. Perform. Eval. 68, 90–104 (2011)CrossRefGoogle Scholar
  38. 38.
    Kwiatkowska, M., Norman, G., Parker, D.: Stochastic model checking. In: Bernardo, M., Hillston, J. (eds.) Formal Methods for the Design of Computer, Communication and Software Systems: Performance Evaluation (SFM 2007), LNCS, vol. 4486, pp. 220–270 (2007)Google Scholar
  39. 39.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Computer Aided Verification (CAV 2011), LNCS, vol. 6806, pp. 585–591 (2011)Google Scholar
  40. 40.
    Liu, L., Hasan, O., Aravantinos, V., Tahar, S.: Formal reasoning about classified Markov chains in HOL. In: Blazy et al. [10], pp. 295–310Google Scholar
  41. 41.
    Lochbihler, A.: Probabilistic functions and cryptographic oracles in higher order logic. In: ESOP, LNCS, vol. 9632, pp. 503–531. Springer (2016)Google Scholar
  42. 42.
    McIver, A., Morgan, C.: Abstraction, Refinement And Proof For Probabilistic Systems. Monographs in Computer Science. Springer, Berlin (2004)Google Scholar
  43. 43.
    Monniaux, D.: Abstract interpretation of programs as Markov decision processes. Sci. Comput. Program. 58(1–2), 179–205 (2005). (Special Issue on the Static Analysis Symposium (SAS 2003)) MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Paulson, L.C.: A fixedpoint approach to (co)inductive and (co)datatype definitions. In: Plotkin, G.D., Stirling, C., Tofte M. (eds.) Proof, Language, and Interaction, Essays in Honour of Robin Milner, pp. 187–212. The MIT Press (2000)Google Scholar
  45. 45.
    Petcher, A., Morrisett, G.: The foundational cryptography framework. In: POST, LNCS, vol. 9036, pp. 53–72. Springer (2015)Google Scholar
  46. 46.
    Pollard, D.: A Users’s Guide to Measure Theoretic Probability, Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge (2002)Google Scholar
  47. 47.
    Popescu, A., Hölzl, J., Nipkow, T.: Formalizing probabilistic noninterference. In: Gonthier and Norrish [21], pp. 259–275Google Scholar
  48. 48.
    Rand, R., Zdancewic, S.: VPHL: a verified partial-correctness logic for probabilistic programs. ENTCS 319, 351–367 (2015). doi: 10.1016/j.entcs.2015.12.021 MathSciNetzbMATHGoogle Scholar
  49. 49.
    Richter, S.: Formalizing integration theory with an application to probabilistic algorithms. In: TPHOLs, LNCS, vol. 3223, pp. 271–286. Springer (2004)Google Scholar
  50. 50.
    Trivedi, K.S.: Probability & Statistics with Reliability, Queuing, and Computer Science Applications. Prentice-Hall, Englewood Cliffs (1982)zbMATHGoogle Scholar
  51. 51.
    Woess, W.: Denumerable Markov Chains. European Mathematical Society, Warsaw (2009)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  1. 1.Department of InformaticsTechnical University of MunichGarchingGermany

Personalised recommendations