Scalable Verification of Markov Decision Processes

  • Axel Legay
  • Sean Sedwards
  • Louis-Marie Traonouez
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8938)


Markov decision processes (MDP) are useful to model concurrent process optimisation problems, but verifying them with numerical methods is often intractable. Existing approximative approaches do not scale well and are limited to memoryless schedulers. Here we present the basis of scalable verification for MDPSs, using an \(\mathcal {O}(1)\) memory representation of history-dependent schedulers. We thus facilitate scalable learning techniques and the use of massively parallel verification.



This work was partially supported by the European Union Seventh Framework Programme under grant agreement no. 295261 (MEALS).


  1. 1.
    PLASMA project web page.
  2. 2.
    Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press, Cambridge (2008)zbMATHGoogle Scholar
  3. 3.
    Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)zbMATHGoogle Scholar
  4. 4.
    Bianco, A., De Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Thiagarajan, P.S. (ed.) FSTTCS 1995. LNCS, vol. 1026. Springer, Heidelberg (1995)CrossRefGoogle Scholar
  5. 5.
    Bogdoll, J., Ferrer Fioriti, L.M., Hartmanns, A., Hermanns, H.: Partial order methods for statistical model checking and simulation. In: Bruni, R., Dingel, J. (eds.) FORTE 2011 and FMOODS 2011. LNCS, vol. 6722, pp. 59–74. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  6. 6.
    Chernoff, H.: A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. Ann. Math. Statist. 23(4), 493–507 (1952)CrossRefzbMATHMathSciNetGoogle Scholar
  7. 7.
    Clarke, E., Emerson, E.A., Sifakis, J.: Model checking: algorithmic verification and debugging. Commun. ACM 52(11), 74–84 (2009)CrossRefGoogle Scholar
  8. 8.
    Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 3rd edn. MIT Press, New York (2009)zbMATHGoogle Scholar
  9. 9.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects Comput. 6(5), 512–535 (1994)CrossRefzbMATHGoogle Scholar
  10. 10.
    Hartmanns, A., Timmer, M.: On-the-fly confluence detection for statistical model checking. In: Brat, G., Rungta, N., Venet, A. (eds.) NFM 2013. LNCS, vol. 7871, pp. 337–351. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  11. 11.
    Henriques, D., Martins, J.G., Zuliani, P., Platzer, A., Clarke, E.M.: Statistical model checking for Markov decision processes. In: 2012 Ninth International Conference on Quantitative Evaluation of Systems, pp. 84–93. IEEE (2012)Google Scholar
  12. 12.
    Horner, W.G.: A new method of solving numerical equations of all orders, by continuous approximation. Philos. Trans. R. Soc. Lond. 109, 308–335 (1819)CrossRefGoogle Scholar
  13. 13.
    Kearns, M., Mansour, Y., Ng, A.Y.: A sparse sampling algorithm for near-optimal planning in large Markov decision processes. Mach. Learn. 49(2–3), 193–208 (2002)CrossRefzbMATHGoogle Scholar
  14. 14.
    Knuth, D.E.: The Art of Computer Programming, 3rd edn. Addison-Wesley, Reading (1998)Google Scholar
  15. 15.
    Kwiatkowska, M., Norman, G., Sproston, J.: Probabilistic model checking of the IEEE 802.11 wireless local area network protocol. In: Hermanns, H., Segala, R. (eds.) PROBMIV 2002, PAPM-PROBMIV 2002, and PAPM 2002. LNCS, vol. 2399, pp. 169–187. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  16. 16.
    Lassaigne, R., Peyronnet, S.: Approximate planning and verification for large Markov decision processes. In: Proceddings of 27th Annual ACM Symposium on Applied Computing, pp. 1314–1319. ACM (2012)Google Scholar
  17. 17.
    Ndukwu, U., McIver, A.: An expectation transformer approach to predicate abstraction and data independence for probabilistic programs. In: Proceedings of 8th Workshop on Quantitative Aspects of Programming Languages (QAPL’10) (2010)Google Scholar
  18. 18.
    Okamoto, M.: Some inequalities relating to the partial sum of binomial probabilities. Ann. Inst. Stat. Math. 10(1), 29–35 (1958)CrossRefzbMATHGoogle Scholar
  19. 19.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley-Interscience, New York (1994)CrossRefzbMATHGoogle Scholar
  20. 20.
    Wald, A.: Sequential tests of statistical hypotheses. Ann. Math. Stat. 16(2), 117–186 (1945)CrossRefzbMATHMathSciNetGoogle Scholar
  21. 21.
    Younes, H.L.S., Simmons, R.G.: Probabilistic verification of discrete event systems using acceptance sampling. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 223–235. Springer, Heidelberg (2002)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Axel Legay
    • 1
  • Sean Sedwards
    • 1
  • Louis-Marie Traonouez
    • 1
  1. 1.Inria Rennes – Bretagne AtlantiqueRennesFrance

Personalised recommendations