Smart sampling for lightweight verification of Markov decision processes

  • Pedro D’Argenio
  • Axel Legay
  • Sean Sedwards
  • Louis-Marie Traonouez


Markov decision processes (MDP) are useful to model optimisation problems in concurrent systems. To verify MDPs with efficient Monte Carlo techniques requires that their nondeterminism be resolved by a scheduler. Recent work has introduced the elements of lightweight techniques to sample directly from scheduler space, but finding optimal schedulers by simple sampling may be inefficient. Here we describe “smart” sampling algorithms that can make substantial improvements in performance.


Statistical model checking Sampling Nondeterminism 



We are grateful to Benoît Delahaye for useful prior discussions. This work was partially supported by the European Union Seventh Framework Programme under Grant Agreement No. 295261 (MEALS).


  1. 1.
    Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press, Cambridge (2008)Google Scholar
  2. 2.
    Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)Google Scholar
  3. 3.
    Bianco, A., De Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Foundations of Software Technology and Theoretical Computer Science, pp. 499–513. Springer, New York (1995)Google Scholar
  4. 4.
    Bogdoll, J., Fioriti, L.M.F., Hartmanns, A., Hermanns, H.: Partial order methods for statistical model checking and simulation. In: Formal Techniques for Distributed Systems, pp. 59–74. Springer, New York (2011)Google Scholar
  5. 5.
    Boyer, B., Corre, K., Legay, A., Sedwards, S.: PLASMA-lab: a flexible, distributable statistical model checking library. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) Quantitative Evaluation of Systems. LNCS, vol. 8054, pp. 160–164. Springer, New York (2013)Google Scholar
  6. 6.
    Brázdil, T., Chatterjee, K., Chmelík, M., Forejt, V., Křetínský, J., Kwiatkowska, M., Parker, D., Ujma, M.: Verification of Markov decision processes using learning algorithms. In: Cassez, F., Raskin, J.-F. (eds.) Automated Technology for Verification and Analysis. Lecture Notes in Computer Science, vol. 8837, pp. 98–114. Springer, New York (2014)Google Scholar
  7. 7.
    Clarke, E.M., Emerson, E.A., Sifakis, J.: Model checking: algorithmic verification and debugging. Commun. ACM 52(11), 74–84 (2009)CrossRefGoogle Scholar
  8. 8.
    Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 3rd edn. MIT Press, Cambridge (2009)Google Scholar
  9. 9.
    Eisner, C., Fisman, D., Havlicek, J., Lustig, Y., McIsaac, A., Van Campenhout D.: Reasoning with temporal logic on truncated paths. In: Computer Aided Verification, pp. 27–39. Springer, New York (2003)Google Scholar
  10. 10.
    Geilen, M.C.W.: On the construction of monitors for temporal logic properties. Electron. Notes Theor. Comput. Sci. 55(2), 181–199 (2001)CrossRefGoogle Scholar
  11. 11.
    Gerth, R., Peled, D., Vardi, M.Y., Wolper, P.: Simple on-the-fly automatic verification of linear temporal logic. In: Protocol Specification Testing and Verification, pp. 3–18. Chapman & Hall, London (1995)Google Scholar
  12. 12.
    Giannakopoulou, D., Havelund, K.: Automata-based verification of temporal properties on running programs. In: Proceedings of 16th Annual International Conference on Automated Software Engineering (ASE’01), pp. 412–416. IEEE (2001)Google Scholar
  13. 13.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Form. Asp. Comput. 6(5), 512–535 (1994)Google Scholar
  14. 14.
    Hartmanns, A., Timmer, M.: On-the-fly confluence detection for statistical model checking. In: NASA Formal Methods, pp. 337–351. Springer, New York (2013)Google Scholar
  15. 15.
    Henriques, D., Martins, J.G., Zuliani, P., Platzer, A., Clarke Edmund, M.: Statistical model checking for Markov decision processes. In: Ninth International Conference on Quantitative Evaluation of Systems, pp. 84–93. IEEE (2012)Google Scholar
  16. 16.
    Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)Google Scholar
  17. 17.
    Horner, W.G.: A new method of solving numerical equations of all orders, by continuous approximation. Philos. Trans. R. Soc. Lond. 109, 308–335 (1819)Google Scholar
  18. 18.
    Israeli, A., Jalfon, M.: Token management schemes and random walks yield self-stabilizating mutual exclusion. In: Proceedings of 9th Annual ACM Symposium on Principles of Distributed Computing (PODC’90), pp. 119–131. ACM, New York (1990)Google Scholar
  19. 19.
    Kearns, M., Mansour, Y., Ng, A.Y.: A sparse sampling algorithm for near-optimal planning in large Markov decision processes. Mach. Learn. 49(2–3), 193–208 (2002)Google Scholar
  20. 20.
    Knuth, D.E.: The Art of Computer Programming, 3rd edn. Addison-Wesley, Menlo Park (1998)Google Scholar
  21. 21.
    Kwiatkowska, M., Norman, G., Parker D.: Stochastic model checking. In: Bernardo, M., Hillston, J. (eds.) Formal Methods for the Design of Computer, Communication and Software Systems: Performance Evaluation (SFM’07). LNCS (Tutorial Volume), vol. 4486, pp. 220–270. Springer, New York (2007)Google Scholar
  22. 22.
    Kwiatkowska, M., Norman, G., Parker, D.: Analysis of a gossip protocol in PRISM. SIGMETRICS Perform. Eval. Rev. 36(3), 17–22 (2008)Google Scholar
  23. 23.
    Kwiatkowska, M., Norman, G., Parker, D., Sproston, J.: Performance analysis of probabilistic timed automata using digital clocks. Formal Methods in System Design 29, 33–78 (2006)Google Scholar
  24. 24.
    Kwiatkowska, M., Norman, G., Parker, D., Vigliotti, M.G.: Probabilistic mobile ambients. Theor. Comput. Sci. 410(12–13), 1272–1303 (2009)Google Scholar
  25. 25.
    Kwiatkowska, M.Z., Norman, G., Sproston, J.: Probabilistic model checking of the IEEE 802.11 wireless local area network protocol. In: Proceedings of \(2{\rm {nd}}\) Joint International Workshop on Process Algebra and Probabilistic Methods, Performance Modeling and Verification, pp. 169–187. Springer, New York (2002)Google Scholar
  26. 26.
    Lassaigne, R., Peyronnet, S.: Approximate planning and verification for large Markov decision processes. In: Proceedings of \(27{\rm th}\) Annual ACM Symposium on Applied Computing, pp. 1314–1319. ACM, New York (2012)Google Scholar
  27. 27.
    Legay, A., Sedwards, S., Traonouez, L.-M.: Scalable verification of Markov decision processes. In: 4th Workshop on Formal Methods in the Development of Software (FMDS’14), LNCS. Springer, New York (2014)Google Scholar
  28. 28.
    Manna, Z., Pnueli, A.: Temporal Verification of Reactive Systems: Safety, vol. 2. Springer, New York (1995)Google Scholar
  29. 29.
    Ndukwu, U., McIver, A.: An expectation transformer approach to predicate abstraction and data independence for probabilistic programs. In: Proceedings of 8th Workshop on Quantitative Aspects of Programming Languages (QAPL’10) (2010)Google Scholar
  30. 30.
    Okamoto, M.: Some inequalities relating to the partial sum of binomial probabilities. Ann. Inst. Stat. Math. 10(1), 29–35 (1958)Google Scholar
  31. 31.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley-Interscience, USA (1994)Google Scholar
  32. 32.
    Wald, A.: Sequential tests of statistical hypotheses. Ann. Math. Stat.16(2), 117–186 (1945)Google Scholar
  33. 33.
    White, D.J.: Real applications of Markov decision processes. Interfaces 15(6), 73–83 (1985)Google Scholar
  34. 34.
    White, D.J.: Further real applications of Markov decision processes. Interfaces 18(5), 55–61 (1988)Google Scholar
  35. 35.
    White, D.J.: A survey of applications of Markov decision processes. J. Oper. Res. Soc. 44(11), 1073–1096 (1993)Google Scholar
  36. 36.
    Younes, H.L.S.: Verification and planning for stochastic processes with asynchronous events. PhD thesis, Carnegie Mellon University, Pittsburgh (2005)Google Scholar
  37. 37.
    Younes, H.L.S., Simmons, R.G.: Probabilistic verification of discrete event systems using acceptance sampling. In: Computer Aided Verification, pp. 223–235. Springer, New York (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Pedro D’Argenio
    • 1
  • Axel Legay
    • 2
  • Sean Sedwards
    • 2
  • Louis-Marie Traonouez
    • 2
  1. 1.Universidad Nacional de CórdobaCórdobaArgentina
  2. 2.Inria Rennes-Bretagne AtlantiqueRennesFrance

Personalised recommendations