Skip to main content
SpringerLink
Log in

Smart sampling for lightweight verification of Markov decision processes

  • SMC
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

Markov decision processes (MDP) are useful to model optimisation problems in concurrent systems. To verify MDPs with efficient Monte Carlo techniques requires that their nondeterminism be resolved by a scheduler. Recent work has introduced the elements of lightweight techniques to sample directly from scheduler space, but finding optimal schedulers by simple sampling may be inefficient. Here we describe “smart” sampling algorithms that can make substantial improvements in performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://project.inria.fr/plasma-lab/.

  2. http://igrida.gforge.inria.fr.

  3. http://www.prismmodelchecker.org/casestudies/.

References

  1. Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press, Cambridge (2008)

  2. Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)

  3. Bianco, A., De Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Foundations of Software Technology and Theoretical Computer Science, pp. 499–513. Springer, New York (1995)

  4. Bogdoll, J., Fioriti, L.M.F., Hartmanns, A., Hermanns, H.: Partial order methods for statistical model checking and simulation. In: Formal Techniques for Distributed Systems, pp. 59–74. Springer, New York (2011)

  5. Boyer, B., Corre, K., Legay, A., Sedwards, S.: PLASMA-lab: a flexible, distributable statistical model checking library. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) Quantitative Evaluation of Systems. LNCS, vol. 8054, pp. 160–164. Springer, New York (2013)

  6. Brázdil, T., Chatterjee, K., Chmelík, M., Forejt, V., Křetínský, J., Kwiatkowska, M., Parker, D., Ujma, M.: Verification of Markov decision processes using learning algorithms. In: Cassez, F., Raskin, J.-F. (eds.) Automated Technology for Verification and Analysis. Lecture Notes in Computer Science, vol. 8837, pp. 98–114. Springer, New York (2014)

  7. Clarke, E.M., Emerson, E.A., Sifakis, J.: Model checking: algorithmic verification and debugging. Commun. ACM 52(11), 74–84 (2009)

    Article  Google Scholar 

  8. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 3rd edn. MIT Press, Cambridge (2009)

  9. Eisner, C., Fisman, D., Havlicek, J., Lustig, Y., McIsaac, A., Van Campenhout D.: Reasoning with temporal logic on truncated paths. In: Computer Aided Verification, pp. 27–39. Springer, New York (2003)

  10. Geilen, M.C.W.: On the construction of monitors for temporal logic properties. Electron. Notes Theor. Comput. Sci. 55(2), 181–199 (2001)

    Article  Google Scholar 

  11. Gerth, R., Peled, D., Vardi, M.Y., Wolper, P.: Simple on-the-fly automatic verification of linear temporal logic. In: Protocol Specification Testing and Verification, pp. 3–18. Chapman & Hall, London (1995)

  12. Giannakopoulou, D., Havelund, K.: Automata-based verification of temporal properties on running programs. In: Proceedings of 16th Annual International Conference on Automated Software Engineering (ASE’01), pp. 412–416. IEEE (2001)

  13. Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Form. Asp. Comput. 6(5), 512–535 (1994)

  14. Hartmanns, A., Timmer, M.: On-the-fly confluence detection for statistical model checking. In: NASA Formal Methods, pp. 337–351. Springer, New York (2013)

  15. Henriques, D., Martins, J.G., Zuliani, P., Platzer, A., Clarke Edmund, M.: Statistical model checking for Markov decision processes. In: Ninth International Conference on Quantitative Evaluation of Systems, pp. 84–93. IEEE (2012)

  16. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)

  17. Horner, W.G.: A new method of solving numerical equations of all orders, by continuous approximation. Philos. Trans. R. Soc. Lond. 109, 308–335 (1819)

  18. Israeli, A., Jalfon, M.: Token management schemes and random walks yield self-stabilizating mutual exclusion. In: Proceedings of 9th Annual ACM Symposium on Principles of Distributed Computing (PODC’90), pp. 119–131. ACM, New York (1990)

  19. Kearns, M., Mansour, Y., Ng, A.Y.: A sparse sampling algorithm for near-optimal planning in large Markov decision processes. Mach. Learn. 49(2–3), 193–208 (2002)

  20. Knuth, D.E.: The Art of Computer Programming, 3rd edn. Addison-Wesley, Menlo Park (1998)

  21. Kwiatkowska, M., Norman, G., Parker D.: Stochastic model checking. In: Bernardo, M., Hillston, J. (eds.) Formal Methods for the Design of Computer, Communication and Software Systems: Performance Evaluation (SFM’07). LNCS (Tutorial Volume), vol. 4486, pp. 220–270. Springer, New York (2007)

  22. Kwiatkowska, M., Norman, G., Parker, D.: Analysis of a gossip protocol in PRISM. SIGMETRICS Perform. Eval. Rev. 36(3), 17–22 (2008)

  23. Kwiatkowska, M., Norman, G., Parker, D., Sproston, J.: Performance analysis of probabilistic timed automata using digital clocks. Formal Methods in System Design 29, 33–78 (2006)

  24. Kwiatkowska, M., Norman, G., Parker, D., Vigliotti, M.G.: Probabilistic mobile ambients. Theor. Comput. Sci. 410(12–13), 1272–1303 (2009)

  25. Kwiatkowska, M.Z., Norman, G., Sproston, J.: Probabilistic model checking of the IEEE 802.11 wireless local area network protocol. In: Proceedings of \(2{\rm {nd}}\) Joint International Workshop on Process Algebra and Probabilistic Methods, Performance Modeling and Verification, pp. 169–187. Springer, New York (2002)

  26. Lassaigne, R., Peyronnet, S.: Approximate planning and verification for large Markov decision processes. In: Proceedings of \(27{\rm th}\) Annual ACM Symposium on Applied Computing, pp. 1314–1319. ACM, New York (2012)

  27. Legay, A., Sedwards, S., Traonouez, L.-M.: Scalable verification of Markov decision processes. In: 4th Workshop on Formal Methods in the Development of Software (FMDS’14), LNCS. Springer, New York (2014)

  28. Manna, Z., Pnueli, A.: Temporal Verification of Reactive Systems: Safety, vol. 2. Springer, New York (1995)

  29. Ndukwu, U., McIver, A.: An expectation transformer approach to predicate abstraction and data independence for probabilistic programs. In: Proceedings of 8th Workshop on Quantitative Aspects of Programming Languages (QAPL’10) (2010)

  30. Okamoto, M.: Some inequalities relating to the partial sum of binomial probabilities. Ann. Inst. Stat. Math. 10(1), 29–35 (1958)

  31. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley-Interscience, USA (1994)

  32. Wald, A.: Sequential tests of statistical hypotheses. Ann. Math. Stat.16(2), 117–186 (1945)

  33. White, D.J.: Real applications of Markov decision processes. Interfaces 15(6), 73–83 (1985)

  34. White, D.J.: Further real applications of Markov decision processes. Interfaces 18(5), 55–61 (1988)

  35. White, D.J.: A survey of applications of Markov decision processes. J. Oper. Res. Soc. 44(11), 1073–1096 (1993)

  36. Younes, H.L.S.: Verification and planning for stochastic processes with asynchronous events. PhD thesis, Carnegie Mellon University, Pittsburgh (2005)

  37. Younes, H.L.S., Simmons, R.G.: Probabilistic verification of discrete event systems using acceptance sampling. In: Computer Aided Verification, pp. 223–235. Springer, New York (2002)

Download references

Acknowledgments

We are grateful to Benoît Delahaye for useful prior discussions. This work was partially supported by the European Union Seventh Framework Programme under Grant Agreement No. 295261 (MEALS).

Author information

Authors and Affiliations

  1. Universidad Nacional de Córdoba, Córdoba, Argentina

    Pedro D’Argenio

  2. Inria Rennes-Bretagne Atlantique, Rennes, France

    Axel Legay, Sean Sedwards & Louis-Marie Traonouez

Authors
  1. Pedro D’Argenio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Axel Legay
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sean Sedwards
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Louis-Marie Traonouez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Axel Legay.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

D’Argenio, P., Legay, A., Sedwards, S. et al. Smart sampling for lightweight verification of Markov decision processes. Int J Softw Tools Technol Transfer 17, 469–484 (2015). https://doi.org/10.1007/s10009-015-0383-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-015-0383-0

Keywords

Search

Navigation