Explicit Model Checking of Very Large MDP Using Partitioning and Secondary Storage

  • Arnd HartmannsEmail author
  • Holger Hermanns
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9364)


The applicability of model checking is hindered by the state space explosion problem in combination with limited amounts of main memory. To extend its reach, the large available capacities of secondary storage such as hard disks can be exploited. Due to the specific performance characteristics of secondary storage technologies, specialised algorithms are required. In this paper, we present a technique to use secondary storage for probabilistic model checking of Markov decision processes. It combines state space exploration based on partitioning with a block-iterative variant of value iteration over the same partitions for the analysis of probabilistic reachability and expected-reward properties. A sparse matrix-like representation is used to store partitions on secondary storage in a compact format. All file accesses are sequential, and compression can be used without affecting runtime. The technique has been implemented within the Modest Toolset. We evaluate its performance on several benchmark models of up to 3.5 billion states. In the analysis of time-bounded properties on real-time models, our method neutralises the state space explosion induced by the time bound in its entirety.


Model Check Markov Decision Process Partition Graph Binary Decision Diagram Expected Reward 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Aggarwal, A., Vitter, J.S.: The input/output complexity of sorting and related problems. Commun. ACM 31(9), 1116–1127 (1988)MathSciNetCrossRefGoogle Scholar
  2. 2.
    de Alfaro, L., Kwiatkowska, M., Norman, G., Parker, D., Segala, R.: Symbolic model checking of probabilistic processes using MTBDDs and the kronecker representation. In: Graf, S. (ed.) TACAS 2000. LNCS, vol. 1785, pp. 395–410. Springer, Heidelberg (2000) CrossRefGoogle Scholar
  3. 3.
    Alur, R., Dill, D.L.: A theory of timed automata. Theor. Comput. Sci. 126(2), 183–235 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Baier, C., D’Argenio, P.R., Größer, M.: Partial order reduction for probabilistic branching time. Electron. Notes Theor. Comput. Sci. 153(2), 97–116 (2006)CrossRefGoogle Scholar
  5. 5.
    Bao, T., Jones, M.: Time-efficient model checking with magnetic disk. In: Halbwachs, N., Zuck, L.D. (eds.) TACAS 2005. LNCS, vol. 3440, pp. 526–540. Springer, Heidelberg (2005) CrossRefGoogle Scholar
  6. 6.
    Barnat, J., Brim, L., Šimeček, P.: I/O efficient accepting cycle detection. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 281–293. Springer, Heidelberg (2007) CrossRefGoogle Scholar
  7. 7.
    Bell, A., Haverkort, B.R.: Distributed disk-based algorithms for model checking very large Markov chains. Formal Methods Syst. Des. 29(2), 177–196 (2006)CrossRefzbMATHGoogle Scholar
  8. 8.
    Bohnenkamp, H.C., D’Argenio, P.R., Hermanns, H., Katoen, J.: MoDeST: a compositional modeling formalism for hard and softly timed systems. IEEE Trans. Softw. Eng. 32(10), 812–830 (2006)CrossRefGoogle Scholar
  9. 9.
    Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. MIT Press, Cambridge (1999)Google Scholar
  10. 10.
    Dai, P., Goldsmith, J.: Topological value iteration algorithm for Markov decision processes. In: IJCAI, pp. 1860–1865 (2007)Google Scholar
  11. 11.
    Deavours, D.D., Sanders, W.H.: An efficient disk-based tool for solving very large Markov models. In: Marie, R., Plateau, B., Calzarossa, M., Rubino, G. (eds.) Computer Performance Evaluation Modelling Techniques and Tools. LNCS, vol. 1245, pp. 58–71. Springer, Heidelberg (1997)CrossRefGoogle Scholar
  12. 12.
    Penna, G.D., Intrigila, B., Tronci, E., Zilli, M.V.: Exploiting transition locality in the disk based Mur\(\phi \) verifier. In: Aagaard, M.D., O’Leary, J.W. (eds.) Formal Methods in Computer-Aided Design. LNCS, vol. 2517, pp. 202–219. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Edelkamp, S., Jabbar, S.: Large-scale directed model checking LTL. In: Valmari, A. (ed.) SPIN 2006. LNCS, vol. 3925, pp. 1–18. Springer, Heidelberg (2006) CrossRefGoogle Scholar
  14. 14.
    Edelkamp, S., Jabbar, S., Bonet, B.: External memory value iteration. In: ICAPS, pp. 128–135. AAAI (2007)Google Scholar
  15. 15.
    Edelkamp, S., Sanders, P., Šimeček, P.: Semi-external LTL model checking. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 530–542. Springer, Heidelberg (2008) CrossRefGoogle Scholar
  16. 16.
    Evangelista, S., Kristensen, L.M.: Dynamic state space partitioning for external memory state space exploration. Sci. Comput. Program. 78(7), 778–795 (2013)CrossRefGoogle Scholar
  17. 17.
    Forejt, V., Kwiatkowska, M., Norman, G., Parker, D.: Automated verification techniques for probabilistic systems. In: Bernardo, M., Issarny, V. (eds.) SFM 2011. LNCS, vol. 6659, pp. 53–113. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  18. 18.
    Hahn, E.M., Hartmanns, A., Hermanns, H.: Reachability and reward checking for stochastic timed automata. ECEASST 70 (2014)Google Scholar
  19. 19.
    Hammer, M., Weber, M.: “To store or not to store” reloaded: reclaiming memory on demand. In: Brim, L., Haverkort, B.R., Leucker, M., van de Pol, J. (eds.) FMICS 2006 and PDMC 2006. LNCS, vol. 4346, pp. 51–66. Springer, Heidelberg (2007) CrossRefGoogle Scholar
  20. 20.
    Harrison, P.G., Knottenbelt, W.J.: Distributed disk-based solution techniques for large Markov models. In: Numerical Solution of Markov Chains, pp. 58–75 (1999)Google Scholar
  21. 21.
    Hartmanns, A., Hermanns, H.: A Modest approach to checking probabilistic timed automata. In: QEST, pp. 187–196. IEEE Computer Society (2009)Google Scholar
  22. 22.
    Hartmanns, A., Hermanns, H.: The Modest Toolset: an integrated environment for quantitative modelling and verification. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014 (ETAPS). LNCS, vol. 8413, pp. 593–598. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  23. 23.
    Hermanns, H., Wachter, B., Zhang, L.: Probabilistic CEGAR. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 162–175. Springer, Heidelberg (2008) CrossRefGoogle Scholar
  24. 24.
    Kwiatkowska, M.Z., Mehmood, R., Norman, G., Parker, D.: A symbolic out-of-core solution method for Markov models. Electron. Notes Theor. Comput. Sci. 68(4), 589–604 (2002)CrossRefzbMATHGoogle Scholar
  25. 25.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  26. 26.
    Kwiatkowska, M.Z., Norman, G., Parker, D., Sproston, J.: Performance analysis of probabilistic timed automata using digital clocks. Formal Methods Syst. Des. 29(1), 33–78 (2006)CrossRefzbMATHGoogle Scholar
  27. 27.
    Kwiatkowska, M.Z., Norman, G., Segala, R., Sproston, J.: Automatic verification of real-time systems with discrete probability distributions. Theoretical Comput. Sci. 282(1), 101–150 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Kwiatkowska, M., Norman, G., Sproston, J.: Probabilistic model checking of the IEEE 802.11 wireless local area network protocol. In: Hermanns, H., Segala, R. (eds.) PROBMIV 2002, PAPM-PROBMIV 2002, and PAPM 2002. LNCS, vol. 2399, pp. 169–187. Springer, Heidelberg (2002) CrossRefGoogle Scholar
  29. 29.
    LZ4. Accessed 2 July 2015
  30. 30.
    Mehmood, R.: Serial disk-based analysis of large stochastic models. In: Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.-P., Siegle, M. (eds.) Validation of Stochastic Systems. LNCS, vol. 2925, pp. 230–255. Springer, Heidelberg (2004) CrossRefGoogle Scholar
  31. 31.
    Norman, G., Parker, D., Sproston, J.: Model checking for probabilistic timed automata. Formal Methods Syst. Des. 43(2), 164–190 (2013)CrossRefzbMATHGoogle Scholar
  32. 32.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994) CrossRefzbMATHGoogle Scholar
  33. 33.
    Stern, U., Dill, D.L.: Using magnetic disk instead of main memory in the Mur\(\phi \) verifier. In: Hu, A.J., Vardi, M.Y. (eds.) Computer Aided Verification. LNCS, vol. 1427, pp. 172–183. Springer, Heidelberg (1998) CrossRefGoogle Scholar
  34. 34.
    Stewart, W.J.: Introduction to the Numerical Solution of Markov Chains. Princeton University Press, Princeton (1994) zbMATHGoogle Scholar
  35. 35.
    Timmer, M., Stoelinga, M., van de Pol, J.: Confluence reduction for probabilistic systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 311–325. Springer, Heidelberg (2011) CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Department of Computer ScienceSaarland UniversitySaarbrückenGermany

Personalised recommendations