Ensuring the Reliability of Your Model Checker: Interval Iteration for Markov Decision Processes

  • Christel BaierEmail author
  • Joachim KleinEmail author
  • Linda LeuschnerEmail author
  • David ParkerEmail author
  • Sascha WunderlichEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10426)


Probabilistic model checking provides formal guarantees on quantitative properties such as reliability, performance or risk, so the accuracy of the numerical results that it returns is critical. However, recent results have shown that implementations of value iteration, a widely used iterative numerical method for computing reachability probabilities, can return results that are incorrect by several orders of magnitude. To remedy this, interval iteration, which instead converges simultaneously from both above and below, has been proposed. In this paper, we present interval iteration techniques for computing expected accumulated weights (or costs), a considerably broader class of properties. This relies on an efficient, mainly graph-based method to determine lower and upper bounds for extremal expected accumulated weights. To offset the additional effort of dual convergence, we also propose topological interval iteration, which increases efficiency using a model decomposition into strongly connected components. Finally, we present a detailed experimental evaluation, which highlights inaccuracies in standard benchmarks, rather than just artificial examples, and illustrates the feasibility of our techniques.


  1. 1.
    Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.: Model-checking algorithms for continuous-time Markov chains. IEEE Trans. Softw. Eng. 29(6), 524–541 (2003)CrossRefzbMATHGoogle Scholar
  2. 2.
    Baier, C., Klein, J., Klüppelholz, S., Wunderlich, S.: Weight monitoring with linear temporal logic: complexity and decidability. In: 23rd Conference on Computer Science Logic and the 29th Symposium on Logic in Computer Science (CSL-LICS), pp. 11:1–11:10. ACM (2014)Google Scholar
  3. 3.
    Baier, C., Klein, J., Leuschner, L., Parker, D., Wunderlich, S.: Ensuring the reliability of your model checker: interval iteration for Markov decision processes (extended version) (2017).
  4. 4.
    Baier, C., Kwiatkowska, M.Z.: Model checking for a probabilistic branching time logic with fairness. Distrib. Comput. 11(3), 125–155 (1998)CrossRefGoogle Scholar
  5. 5.
    Bell, A., Haverkort, B.R.: Untold horrors about steady-state probabilities: what reward-based measures won’t tell about the equilibrium distribution. In: Wolter, K. (ed.) EPEW 2007. LNCS, vol. 4748, pp. 2–17. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-75211-0_2 CrossRefGoogle Scholar
  6. 6.
    Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)zbMATHGoogle Scholar
  7. 7.
    Bertsekas, D.P., Tsitsiklis, J.N.: An analysis of stochastic shortest path problems. Math. Oper. Res. 16(3), 580–595 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Bianco, A., de Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Thiagarajan, P.S. (ed.) FSTTCS 1995. LNCS, vol. 1026, pp. 499–513. Springer, Heidelberg (1995). doi: 10.1007/3-540-60692-0_70 CrossRefGoogle Scholar
  9. 9.
    Brázdil, T., Chatterjee, K., Chmelík, M., Forejt, V., Křetínský, J., Kwiatkowska, M., Parker, D., Ujma, M.: Verification of Markov decision processes using learning algorithms. In: Cassez, F., Raskin, J.-F. (eds.) ATVA 2014. LNCS, vol. 8837, pp. 98–114. Springer, Cham (2014). doi: 10.1007/978-3-319-11936-6_8 Google Scholar
  10. 10.
    Chades, I., Chapron, G., Cros, M., Garcia, F., Sabbadin, R.: MDPtoolbox: a multi-platform toolbox to solve stochastic dynamic programming problems. Ecography 37, 916–920 (2014)CrossRefGoogle Scholar
  11. 11.
    Ciesinski, F., Baier, C., Größer, M., Klein, J.: Reduction techniques for model checking Markov decision processes. In: 5th International Conference on Quantitative Evaluation of Systems (QEST), pp. 45–54. IEEE Computer Society Press (2008)Google Scholar
  12. 12.
    Courcoubetis, C., Yannakakis, M.: The complexity of probabilistic verification. J. ACM 42(4), 857–907 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Dai, P., Goldsmith, J.: Topological value iteration algorithm for Markov decision processes. In: 20th International Joint Conference on Artificial Intelligence (IJCAI), pp. 1860–1865 (2007)Google Scholar
  14. 14.
    Dai, P., Mausam, M., Weld, D.S., Goldsmith, J.: Topological value iteration algorithms. J. Artif. Intell. Res. (JAIR) 42, 181–209 (2011)MathSciNetzbMATHGoogle Scholar
  15. 15.
    de Alfaro, L.: Formal verification of probabilistic systems. Ph.D. thesis, Department of Computer Science. Stanford University (1997)Google Scholar
  16. 16.
    de Alfaro, L.: Computing minimum and maximum reachability times in probabilistic systems. In: Baeten, J.C.M., Mauw, S. (eds.) CONCUR 1999. LNCS, vol. 1664, pp. 66–81. Springer, Heidelberg (1999). doi: 10.1007/3-540-48320-9_7 CrossRefGoogle Scholar
  17. 17.
    Giro, S.: Optimal schedulers vs optimal bases: an approach for efficient exact solving of Markov decision processes. Theor. Comput. Sci. 538, 70–83 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Haddad, S., Monmege, B.: Reachability in MDPs: refining convergence of value iteration. In: Ouaknine, J., Potapov, I., Worrell, J. (eds.) RP 2014. LNCS, vol. 8762, pp. 125–137. Springer, Cham (2014). doi: 10.1007/978-3-319-11439-2_10 Google Scholar
  19. 19.
    Hahn, E.M., Li, Y., Schewe, S., Turrini, A., Zhang, L.: iscasMc: a web-based probabilistic model checker. In: Jones, C., Pihlajasaari, P., Sun, J. (eds.) FM 2014. LNCS, vol. 8442, pp. 312–317. Springer, Cham (2014). doi: 10.1007/978-3-319-06410-9_22 CrossRefGoogle Scholar
  20. 20.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Form. Asp. Comput. 6, 512–535 (1994)CrossRefzbMATHGoogle Scholar
  21. 21.
    Howard, R.: Dynamic Programming and Markov Processes. MIT Press, Cambridge (1960)zbMATHGoogle Scholar
  22. 22.
    Kallenberg, L.: Markov decision processes. Lecture Notes, University of Leiden (2011)Google Scholar
  23. 23.
    Katoen, J.-P., Zapreev, I.S., Hahn, E.M., Hermanns, H., Jansen, D.N.: The ins and outs of the probabilistic model checker MRMC. Perform. Eval. 68(2), 90–104 (2011)CrossRefGoogle Scholar
  24. 24.
    Kuvaiskii, D., Faqeh, R., Bhatotia, P., Felber, P., Fetzer, C.: HAFT: hardware-assisted fault tolerance. In: 11th European Conference on Computer Systems (EuroSys), pp. 25:1–25:17. ACM (2016)Google Scholar
  25. 25.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-22110-1_47 CrossRefGoogle Scholar
  26. 26.
    Kwiatkowska, M.Z., Norman, G., Parker, D.: The PRISM benchmark suite. In: 9th International Conference on Quantitative Evaluation of SysTems (QEST), pp. 203–204. IEEE Computer Society (2012)Google Scholar
  27. 27.
    McMahan, H.B., Likhachev, M., Gordon, G.J.: Bounded real-time dynamic programming: RTDP with monotone upper bounds and performance guarantees. In: 22nd International Conference on Machine Learning (ICML), vol. 119, pp. 569–576. ACM (2005)Google Scholar
  28. 28.
    Parker, D.: Implementation of symbolic model checking for probabilistic systems. Ph.D. thesis, University of Birmingham (2002)Google Scholar
  29. 29.
    Puterman, M., Shin, M.: Modified policy iteration algorithms for discounted Markov decision problems. Manag. Sci. 24, 1127–1137 (1978)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (1994)CrossRefzbMATHGoogle Scholar
  31. 31.
    Wimmer, R., Kortus, A., Herbstritt, M., Becker, B.: Probabilistic model checking and reliability of results. In: 11th IEEE Workshop on Design and Diagnostics of Electronic Circuits and Systems (DDECS), pp. 207–212. IEEE Computer Society (2008)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Technische Universität DresdenDresdenGermany
  2. 2.School of Computer ScienceUniversity of BirminghamBirminghamUK

Personalised recommendations