SCC-Based Improved Reachability Analysis for Markov Decision Processes

  • Lin Gui
  • Jun Sun
  • Songzheng Song
  • Yang Liu
  • Jin Song Dong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8829)


Markov decision processes (MDPs) are extensively used to model systems with both probabilistic and nondeterministic behavior. The problem of calculating the probability of reaching certain system states (hereafter reachability analysis) is central to the MDP-based system analysis. It is known that existing approaches on reachability analysis for MDPs are often inefficient when a given MDP contains a large number of states and loops, especially with the existence of multiple probability distributions. In this work, we propose a method to eliminate strongly connected components (SCCs) in an MDP using a divide-and-conquer algorithm, and actively remove redundant probability distributions in the MDP based on the convex property. With the removal of loops and parts of probability distributions, the probabilistic reachability analysis can be accelerated, as evidenced by our experiment results.


Convex Hull Model Check Markov Decision Process Discrete Time Markov Chain Reachability Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Abrahám, E., Jansen, N., Wimmer, R., Katoen, J., Becker, B.: DTMC model checking by SCC reduction. In: QEST, pp. 37–46. IEEE (2010)Google Scholar
  3. 3.
    Andrés, M.E., D’Argenio, P.R., Rossum, P.V.: Significant diagnostic counterexamples in probabilistic model checking. In: HCV, pp. 129–148 (2008)Google Scholar
  4. 4.
    Baier, C., Katoen, J.: Principles of model checking. The MIT Press (2008)Google Scholar
  5. 5.
    Bentley, J.L., Preparata, F.P., Faust, M.G.: Approximation algorithms for convex hulls. Communications of the ACM 25(1), 64–68 (1982)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Cattani, S., Segala, R.: Decision algorithms for probabilistic bisimulation. In: Brim, L., Jančar, P., Křetínský, M., Kučera, A. (eds.) CONCUR 2002. LNCS, vol. 2421, pp. 371–386. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  7. 7.
    Ciesinski, F., Baier, C., Grosser, M., Klein, J.: Reduction techniques for model checking Markov decision processes. In: QEST, pp. 45–54. IEEE (2008)Google Scholar
  8. 8.
    De Berg, M., Van Kreveld, M., Overmars, M., Schwarzkopf, O.C.: Computational geometry. Springer, Heidelberg (2000)CrossRefMATHGoogle Scholar
  9. 9.
    Forejt, V., Kwiatkowska, M., Norman, G., Parker, D.: Automated verification techniques for probabilistic systems. In: FMENSS, pp. 53–113. Springer, Heidelberg (2011)Google Scholar
  10. 10.
    Gui, L., Sun, J., Liu, Y., Si, Y.J., Dong, J.S., Wang, X.Y.: Combining model checking and testing with an application to reliability prediction and distribution. In: ISSTA, pp. 101–111. ACM (2013)Google Scholar
  11. 11.
    Kwiatkowska, M., Parker, D., Qu, H.: Incremental quantitative verification for Markov decision processes. In: DSN, pp. 359–370. IEEE (2011)Google Scholar
  12. 12.
    Liu, Y., Gui, L., Liu, Y.: MDP-based reliability analysis of an ambient assisted living system. In: FM Industry Track, Singapore (May 2014)Google Scholar
  13. 13.
    Song, S., Gui, L., Sun, J., Liu, Y., Dong, J.S.: Improved reachability analysis in DTMC via divide and conquer. In: Johnsen, E.B., Petre, L. (eds.) IFM 2013. LNCS, vol. 7940, pp. 162–176. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  14. 14.
    Stewart, W.J.: Introduction to the numerical solution of Markov chains. Princeton University Press (1994)Google Scholar
  15. 15.
    Sun, J., Liu, Y., Dong, J.S., Pang, J.: PAT: Towards flexible verification under fairness. In: Bouajjani, A., Maler, O. (eds.) CAV 2009. LNCS, vol. 5643, pp. 709–714. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  16. 16.
    Tarjan, R.E.: Depth-first search and linear graph algorithms. SIAM J. Comput. 1(2), 146–160 (1972)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Lin Gui
    • 1
  • Jun Sun
    • 2
  • Songzheng Song
    • 3
  • Yang Liu
    • 3
  • Jin Song Dong
    • 1
  1. 1.National University of SingaporeSingapore
  2. 2.Singapore University of Technology and DesignSingapore
  3. 3.Nanyang Technological UniversitySingapore

Personalised recommendations