SCC-Based Improved Reachability Analysis for Markov Decision Processes

  • Lin Gui
  • Jun Sun
  • Songzheng Song
  • Yang Liu
  • Jin Song Dong
Conference paper

DOI: 10.1007/978-3-319-11737-9_12

Part of the Lecture Notes in Computer Science book series (LNCS, volume 8829)
Cite this paper as:
Gui L., Sun J., Song S., Liu Y., Dong J.S. (2014) SCC-Based Improved Reachability Analysis for Markov Decision Processes. In: Merz S., Pang J. (eds) Formal Methods and Software Engineering. ICFEM 2014. Lecture Notes in Computer Science, vol 8829. Springer, Cham

Abstract

Markov decision processes (MDPs) are extensively used to model systems with both probabilistic and nondeterministic behavior. The problem of calculating the probability of reaching certain system states (hereafter reachability analysis) is central to the MDP-based system analysis. It is known that existing approaches on reachability analysis for MDPs are often inefficient when a given MDP contains a large number of states and loops, especially with the existence of multiple probability distributions. In this work, we propose a method to eliminate strongly connected components (SCCs) in an MDP using a divide-and-conquer algorithm, and actively remove redundant probability distributions in the MDP based on the convex property. With the removal of loops and parts of probability distributions, the probabilistic reachability analysis can be accelerated, as evidenced by our experiment results.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Lin Gui
    • 1
  • Jun Sun
    • 2
  • Songzheng Song
    • 3
  • Yang Liu
    • 3
  • Jin Song Dong
    • 1
  1. 1.National University of SingaporeSingapore
  2. 2.Singapore University of Technology and DesignSingapore
  3. 3.Nanyang Technological UniversitySingapore

Personalised recommendations