Advertisement

Formal Methods in System Design

, Volume 36, Issue 3, pp 246–280 | Cite as

A game-based abstraction-refinement framework for Markov decision processes

  • Mark Kattenbelt
  • Marta Kwiatkowska
  • Gethin Norman
  • David ParkerEmail author
Article

Abstract

In the field of model checking, abstraction refinement has proved to be an extremely successful methodology for combating the state-space explosion problem. However, little practical progress has been made in the setting of probabilistic verification. In this paper we present a novel abstraction-refinement framework for Markov decision processes (MDPs), which are widely used for modelling and verifying systems that exhibit both probabilistic and nondeterministic behaviour. Our framework comprises an abstraction approach based on stochastic two-player games, two refinement methods and an efficient algorithm for an abstraction-refinement loop. The key idea behind the abstraction approach is to maintain a separation between nondeterminism present in the original MDP and nondeterminism introduced during the abstraction process, each type being represented by a different player in the game. Crucially, this allows lower and upper bounds to be computed for the values of reachability properties of the MDP. These give a quantitative measure of the quality of the abstraction and form the basis of the corresponding refinement methods. We describe a prototype implementation of our framework and present experimental results demonstrating automatic generation of compact, yet precise, abstractions for a large selection of real-world case studies.

Keywords

Probabilistic verification Markov decision processes Abstraction Abstraction refinement 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Baier C, Kwiatkowska M (1998) Model checking for a probabilistic branching time logic with fairness. Distrib Comput 11(3):125–155 CrossRefGoogle Scholar
  2. 2.
    Bertsekas D, Tsitsiklis J (1991) An analysis of stochastic shortest path problems. Math Oper Res 16(3):580–595 zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Billingsley P (1979) Probability and measure. Wiley, New York zbMATHGoogle Scholar
  4. 4.
    Chadha R, Viswanathan M (2010) A counterexample guided abstraction-refinement framework for Markov decision processes. ACM Trans Comput Logic (to appear) Google Scholar
  5. 5.
    Chatterjee K, de Alfaro L, Henzinger T (2004) Trading memory for randomness. In: Proc. 1st int. conf. quantitative evaluation of systems (QEST’04). IEEE Comput. Soc., Los Alamitos, pp 206–217 CrossRefGoogle Scholar
  6. 6.
    Chatterjee K, Henzinger T, Jhala R, Majumdar R (2005) Counterexample-guided planning. In: Proc. 21st conference in uncertainty in artificial intelligence (UAI’05), pp 104–111 Google Scholar
  7. 7.
    Cheshire S, Adoba B, Guttman E (2002) Dynamic configuration of IPv4 link-local addresses (draft August 2002). Zeroconf Working Group of the Internet Engineering Task Force (www.zeroconf.org)
  8. 8.
    Clarke E, Grumberg O, Jha S, Lu Y, Veith H (2000) Counterexample-guided abstraction refinement. In: Emerson A, Sistla A (eds) Proc. 12th int. conf. computer aided verification (CAV’00). Lecture notes in computer science, vol 1855. Springer, Berlin, pp 154–169 CrossRefGoogle Scholar
  9. 9.
    Condon A (1992) The complexity of stochastic games. Inf Comput 96(2):203–224 zbMATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Condon A (1993) On algorithms for simple stochastic games. Advances in computational complexity theory. DIMACS Ser Discrete Math Theor Comput Sci 13:51–73 MathSciNetGoogle Scholar
  11. 11.
    D’Argenio P, Jeannet B, Jensen H, Larsen K (2001) Reachability analysis of probabilistic systems by successive refinements. In: de Alfaro L, Gilmore S (eds) Proc. 1st joint int workshop process algebra and probabilistic methods, performance modelling and verification (PAPM/PROBMIV’01). Lecture notes in computer science, vol 2165. Springer, Berlin, pp 39–56 CrossRefGoogle Scholar
  12. 12.
    de Alfaro L (1999) Computing minimum and maximum reachability times in probabilistic systems. In: Baeten J, Mauw S (eds) Proc. 10th int. conf. concurrency theory (CONCUR’99). Lecture notes in computer science, vol 1664. Springer, Berlin, pp 66–81 CrossRefGoogle Scholar
  13. 13.
    de Alfaro L (1997) Formal verification of probabilistic systems. Ph.D. thesis, Stanford University Google Scholar
  14. 14.
    de Alfaro L, Roy P (2007) Magnifying-lens abstraction for Markov decision processes. In: Damm W, Hermanns H (eds) Proc. 19th int. conf. computer aided verification (CAV’07). Lecture notes in computer science, vol 4590. Springer, Berlin, pp 325–338 CrossRefGoogle Scholar
  15. 15.
    de Alfaro L, Henzinger T, Kupferman O (1998) Concurrent reachability games. In: Proc. 39th symp. foundations of computer science (FOCS’98). IEEE Comput. Soc., Los Alamitos, pp 564–575 Google Scholar
  16. 16.
    de Alfaro L, Henzinger T, Kupferman O (2007) Concurrent reachability games. Theor Comput Sci 386(3):188–217 zbMATHGoogle Scholar
  17. 17.
    Desharnais J, Gupta V, Jagadeesan R, Panangaden P (2003) Approximating labelled Markov processes. Inf Comput 184(1):160–200 zbMATHCrossRefMathSciNetGoogle Scholar
  18. 18.
    Fecher H, Leucker M, Wolf V (2006) Don’t know in probabilistic systems. In: Valmari A (ed) Proc. 13th int. spin workshop on model checking of software (SPIN’06). Lecture notes in computer science, vol 3925. Springer, Berlin, pp 71–88 Google Scholar
  19. 19.
    Graf S, Saidi H (1997) Construction of abstract state graphs with PVS. In: Grumberg O (ed) Proc. 9th int. conf. computer aided verification (CAV’97). Lecture notes in computer science, vol 1254. Springer, Berlin, pp 72–83 Google Scholar
  20. 20.
    Han T, Katoen JP, Damman B (2009) Counterexample generation in probabilistic model checking. IEEE Trans Softw Eng 35(2):241–257 CrossRefGoogle Scholar
  21. 21.
    Hermanns H, Wachter B, Zhang L (2008) Probabilistic CEGAR. In: Gupta A, Malik S (eds) Proc. 20th int. conf. computer aided verification (CAV’08). Lecture notes in computer science, vol 5123. Springer, Berlin, pp 162–175 CrossRefGoogle Scholar
  22. 22.
    Hinton A, Kwiatkowska M, Norman G, Parker D (2006) PRISM: A tool for automatic verification of probabilistic systems. In: Hermanns H, Palsberg J (eds) Proc. 12th int. conf. tools and algorithms for the construction and analysis of systems (TACAS’06). Lecture notes in computer science, vol 3920. Springer, Berlin, pp 441–444 CrossRefGoogle Scholar
  23. 23.
    Hurd J, McIver A, Morgan C (2005) Probabilistic guarded commands mechanized in HOL. Theor Comput Sci 346(1):96–112 zbMATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    Huth M (2004) An abstraction framework for mixed nondeterministic and probabilistic systems. In: Baier C, Haverkort B, Hermanns H, Katoen JP, Siegle M (eds) Validation of stochastic systems. Lecture notes in computer science, vol 2925. Springer, Berlin, pp 419–444 CrossRefGoogle Scholar
  25. 25.
    Kattenbelt M, Kwiatkowska M, Norman G, Parker D (2008) Game-based probabilistic predicate abstraction in PRISM. In: Proc. 6th workshop quantitative aspects of programming languages (QAPL’08) Google Scholar
  26. 26.
    Kattenbelt M, Kwiatkowska M, Norman G, Parker D (2009) Abstraction refinement for probabilistic software. In: Jones N, Muller-Olm M (eds) Proc. 10th int. conf. verification, model checking and abstract interpretation (VMCAI’09). Lecture notes in computer science, vol 5403. Springer, Berlin, pp 182–197 CrossRefGoogle Scholar
  27. 27.
    Kemeny J, Snell J, Knapp A (1976) Denumerable Markov chains, 2nd edn. Springer, Berlin zbMATHGoogle Scholar
  28. 28.
    Kwiatkowska M, Norman G, Parker D (2006) Game-based abstraction for Markov decision processes. In: Proc. 3th int. conf. quantitative evaluation of systems (QEST’06). IEEE Comput. Soc., Los Alamitos, pp 157–166 Google Scholar
  29. 29.
    Kwiatkowska M, Norman G, Parker D, Sproston J (2006) Performance analysis of probabilistic timed automata using digital clocks. Form Methods Syst Des 29:33–78 zbMATHCrossRefGoogle Scholar
  30. 30.
    Kwiatkowska M, Norman G, Parker D (2009) Stochastic games for verification of probabilistic timed automata. In: Ouaknine J, Vaandrager F (eds) Proc. 7th international conference on formal modelling and analysis of timed systems (FORMATS’09). Lecture notes in computer science, vol 5813. Springer, Berlin, pp 212–227 CrossRefGoogle Scholar
  31. 31.
    Larsen K, Skou A (1991) Bisimulation through probabilistic testing. Inf Comput 94:1–28 zbMATHCrossRefMathSciNetGoogle Scholar
  32. 32.
    McIver A, Morgan C (2004) Abstraction, refinement and proof for probabilistic systems. Monographs in computer science. Springer, Berlin Google Scholar
  33. 33.
    Monniaux D (2005) Abstract interpretation of programs as Markov decision processes. Sci Comput Program 58(1–2):179–205 zbMATHCrossRefMathSciNetGoogle Scholar
  34. 34.
    Norman G (2004) Analysing randomized distributed algorithms. In: Baier C, Haverkort B, Hermanns H, Katoen JP, Siegle M (eds) Validation of stochastic systems. Lecture notes in computer science, vol 2925. Springer, Berlin, pp 384–418 CrossRefGoogle Scholar
  35. 35.
    PASS tool homepage. http://depend.cs.uni-sb.de/PASS/
  36. 36.
    Pierro AD, Hankin C, Wiklicky H (2006) Abstract interpretation for worst and average case analysis. In: Reps T, Sagiv M, Bauer J (eds) Program analysis and compilation, theory and practice, essays dedicated to Reinhard Wilhelm on the occasion of his 60th birthday. Lecture notes in computer science, vol 4444. Springer, Berlin, pp 160–174 Google Scholar
  37. 37.
  38. 38.
    Segala R (1995) Modelling and verification of randomized distributed real time systems. Ph.D. thesis, Massachusetts Institute of Technology Google Scholar
  39. 39.
    Sen K, Viswanathan M, Agha G (2006) Model-checking Markov chains in the presence of uncertainties. In: Hermanns H, Palsberg J (eds) Proc. 12th int. conf. tools and algorithms for the construction and analysis of systems (TACAS’06). Lecture notes in computer science, vol 3920. Springer, Berlin, pp 394–410 CrossRefGoogle Scholar
  40. 40.
    Shapley L (1953) Stochastic games. In: Proc. national academy of science, vol 39, pp 1095–1100 Google Scholar
  41. 41.
    Smith M (2008) Probabilistic abstract interpretation of imperative programs using truncated normal distributions. In: Aldini A, Baier C (eds) Proc. 6th workshop on quantitative aspects of programming languages (QAPL’08). Electronic notes in theoretical computer science, vol 220(3). Elsevier, Dordrecht, pp 43–59 Google Scholar
  42. 42.
    Stoelinga M, Vaandrager F (1999) Root contention in IEEE 1394. In: Katoen JP (ed) Proc. 5th int. AMAST workshop real-time and probabilistic systems (ARTS’99). Lecture notes in computer science, vol 1601. Springer, Berlin, pp 53–74 Google Scholar
  43. 43.
    Wachter B, Zhang L (2010) Best probabilistic transformers. In: Barthe G, Hermenegildo M (eds) Proc. 11th int. conf. verification, model checking and abstract interpretation (VMCAI’10). Lecture notes in computer science, vol 5944. Springer, Berlin, pp 362–379 CrossRefGoogle Scholar
  44. 44.
    Wachter B, Zhang L, Hermanns H (2006) Probabilistic model checking modulo theories. In: Proc. 4th int. conf. quantitative evaluation of systems (QEST’07). IEEE Comput. Soc., Los Alamitos, pp 129–138 Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Mark Kattenbelt
    • 1
  • Marta Kwiatkowska
    • 1
  • Gethin Norman
    • 2
  • David Parker
    • 1
    Email author
  1. 1.Oxford University Computing LaboratoryOxfordUK
  2. 2.Department of Computing ScienceUniversity of GlasgowGlasgowScotland

Personalised recommendations