Skip to main content
SpringerLink
Log in

A game-based abstraction-refinement framework for Markov decision processes

  • Published:
Formal Methods in System Design Aims and scope Submit manuscript

Abstract

In the field of model checking, abstraction refinement has proved to be an extremely successful methodology for combating the state-space explosion problem. However, little practical progress has been made in the setting of probabilistic verification. In this paper we present a novel abstraction-refinement framework for Markov decision processes (MDPs), which are widely used for modelling and verifying systems that exhibit both probabilistic and nondeterministic behaviour. Our framework comprises an abstraction approach based on stochastic two-player games, two refinement methods and an efficient algorithm for an abstraction-refinement loop. The key idea behind the abstraction approach is to maintain a separation between nondeterminism present in the original MDP and nondeterminism introduced during the abstraction process, each type being represented by a different player in the game. Crucially, this allows lower and upper bounds to be computed for the values of reachability properties of the MDP. These give a quantitative measure of the quality of the abstraction and form the basis of the corresponding refinement methods. We describe a prototype implementation of our framework and present experimental results demonstrating automatic generation of compact, yet precise, abstractions for a large selection of real-world case studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Baier C, Kwiatkowska M (1998) Model checking for a probabilistic branching time logic with fairness. Distrib Comput 11(3):125–155

    Article  Google Scholar 

  2. Bertsekas D, Tsitsiklis J (1991) An analysis of stochastic shortest path problems. Math Oper Res 16(3):580–595

    Article  MATH  MathSciNet  Google Scholar 

  3. Billingsley P (1979) Probability and measure. Wiley, New York

    MATH  Google Scholar 

  4. Chadha R, Viswanathan M (2010) A counterexample guided abstraction-refinement framework for Markov decision processes. ACM Trans Comput Logic (to appear)

  5. Chatterjee K, de Alfaro L, Henzinger T (2004) Trading memory for randomness. In: Proc. 1st int. conf. quantitative evaluation of systems (QEST’04). IEEE Comput. Soc., Los Alamitos, pp 206–217

    Chapter  Google Scholar 

  6. Chatterjee K, Henzinger T, Jhala R, Majumdar R (2005) Counterexample-guided planning. In: Proc. 21st conference in uncertainty in artificial intelligence (UAI’05), pp 104–111

  7. Cheshire S, Adoba B, Guttman E (2002) Dynamic configuration of IPv4 link-local addresses (draft August 2002). Zeroconf Working Group of the Internet Engineering Task Force (www.zeroconf.org)

  8. Clarke E, Grumberg O, Jha S, Lu Y, Veith H (2000) Counterexample-guided abstraction refinement. In: Emerson A, Sistla A (eds) Proc. 12th int. conf. computer aided verification (CAV’00). Lecture notes in computer science, vol 1855. Springer, Berlin, pp 154–169

    Chapter  Google Scholar 

  9. Condon A (1992) The complexity of stochastic games. Inf Comput 96(2):203–224

    Article  MATH  MathSciNet  Google Scholar 

  10. Condon A (1993) On algorithms for simple stochastic games. Advances in computational complexity theory. DIMACS Ser Discrete Math Theor Comput Sci 13:51–73

    MathSciNet  Google Scholar 

  11. D’Argenio P, Jeannet B, Jensen H, Larsen K (2001) Reachability analysis of probabilistic systems by successive refinements. In: de Alfaro L, Gilmore S (eds) Proc. 1st joint int workshop process algebra and probabilistic methods, performance modelling and verification (PAPM/PROBMIV’01). Lecture notes in computer science, vol 2165. Springer, Berlin, pp 39–56

    Chapter  Google Scholar 

  12. de Alfaro L (1999) Computing minimum and maximum reachability times in probabilistic systems. In: Baeten J, Mauw S (eds) Proc. 10th int. conf. concurrency theory (CONCUR’99). Lecture notes in computer science, vol 1664. Springer, Berlin, pp 66–81

    Chapter  Google Scholar 

  13. de Alfaro L (1997) Formal verification of probabilistic systems. Ph.D. thesis, Stanford University

  14. de Alfaro L, Roy P (2007) Magnifying-lens abstraction for Markov decision processes. In: Damm W, Hermanns H (eds) Proc. 19th int. conf. computer aided verification (CAV’07). Lecture notes in computer science, vol 4590. Springer, Berlin, pp 325–338

    Chapter  Google Scholar 

  15. de Alfaro L, Henzinger T, Kupferman O (1998) Concurrent reachability games. In: Proc. 39th symp. foundations of computer science (FOCS’98). IEEE Comput. Soc., Los Alamitos, pp 564–575

    Google Scholar 

  16. de Alfaro L, Henzinger T, Kupferman O (2007) Concurrent reachability games. Theor Comput Sci 386(3):188–217

    MATH  Google Scholar 

  17. Desharnais J, Gupta V, Jagadeesan R, Panangaden P (2003) Approximating labelled Markov processes. Inf Comput 184(1):160–200

    Article  MATH  MathSciNet  Google Scholar 

  18. Fecher H, Leucker M, Wolf V (2006) Don’t know in probabilistic systems. In: Valmari A (ed) Proc. 13th int. spin workshop on model checking of software (SPIN’06). Lecture notes in computer science, vol 3925. Springer, Berlin, pp 71–88

    Google Scholar 

  19. Graf S, Saidi H (1997) Construction of abstract state graphs with PVS. In: Grumberg O (ed) Proc. 9th int. conf. computer aided verification (CAV’97). Lecture notes in computer science, vol 1254. Springer, Berlin, pp 72–83

    Google Scholar 

  20. Han T, Katoen JP, Damman B (2009) Counterexample generation in probabilistic model checking. IEEE Trans Softw Eng 35(2):241–257

    Article  Google Scholar 

  21. Hermanns H, Wachter B, Zhang L (2008) Probabilistic CEGAR. In: Gupta A, Malik S (eds) Proc. 20th int. conf. computer aided verification (CAV’08). Lecture notes in computer science, vol 5123. Springer, Berlin, pp 162–175

    Chapter  Google Scholar 

  22. Hinton A, Kwiatkowska M, Norman G, Parker D (2006) PRISM: A tool for automatic verification of probabilistic systems. In: Hermanns H, Palsberg J (eds) Proc. 12th int. conf. tools and algorithms for the construction and analysis of systems (TACAS’06). Lecture notes in computer science, vol 3920. Springer, Berlin, pp 441–444

    Chapter  Google Scholar 

  23. Hurd J, McIver A, Morgan C (2005) Probabilistic guarded commands mechanized in HOL. Theor Comput Sci 346(1):96–112

    Article  MATH  MathSciNet  Google Scholar 

  24. Huth M (2004) An abstraction framework for mixed nondeterministic and probabilistic systems. In: Baier C, Haverkort B, Hermanns H, Katoen JP, Siegle M (eds) Validation of stochastic systems. Lecture notes in computer science, vol 2925. Springer, Berlin, pp 419–444

    Chapter  Google Scholar 

  25. Kattenbelt M, Kwiatkowska M, Norman G, Parker D (2008) Game-based probabilistic predicate abstraction in PRISM. In: Proc. 6th workshop quantitative aspects of programming languages (QAPL’08)

  26. Kattenbelt M, Kwiatkowska M, Norman G, Parker D (2009) Abstraction refinement for probabilistic software. In: Jones N, Muller-Olm M (eds) Proc. 10th int. conf. verification, model checking and abstract interpretation (VMCAI’09). Lecture notes in computer science, vol 5403. Springer, Berlin, pp 182–197

    Chapter  Google Scholar 

  27. Kemeny J, Snell J, Knapp A (1976) Denumerable Markov chains, 2nd edn. Springer, Berlin

    MATH  Google Scholar 

  28. Kwiatkowska M, Norman G, Parker D (2006) Game-based abstraction for Markov decision processes. In: Proc. 3th int. conf. quantitative evaluation of systems (QEST’06). IEEE Comput. Soc., Los Alamitos, pp 157–166

    Google Scholar 

  29. Kwiatkowska M, Norman G, Parker D, Sproston J (2006) Performance analysis of probabilistic timed automata using digital clocks. Form Methods Syst Des 29:33–78

    Article  MATH  Google Scholar 

  30. Kwiatkowska M, Norman G, Parker D (2009) Stochastic games for verification of probabilistic timed automata. In: Ouaknine J, Vaandrager F (eds) Proc. 7th international conference on formal modelling and analysis of timed systems (FORMATS’09). Lecture notes in computer science, vol 5813. Springer, Berlin, pp 212–227

    Chapter  Google Scholar 

  31. Larsen K, Skou A (1991) Bisimulation through probabilistic testing. Inf Comput 94:1–28

    Article  MATH  MathSciNet  Google Scholar 

  32. McIver A, Morgan C (2004) Abstraction, refinement and proof for probabilistic systems. Monographs in computer science. Springer, Berlin

    Google Scholar 

  33. Monniaux D (2005) Abstract interpretation of programs as Markov decision processes. Sci Comput Program 58(1–2):179–205

    Article  MATH  MathSciNet  Google Scholar 

  34. Norman G (2004) Analysing randomized distributed algorithms. In: Baier C, Haverkort B, Hermanns H, Katoen JP, Siegle M (eds) Validation of stochastic systems. Lecture notes in computer science, vol 2925. Springer, Berlin, pp 384–418

    Chapter  Google Scholar 

  35. PASS tool homepage. http://depend.cs.uni-sb.de/PASS/

  36. Pierro AD, Hankin C, Wiklicky H (2006) Abstract interpretation for worst and average case analysis. In: Reps T, Sagiv M, Bauer J (eds) Program analysis and compilation, theory and practice, essays dedicated to Reinhard Wilhelm on the occasion of his 60th birthday. Lecture notes in computer science, vol 4444. Springer, Berlin, pp 160–174

    Google Scholar 

  37. PRISM web site. http://www.prismmodelchecker.org/

  38. Segala R (1995) Modelling and verification of randomized distributed real time systems. Ph.D. thesis, Massachusetts Institute of Technology

  39. Sen K, Viswanathan M, Agha G (2006) Model-checking Markov chains in the presence of uncertainties. In: Hermanns H, Palsberg J (eds) Proc. 12th int. conf. tools and algorithms for the construction and analysis of systems (TACAS’06). Lecture notes in computer science, vol 3920. Springer, Berlin, pp 394–410

    Chapter  Google Scholar 

  40. Shapley L (1953) Stochastic games. In: Proc. national academy of science, vol 39, pp 1095–1100

  41. Smith M (2008) Probabilistic abstract interpretation of imperative programs using truncated normal distributions. In: Aldini A, Baier C (eds) Proc. 6th workshop on quantitative aspects of programming languages (QAPL’08). Electronic notes in theoretical computer science, vol 220(3). Elsevier, Dordrecht, pp 43–59

    Google Scholar 

  42. Stoelinga M, Vaandrager F (1999) Root contention in IEEE 1394. In: Katoen JP (ed) Proc. 5th int. AMAST workshop real-time and probabilistic systems (ARTS’99). Lecture notes in computer science, vol 1601. Springer, Berlin, pp 53–74

    Google Scholar 

  43. Wachter B, Zhang L (2010) Best probabilistic transformers. In: Barthe G, Hermenegildo M (eds) Proc. 11th int. conf. verification, model checking and abstract interpretation (VMCAI’10). Lecture notes in computer science, vol 5944. Springer, Berlin, pp 362–379

    Chapter  Google Scholar 

  44. Wachter B, Zhang L, Hermanns H (2006) Probabilistic model checking modulo theories. In: Proc. 4th int. conf. quantitative evaluation of systems (QEST’07). IEEE Comput. Soc., Los Alamitos, pp 129–138

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Oxford University Computing Laboratory, Wolfson Building, Parks Road, Oxford, OX1 3QD, UK

    Mark Kattenbelt, Marta Kwiatkowska & David Parker

  2. Department of Computing Science, University of Glasgow, 18 Lilybank Gardens, Glasgow, G12 8RZ, Scotland

    Gethin Norman

Authors
  1. Mark Kattenbelt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marta Kwiatkowska
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gethin Norman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Parker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Parker.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kattenbelt, M., Kwiatkowska, M., Norman, G. et al. A game-based abstraction-refinement framework for Markov decision processes. Form Methods Syst Des 36, 246–280 (2010). https://doi.org/10.1007/s10703-010-0097-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10703-010-0097-6

Keywords

Search

Navigation