Skip to main content

Scenario-Based Verification of Uncertain MDPs

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 12078)

Abstract

We consider Markov decision processes (MDPs) in which the transition probabilities and rewards belong to an uncertainty set parametrized by a collection of random variables. The probability distributions for these random parameters are unknown. The problem is to compute the probability to satisfy a temporal logic specification within any MDP that corresponds to a sample from these unknown distributions. In general, this problem is undecidable, and we resort to techniques from so-called scenario optimization. Based on a finite number of samples of the uncertain parameters, each of which induces an MDP, the proposed method estimates the probability of satisfying the specification by solving a finite-dimensional convex optimization problem. The number of samples required to obtain a high confidence on this estimate is independent from the number of states and the number of random parameters. Experiments on a large set of benchmarks show that a few thousand samples suffice to obtain high-quality confidence bounds with a high probability.

Keywords

  • MDP
  • Uncertainty
  • Verification
  • Scenario optimisation

References

  1. Abraham Charnes and William W Cooper. Chance-Constrained Programming. Management science, 6(1):73–79, 1959.

    Google Scholar 

  2. Amir Pnueli. The Temporal Logic of Programs. In FOCS, pages 46–57, 1977.

    Google Scholar 

  3. James Aspnes and Maurice Herlihy. Fast Randomized Consensus Using Shared Memory. Journal of Algorithms, 15(1):441–460, 1990.

    Google Scholar 

  4. Hans Hansson and Bengt Jonsson. A Logic for Reasoning about Time and Reliability. Formal Aspects of Computing, 6(5):512–535, 1994.

    Google Scholar 

  5. L. Helmink, M. Sellink, and F. Vaandrager. Proof-Checking a Data Link Protocol. In TYPES, volume 806 of LNCS, pages 127–165. Springer, 1994.

    Google Scholar 

  6. Anthony Cassandra, Michael L Littman, and Nevin L Zhang. Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes. In UAI, pages 54–61, 1997.

    Google Scholar 

  7. Robert Givan, Sonia Leach, and Thomas Dean. Bounded-Parameter Markov Decision Processes. Artificial Intelligence, 122(1-2):71–109, 2000.

    Google Scholar 

  8. Jie Han and Pieter Jonker. A System Architecture Solution for Unreliable Nanoelectronic Devices. IEEE Transactions on Nanotechnology, 1:201–208, 2002.

    Google Scholar 

  9. Christel Baier, Boudewijn R. Haverkort, Holger Hermanns, and Joost-Pieter Katoen. Model-Checking Algorithms for Continuous-Time Markov Chains. IEEE Trans. Software Eng., 29(6):524–541, 2003.

    Google Scholar 

  10. Joelle Pineau, Geoff Gordon, and Sebastian Thrun. Point-Based Value Iteration: an Anytime Algorithm for POMDPs. In IJCAI, pages 1025–1030, 2003.

    Google Scholar 

  11. Conrado Daws. Symbolic and Parametric Model Checking of Discrete-Time Markov chains. In ICTAC, volume 3407 of LNCS, pages 280–294. Springer, 2004.

    Google Scholar 

  12. Vitaly Shmatikov. Probabilistic Analysis of an Anonymity System. Journal of Computer Security, 12(3-4):355–377, 2004.

    Google Scholar 

  13. G.C. Calafiore and M.C. Campi. Uncertain Convex Programs: Randomized Solutions and Confidence Levels. Mathematical Programming, 102(1):25–46, 2005.

    Google Scholar 

  14. Arnab Nilim and Laurent El Ghaoui. Robust Control of Markov Decision Processes with Uncertain Transition Matrices. Operations Research, 53(5):780–798, 2005.

    Google Scholar 

  15. Giuseppe C. Calafiore and Marco C. Campi. The Scenario Approach to Robust Control Design. IEEE Trans. Automat. Contr., 51(5):742–753, 2006.

    Google Scholar 

  16. Marco C. Campi and Simone Garatti. The Exact Feasibility of Randomized Solutions of Uncertain Convex Programs. SIAM Journal on Optimization, 19(3):1211–1230, 2008.

    Google Scholar 

  17. Ernst Moritz Hahn, Holger Hermanns, and Lijun Zhang. Probabilistic Reachability for Parametric Markov Models. STTT, 13(1):3–19, 2010.

    Google Scholar 

  18. Dimitris Bertsimas, David B Brown, and Constantine Caramanis. Theory and Applications of Robust Optimization. SIAM review, 53(3):464–501, 2011.

    Google Scholar 

  19. Marco C Campi and Simone Garatti. A Sampling-and-Discarding Approach to Chance-Constrained Optimization: Feasibility and Optimality. Journal of Optimization Theory and Applications, 148(2):257–280, 2011.

    Google Scholar 

  20. Benoît Delahaye, Kim G. Larsen, Axel Legay, Mikkel L. Pedersen, and Andrzej Wasowski. Decision problems for interval Markov chains. In LATA, volume 6638 of LNCS, pages 274–285. Springer, 2011.

    Google Scholar 

  21. Marta Kwiatkowska, Gethin Norman, and David Parker. PRISM 4.0: Verification of Probabilistic Real-Time Systems. In CAV, volume 6806 of LNCS, pages 585–591. Springer, 2011.

    Google Scholar 

  22. Hassan Hatefi and Holger Hermanns. Model Checking Algorithms for Markov Automata. ECEASST, 53, 2012.

    Google Scholar 

  23. Marta Kwiatkowska, Gethin Norman, and David Parker. The PRISM Benchmark Suite. In QEST, pages 203–204. IEEE CS, 2012.

    Google Scholar 

  24. Rowan McAllister, Thierry Peynot, Robert Fitch, and Salah Sukkarieh. Motion Planning and Stochastic Control with Experimental Validation on a Planetary Rover. In IROS, pages 4716–4723. IEEE, 2012.

    Google Scholar 

  25. Eric M. Wolff, Ufuk Topcu, and Richard M. Murray. Robust Control of Uncertain Markov Decision Processes with Temporal Logic Specifications. In CDC, pages 3372–3379, 2012.

    Google Scholar 

  26. Taolue Chen, Ernst Moritz Hahn, Tingting Han, Marta Kwiatkowska, Hongyang Qu, and Lijun Zhang. Model Repair for Markov Decision Processes. In TASE, pages 85–92. IEEE CS, 2013.

    Google Scholar 

  27. Alberto Puggelli, Wenchao Li, Alberto L Sangiovanni-Vincentelli, and Sanjit A Seshia. Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties. In CAV, pages 527–542. Springer, 2013.

    Google Scholar 

  28. Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov Decision Processes. Mathematics of Operations Research, 38(1):153–183, 2013.

    Google Scholar 

  29. Indika Meedeniya, Irene Moser, Aldeida Aleti, and Lars Grunske. Evaluating Probabilistic Models with Uncertain Model Parameters. Software & Systems Modeling, 13(4):1395–1415, 2014.

    Google Scholar 

  30. Krishnendu Chatterjee, Martin Chmelík, and Mathieu Tracol. What is Decidable about Partially Observable Markov Decision Processes with \(\omega \)-Regular Objectives. Journal of Computer and System Sciences, 82(5):878–911, 2016.

    Google Scholar 

  31. B. O’Donoghue, E. Chu, N. Parikh, and S. Boyd. Conic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding. Journal of Optimization Theory and Applications, 169(3):1042–1068, 2016.

    Google Scholar 

  32. Tim Quatmann, Christian Dehnert, Nils Jansen, Sebastian Junges, and Joost-Pieter Katoen. Parameter Synthesis for Markov Models: Faster Than Ever. In ATVA, volume 9938 of LNCS, pages 50–67, 2016.

    Google Scholar 

  33. Asrar Ahmed, Pradeep Varakantham, Meghna Lowalekar, Yossiri Adulyasak, and Patrick Jaillet. Sampling Based Approaches for Minimizing Regret in Uncertain Markov Decision Processes (MDPs). J. Artif. Intell. Res., 59:229–264, 2017.

    Google Scholar 

  34. Murat Cubuktepe, Nils Jansen, Sebastian Junges, Joost-Pieter Katoen, Ivan Papusha, Hasan A. Poonawala, and Ufuk Topcu. Sequential Convex Programming for the Efficient Verification of Parametric MDPs. In TACAS (2), volume 10206 of LNCS, pages 133–150, 2017.

    Google Scholar 

  35. Christian Dehnert, Sebastian Junges, Joost-Pieter Katoen, and Matthias Volk. A Storm is Coming: A Modern Probabilistic Model Checker. In CAV (2), volume 10427 of LNCS, pages 592–600. Springer, 2017.

    Google Scholar 

  36. Dimitri Scheftelowitsch, Peter Buchholz, Vahid Hashemi, and Holger Hermanns. Multi-Objective Approaches to Markov Decision Processes with Uncertain Transition Parameters. In VALUETOOLS, pages 44–51, 2017.

    Google Scholar 

  37. Sebastian Arming, Ezio Bartocci, Krishnendu Chatterjee, Joost-Pieter Katoen, and Ana Sokolova. Parameter-Independent Strategies for pMDPs via POMDPs. In QEST, pages 53–70. Springer, 2018.

    Google Scholar 

  38. Luca Bortolussi and Simone Silvetti. Bayesian Statistical Parameter Synthesis for Linear Temporal Properties of Stochastic Models. In TACAS, pages 396–413, 2018.

    Google Scholar 

  39. Marco Claudio Campi, Simone Garatti, and Federico Alessandro Ramponi. A General Scenario Theory for Nonconvex Optimization and Decision Making. IEEE Trans. Automat. Contr., 63(12):4067–4078, 2018.

    Google Scholar 

  40. Murat Cubuktepe, Nils Jansen, Sebastian Junges, Joost-Pieter Katoen, and Ufuk Topcu. Synthesis in pmdps: A tale of 1001 parameters. In ATVA, volume 11138 of LNCS, pages 160–176. Springer, 2018.

    Google Scholar 

  41. Paul Gainer, Ernst Moritz Hahn, and Sven Schewe. Incremental Verification of Parametric and Reconfigurable Markov Chains. CoRR, abs/1804.01872, 2018.

    Google Scholar 

  42. Chin Pang Ho and Marek Petrik. Fast Bellman Updates for Robust MDPs. In ICML, 2018.

    Google Scholar 

  43. Yamilet R. Serrano Llerena, Marcel Böhme, Marc Brünink, Guoxin Su, and David S. Rosenblum. Verifying the Long-run Behavior of Probabilistic System Models in the Presence of Uncertainty. In ESEC/SIGSOFT FSE, pages 587–597. ACM, 2018.

    Google Scholar 

  44. Lauren N Steimle, David L Kaufman, and Brian T Denton. Multi-Model Markov Decision Processes. Optimization Online, 2018.

    Google Scholar 

  45. Giovanni Bacci, Mikkel Hansen, and Kim Guldstrand Larsen. Model Checking Constrained Markov Reward Models with Uncertainties. In QEST, pages 37–51, 2019.

    Google Scholar 

  46. Sebastian Junges, Erika Ábrahám, Christian Hensel, Nils Jansen, Joost-Pieter Katoen, Tim Quatmann, and Matthias Volk. Parameter Synthesis for Markov Models. CoRR, abs/1903.07993, 2019.

    Google Scholar 

  47. Tobias Winkler, Sebastian Junges, Guillermo A. Pérez, and Joost-Pieter Katoen. On the complexity of reachability in parametric markov decision processes. In CONCUR, volume 140 of LIPIcs, pages 14:1–14:17. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2019.

    Google Scholar 

  48. Christel Baier and Joost-Pieter Katoen. Principles of Model Checking. MIT Press, 2008.

    Google Scholar 

  49. S. Basu, R. Pollack, and M.F. Roy. Algorithms in Real Algebraic Geometry. Springer, 2010.

    Google Scholar 

  50. Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, New York, NY, USA, 2004.

    Google Scholar 

  51. Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.

    Google Scholar 

  52. Stuart J Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. 2016.

    Google Scholar 

  53. Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Nils Jansen .

Editor information

Editors and Affiliations

Rights and permissions

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Reprints and Permissions

Copyright information

© 2020 The Author(s)

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Cubuktepe, M., Jansen, N., Junges, S., Katoen, JP., Topcu, U. (2020). Scenario-Based Verification of Uncertain MDPs. In: Biere, A., Parker, D. (eds) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2020. Lecture Notes in Computer Science(), vol 12078. Springer, Cham. https://doi.org/10.1007/978-3-030-45190-5_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-45190-5_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-45189-9

  • Online ISBN: 978-3-030-45190-5

  • eBook Packages: Computer ScienceComputer Science (R0)