Abstract
We consider Markov decision processes (MDPs) in which the transition probabilities and rewards belong to an uncertainty set parametrized by a collection of random variables. The probability distributions for these random parameters are unknown. The problem is to compute the probability to satisfy a temporal logic specification within any MDP that corresponds to a sample from these unknown distributions. In general, this problem is undecidable, and we resort to techniques from so-called scenario optimization. Based on a finite number of samples of the uncertain parameters, each of which induces an MDP, the proposed method estimates the probability of satisfying the specification by solving a finite-dimensional convex optimization problem. The number of samples required to obtain a high confidence on this estimate is independent from the number of states and the number of random parameters. Experiments on a large set of benchmarks show that a few thousand samples suffice to obtain high-quality confidence bounds with a high probability.
Chapter PDF
Similar content being viewed by others
References
Abraham Charnes and William W Cooper. Chance-Constrained Programming. Management science, 6(1):73–79, 1959.
Amir Pnueli. The Temporal Logic of Programs. In FOCS, pages 46–57, 1977.
James Aspnes and Maurice Herlihy. Fast Randomized Consensus Using Shared Memory. Journal of Algorithms, 15(1):441–460, 1990.
Hans Hansson and Bengt Jonsson. A Logic for Reasoning about Time and Reliability. Formal Aspects of Computing, 6(5):512–535, 1994.
L. Helmink, M. Sellink, and F. Vaandrager. Proof-Checking a Data Link Protocol. In TYPES, volume 806 of LNCS, pages 127–165. Springer, 1994.
Anthony Cassandra, Michael L Littman, and Nevin L Zhang. Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes. In UAI, pages 54–61, 1997.
Robert Givan, Sonia Leach, and Thomas Dean. Bounded-Parameter Markov Decision Processes. Artificial Intelligence, 122(1-2):71–109, 2000.
Jie Han and Pieter Jonker. A System Architecture Solution for Unreliable Nanoelectronic Devices. IEEE Transactions on Nanotechnology, 1:201–208, 2002.
Christel Baier, Boudewijn R. Haverkort, Holger Hermanns, and Joost-Pieter Katoen. Model-Checking Algorithms for Continuous-Time Markov Chains. IEEE Trans. Software Eng., 29(6):524–541, 2003.
Joelle Pineau, Geoff Gordon, and Sebastian Thrun. Point-Based Value Iteration: an Anytime Algorithm for POMDPs. In IJCAI, pages 1025–1030, 2003.
Conrado Daws. Symbolic and Parametric Model Checking of Discrete-Time Markov chains. In ICTAC, volume 3407 of LNCS, pages 280–294. Springer, 2004.
Vitaly Shmatikov. Probabilistic Analysis of an Anonymity System. Journal of Computer Security, 12(3-4):355–377, 2004.
G.C. Calafiore and M.C. Campi. Uncertain Convex Programs: Randomized Solutions and Confidence Levels. Mathematical Programming, 102(1):25–46, 2005.
Arnab Nilim and Laurent El Ghaoui. Robust Control of Markov Decision Processes with Uncertain Transition Matrices. Operations Research, 53(5):780–798, 2005.
Giuseppe C. Calafiore and Marco C. Campi. The Scenario Approach to Robust Control Design. IEEE Trans. Automat. Contr., 51(5):742–753, 2006.
Marco C. Campi and Simone Garatti. The Exact Feasibility of Randomized Solutions of Uncertain Convex Programs. SIAM Journal on Optimization, 19(3):1211–1230, 2008.
Ernst Moritz Hahn, Holger Hermanns, and Lijun Zhang. Probabilistic Reachability for Parametric Markov Models. STTT, 13(1):3–19, 2010.
Dimitris Bertsimas, David B Brown, and Constantine Caramanis. Theory and Applications of Robust Optimization. SIAM review, 53(3):464–501, 2011.
Marco C Campi and Simone Garatti. A Sampling-and-Discarding Approach to Chance-Constrained Optimization: Feasibility and Optimality. Journal of Optimization Theory and Applications, 148(2):257–280, 2011.
Benoît Delahaye, Kim G. Larsen, Axel Legay, Mikkel L. Pedersen, and Andrzej Wasowski. Decision problems for interval Markov chains. In LATA, volume 6638 of LNCS, pages 274–285. Springer, 2011.
Marta Kwiatkowska, Gethin Norman, and David Parker. PRISM 4.0: Verification of Probabilistic Real-Time Systems. In CAV, volume 6806 of LNCS, pages 585–591. Springer, 2011.
Hassan Hatefi and Holger Hermanns. Model Checking Algorithms for Markov Automata. ECEASST, 53, 2012.
Marta Kwiatkowska, Gethin Norman, and David Parker. The PRISM Benchmark Suite. In QEST, pages 203–204. IEEE CS, 2012.
Rowan McAllister, Thierry Peynot, Robert Fitch, and Salah Sukkarieh. Motion Planning and Stochastic Control with Experimental Validation on a Planetary Rover. In IROS, pages 4716–4723. IEEE, 2012.
Eric M. Wolff, Ufuk Topcu, and Richard M. Murray. Robust Control of Uncertain Markov Decision Processes with Temporal Logic Specifications. In CDC, pages 3372–3379, 2012.
Taolue Chen, Ernst Moritz Hahn, Tingting Han, Marta Kwiatkowska, Hongyang Qu, and Lijun Zhang. Model Repair for Markov Decision Processes. In TASE, pages 85–92. IEEE CS, 2013.
Alberto Puggelli, Wenchao Li, Alberto L Sangiovanni-Vincentelli, and Sanjit A Seshia. Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties. In CAV, pages 527–542. Springer, 2013.
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov Decision Processes. Mathematics of Operations Research, 38(1):153–183, 2013.
Indika Meedeniya, Irene Moser, Aldeida Aleti, and Lars Grunske. Evaluating Probabilistic Models with Uncertain Model Parameters. Software & Systems Modeling, 13(4):1395–1415, 2014.
Krishnendu Chatterjee, Martin Chmelík, and Mathieu Tracol. What is Decidable about Partially Observable Markov Decision Processes with \(\omega \)-Regular Objectives. Journal of Computer and System Sciences, 82(5):878–911, 2016.
B. O’Donoghue, E. Chu, N. Parikh, and S. Boyd. Conic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding. Journal of Optimization Theory and Applications, 169(3):1042–1068, 2016.
Tim Quatmann, Christian Dehnert, Nils Jansen, Sebastian Junges, and Joost-Pieter Katoen. Parameter Synthesis for Markov Models: Faster Than Ever. In ATVA, volume 9938 of LNCS, pages 50–67, 2016.
Asrar Ahmed, Pradeep Varakantham, Meghna Lowalekar, Yossiri Adulyasak, and Patrick Jaillet. Sampling Based Approaches for Minimizing Regret in Uncertain Markov Decision Processes (MDPs). J. Artif. Intell. Res., 59:229–264, 2017.
Murat Cubuktepe, Nils Jansen, Sebastian Junges, Joost-Pieter Katoen, Ivan Papusha, Hasan A. Poonawala, and Ufuk Topcu. Sequential Convex Programming for the Efficient Verification of Parametric MDPs. In TACAS (2), volume 10206 of LNCS, pages 133–150, 2017.
Christian Dehnert, Sebastian Junges, Joost-Pieter Katoen, and Matthias Volk. A Storm is Coming: A Modern Probabilistic Model Checker. In CAV (2), volume 10427 of LNCS, pages 592–600. Springer, 2017.
Dimitri Scheftelowitsch, Peter Buchholz, Vahid Hashemi, and Holger Hermanns. Multi-Objective Approaches to Markov Decision Processes with Uncertain Transition Parameters. In VALUETOOLS, pages 44–51, 2017.
Sebastian Arming, Ezio Bartocci, Krishnendu Chatterjee, Joost-Pieter Katoen, and Ana Sokolova. Parameter-Independent Strategies for pMDPs via POMDPs. In QEST, pages 53–70. Springer, 2018.
Luca Bortolussi and Simone Silvetti. Bayesian Statistical Parameter Synthesis for Linear Temporal Properties of Stochastic Models. In TACAS, pages 396–413, 2018.
Marco Claudio Campi, Simone Garatti, and Federico Alessandro Ramponi. A General Scenario Theory for Nonconvex Optimization and Decision Making. IEEE Trans. Automat. Contr., 63(12):4067–4078, 2018.
Murat Cubuktepe, Nils Jansen, Sebastian Junges, Joost-Pieter Katoen, and Ufuk Topcu. Synthesis in pmdps: A tale of 1001 parameters. In ATVA, volume 11138 of LNCS, pages 160–176. Springer, 2018.
Paul Gainer, Ernst Moritz Hahn, and Sven Schewe. Incremental Verification of Parametric and Reconfigurable Markov Chains. CoRR, abs/1804.01872, 2018.
Chin Pang Ho and Marek Petrik. Fast Bellman Updates for Robust MDPs. In ICML, 2018.
Yamilet R. Serrano Llerena, Marcel Böhme, Marc Brünink, Guoxin Su, and David S. Rosenblum. Verifying the Long-run Behavior of Probabilistic System Models in the Presence of Uncertainty. In ESEC/SIGSOFT FSE, pages 587–597. ACM, 2018.
Lauren N Steimle, David L Kaufman, and Brian T Denton. Multi-Model Markov Decision Processes. Optimization Online, 2018.
Giovanni Bacci, Mikkel Hansen, and Kim Guldstrand Larsen. Model Checking Constrained Markov Reward Models with Uncertainties. In QEST, pages 37–51, 2019.
Sebastian Junges, Erika Ábrahám, Christian Hensel, Nils Jansen, Joost-Pieter Katoen, Tim Quatmann, and Matthias Volk. Parameter Synthesis for Markov Models. CoRR, abs/1903.07993, 2019.
Tobias Winkler, Sebastian Junges, Guillermo A. Pérez, and Joost-Pieter Katoen. On the complexity of reachability in parametric markov decision processes. In CONCUR, volume 140 of LIPIcs, pages 14:1–14:17. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2019.
Christel Baier and Joost-Pieter Katoen. Principles of Model Checking. MIT Press, 2008.
S. Basu, R. Pollack, and M.F. Roy. Algorithms in Real Algebraic Geometry. Springer, 2010.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, New York, NY, USA, 2004.
Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
Stuart J Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. 2016.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2020 The Author(s)
About this paper
Cite this paper
Cubuktepe, M., Jansen, N., Junges, S., Katoen, JP., Topcu, U. (2020). Scenario-Based Verification of Uncertain MDPs. In: Biere, A., Parker, D. (eds) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2020. Lecture Notes in Computer Science(), vol 12078. Springer, Cham. https://doi.org/10.1007/978-3-030-45190-5_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-45190-5_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-45189-9
Online ISBN: 978-3-030-45190-5
eBook Packages: Computer ScienceComputer Science (R0)