Abstract
We consider Markov decision processes with synchronizing objectives, which require that a probability mass of \(1-\varepsilon \) accumulates in a designated set of target states, either once, always, infinitely often, or always from some point on, where \(\varepsilon = 0\) for sure synchronizing, and \(\varepsilon \rightarrow 0\) for almost-sure and limit-sure synchronizing.
We introduce two new qualitative modes of synchronizing, where the probability mass should be either positive, or bounded away from 0. They can be viewed as dual synchronizing objectives. We present algorithms and tight complexity results for the problem of deciding if a Markov decision process is positive, or bounded synchronizing, and we provide explicit bounds on \(\varepsilon \) in all synchronizing modes. In particular, we show that deciding positive and bounded synchronizing always from some point on, is coNP-complete.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The results of [12, Lemma 11 & 12] consider a more general definition of limit-sure synchronizing, where the support of the \((1-\varepsilon )\)-synchronizing distribution is required to have its support contained in a given set Z. We release this constraint by taking \(Z = Q\).
References
Akshay, S., Genest, B., Vyas, N.: Distribution-based objectives for Markov decision processes. In: Proceedings of LICS, pp. 36–45. ACM (2018)
Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press, Cambridge (2008)
Beauquier, D., Rabinovich, A., Slissenko, A.: A logic of probability with decidable model-checking. In: Bradfield, J. (ed.) CSL 2002. LNCS, vol. 2471, pp. 306–321. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45793-3_21
Bell, P.C., Semukhin, P.: Decidability of cutpoint isolation for probabilistic finite automata on letter-bounded inputs. In: Proceedings of CONCUR, vol. 171 of LIPIcs, pp. 22:1–22:16. Schloss Dagstuhl - Leibniz-Zentrum für Informatik (2020)
Bertrand, N., Dewaskar, M., Genest, B., Gimbert, H.: Controlling a population. In: Proceedings of CONCUR: Concurrency Theory, volume 85 of LIPIcs, pp. 12:1–12:16. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik (2017)
Chatterjee, K., Doyen, L.: Computation tree logic for synchronization properties. In: Proc. of ICALP: Automata, Languages, and Programming, volume 55 of LIPIcs, pp. 98:1–98:14. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik (2016)
Chatterjee, K., Henzinger, T.A.: A survey of stochastic \(\omega \)-regular games. J. Comput. Syst. Sci. 78(2), 394–413 (2012)
Colcombet, T., Fijalkow, N., Ohlmann, P.: Controlling a random population. In: FoSSaCS 2020. LNCS, vol. 12077, pp. 119–135. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45231-5_7
Courcoubetis, C., Yannakakis, M.: The complexity of probabilistic verification. J. ACM 42(4), 857–907 (1995)
de Alfaro, L.: Formal verification of probabilistic systems. Ph.D. thesis, Stanford University (1997)
de Alfaro, L., Henzinger, T.A.: Concurrent omega-regular games. In: Proceedings. of LICS: Logic in Computer Science, pp. 141–154. IEEE (2000)
Doyen, L., Massart, T., Shirmohammadi, M.: The complexity of synchronizing Markov decision processes. J. Comput. Syst. Sci. 100, 96–129 (2019)
Doyen, L., van den Bogaard, M.: Bounds for synchronizing Markov decision processes. CoRR, abs/2204.12814 (2022)
Feinberg, E., Shwartz, A. (eds.) Handbook of Markov Decision Processes - Methods and Applications. Kluwer, Boston (2002)
Gallager, R.G.: Stochastic Processes: Theory for Applications. Cambridge University Press, New York (2013)
Hermanns, H., Krčál, J., Křetínský, J.: Probabilistic Bisimulation: naturally on distributions. In: Baldan, P., Gorla, D. (eds.) CONCUR 2014. LNCS, vol. 8704, pp. 249–265. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44584-6_18
Jancar, P., Sawa, Z.: A note on emptiness for alternating finite automata with a one-letter alphabet. Inf. Process. Lett. 104(5), 164–167 (2007)
Korthikanti, V.A., Viswanathan, M., Agha, G., Kwon, Y.: Reasoning about MDPs as transformers of probability distributions. In: Proceedings of QEST: Quantitative Evaluation of Systems, pp. 199–208. IEEE Computer Society (2010)
Puterman. M.L.: Markov Decision Processes. John Wiley and Sons, New York (1994)
Rabin, M.O.: Probabilistic automata. Inf. Control 6, 230–245 (1963)
Shirmohammadi, M.: Qualitative analysis of probabilistic synchronizing systems. Ph.D. thesis, U. Libre de Bruxelles & École Normale Supérieure de Cachan (2014)
Vardi, M.Y.: Automatic verification of probabilistic concurrent finite-state programs. In: Proceedings of FOCS, pp. 327–338. IEEE Computer Society (1985)
Acknowledgment
The authors are grateful to Jean-François Raskin for logistical support, and to Mahsa Shirmohammadi for interesting discussions about adversarial objectives.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Doyen, L., van den Bogaard, M. (2022). Bounds for Synchronizing Markov Decision Processes. In: Kulikov, A.S., Raskhodnikova, S. (eds) Computer Science – Theory and Applications. CSR 2022. Lecture Notes in Computer Science, vol 13296. Springer, Cham. https://doi.org/10.1007/978-3-031-09574-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-09574-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09573-3
Online ISBN: 978-3-031-09574-0
eBook Packages: Computer ScienceComputer Science (R0)