Minimizing Expected Termination Time in One-Counter Markov Decision Processes

  • Tomáš Brázdil
  • Antonín Kučera
  • Petr Novotný
  • Dominik Wojtczak
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7392)


We consider the problem of computing the value and an optimal strategy for minimizing the expected termination time in one-counter Markov decision processes. Since the value may be irrational and an optimal strategy may be rather complicated, we concentrate on the problems of approximating the value up to a given error ε > 0 and computing a finite representation of an ε-optimal strategy. We show that these problems are solvable in exponential time for a given configuration, and we also show that they are computationally hard in the sense that a polynomial-time approximation algorithm cannot exist unless P=NP.


Markov Decision Process Outgoing Edge Stochastic Game Parity Game Current Counter 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Proceedings of FST&TCS 2010. LIPIcs, vol. 8. Schloss Dagstuhl (2010)Google Scholar
  2. 2.
    Brázdil, T., Brožek, V., Etessami, K.: One-counter stochastic games. In: Proceedings of FST&TCS 2010 [1], pp. 108–119Google Scholar
  3. 3.
    Brázdil, T., Brožek, V., Etessami, K., Kučera, A.: Approximating the Termination Value of One-Counter MDPs and Stochastic Games. In: Aceto, L., Henzinger, M., Sgall, J. (eds.) ICALP 2011, Part II. LNCS, vol. 6756, pp. 332–343. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  4. 4.
    Brázdil, T., Brožek, V., Etessami, K., Kučera, A., Wojtczak, D.: One-counter Markov decision processes. In: Proceedings of SODA 2010, pp. 863–874. SIAM (2010)Google Scholar
  5. 5.
    Brázdil, T., Brožek, V., Forejt, V., Kučera, A.: Reachability in recursive Markov decision processes. I&C 206(5), 520–537 (2008)zbMATHGoogle Scholar
  6. 6.
    Brázdil, T., Brožek, V., Kučera, A., Obdržálek, J.: Qualitative reachability in stochastic BPA games. I&C 208(7), 772–796 (2010)Google Scholar
  7. 7.
    Brázdil, T., Kučera, A., Novotný, P., Wojtczak, D.: Minimizing expected termination time in one-counter Markov decision processes. CoRR abs/1205.1473 (2012)Google Scholar
  8. 8.
    Chatterjee, K., Doyen, L.: Energy Parity Games. In: Abramsky, S., Gavoille, C., Kirchner, C., Meyer auf der Heide, F., Spirakis, P.G. (eds.) ICALP 2010, Part II. LNCS, vol. 6199, pp. 599–610. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  9. 9.
    Chatterjee, K., Doyen, L., Henzinger, T., Raskin, J.F.: Generalized mean-payoff and energy games. In: Proceedings of FST&TCS 2010 [1], pp. 505–516Google Scholar
  10. 10.
    Etessami, K., Wojtczak, D., Yannakakis, M.: Recursive Stochastic Games with Positive Rewards. In: Aceto, L., Damgård, I., Goldberg, L.A., Halldórsson, M.M., Ingólfsdóttir, A., Walukiewicz, I. (eds.) ICALP 2008, Part I. LNCS, vol. 5125, pp. 711–723. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  11. 11.
    Etessami, K., Wojtczak, D., Yannakakis, M.: Quasi-birth-death processes, tree-like QBDs, probabilistic 1-counter automata, and pushdown systems. Performance Evaluation 67(9), 837–857 (2010)CrossRefGoogle Scholar
  12. 12.
    Etessami, K., Yannakakis, M.: Recursive Markov Decision Processes and Recursive Stochastic Games. In: Caires, L., Italiano, G.F., Monteiro, L., Palamidessi, C., Yung, M. (eds.) ICALP 2005. LNCS, vol. 3580, pp. 891–903. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  13. 13.
    Etessami, K., Yannakakis, M.: Efficient Qualitative Analysis of Classes of Recursive Markov Decision Processes and Simple Stochastic Games. In: Durand, B., Thomas, W. (eds.) STACS 2006. LNCS, vol. 3884, pp. 634–645. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Filar, J., Vrieze, K.: Competitive Markov Decision Processes. Springer (1996)Google Scholar
  15. 15.
    Göller, S., Lohrey, M.: Branching-time model checking of one-counter processes. In: Proceedings of STACS 2010. LIPIcs, vol. 5, pp. 405–416. Schloss Dagstuhl (2010)Google Scholar
  16. 16.
    Jančar, P., Kučera, A., Moller, F., Sawa, Z.: DP lower bounds for equivalence-checking and model-checking of one-counter automata. I&C 188(1), 1–19 (2004)zbMATHGoogle Scholar
  17. 17.
    Jančar, P., Sawa, Z.: A note on emptiness for alternating finite automata with a one-letter alphabet. IPL 104(5), 164–167 (2007)zbMATHCrossRefGoogle Scholar
  18. 18.
    Kučera, A.: The complexity of bisimilarity-checking for one-counter processes. TCS 304(1-3), 157–183 (2003)zbMATHCrossRefGoogle Scholar
  19. 19.
    Latouche, G., Ramaswami, V.: Introduction to Matrix Analytic Methods in Stochastic Modeling. ASA-SIAM series on statistics and applied probability (1999)Google Scholar
  20. 20.
    Puterman, M.: Markov Decision Processes. Wiley (1994)Google Scholar
  21. 21.
    Serre, O.: Parity Games Played on Transition Graphs of One-Counter Processes. In: Aceto, L., Ingólfsdóttir, A. (eds.) FOSSACS 2006. LNCS, vol. 3921, pp. 337–351. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  22. 22.
    Williams, D.: Probability with Martingales. Cambridge University Press (1991)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Tomáš Brázdil
    • 1
  • Antonín Kučera
    • 1
  • Petr Novotný
    • 1
  • Dominik Wojtczak
    • 2
  1. 1.Faculty of InformaticsMasaryk UniversityCzech Republic
  2. 2.Department of Computer ScienceUniversity of LiverpoolUK

Personalised recommendations