Qualitative Analysis of Partially-Observable Markov Decision Processes

  • Krishnendu Chatterjee
  • Laurent Doyen
  • Thomas A. Henzinger
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6281)


We study observation-based strategies for partially-observable Markov decision processes (POMDPs) with parity objectives. An observation-based strategy relies on partial information about the history of a play, namely, on the past sequence of observations. We consider qualitative analysis problems: given a POMDP with a parity objective, decide whether there exists an observation-based strategy to achieve the objective with probability 1 (almost-sure winning), or with positive probability (positive winning). Our main results are twofold. First, we present a complete picture of the computational complexity of the qualitative analysis problem for POMDPs with parity objectives and its subclasses: safety, reachability, Büchi, and coBüchi objectives. We establish several upper and lower bounds that were not known in the literature. Second, we give optimal bounds (matching upper and lower bounds) for the memory required by pure and randomized observation-based strategies for each class of objectives.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Baier, C., Bertrand, N., Größer, M.: On decision problems for probabilistic Büchi automata. In: Amadio, R.M. (ed.) FOSSACS 2008. LNCS, vol. 4962, pp. 287–301. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  2. 2.
    Bertrand, N., Genest, B., Gimbert, H.: Qualitative determinacy and decidability of stochastic games with signals. In: Proc. of LICS, pp. 319–328. IEEE Computer Society, Los Alamitos (2009)Google Scholar
  3. 3.
    Berwanger, D., Chatterjee, K., Doyen, L., Henzinger, T.A., Raje, S.: Strategy construction for parity games with imperfect information. In: van Breugel, F., Chechik, M. (eds.) CONCUR 2008. LNCS, vol. 5201, pp. 325–339. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  4. 4.
    Berwanger, D., Doyen, L.: On the power of imperfect information. In: Proc. of FSTTCS. Dagstuhl Seminar Proceedings 08004 (2008)Google Scholar
  5. 5.
    Bianco, A., de Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Thiagarajan, P.S. (ed.) FSTTCS 1995. LNCS, vol. 1026, pp. 499–513. Springer, Heidelberg (1995)Google Scholar
  6. 6.
    Chadha, R., Sistla, A.P., Viswanathan, M.: Power of randomization in automata on infinite strings. In: Bravetti, M., Zavattaro, G. (eds.) CONCUR 2009. LNCS, vol. 5710, pp. 229–243. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  7. 7.
    Chatterjee, K., Doyen, L., Gimbert, H., Henzinger, T.A.: Randomness for free. In: Proc. of MFCS (2010)Google Scholar
  8. 8.
    Chatterjee, K., Doyen, L., Henzinger, T.A., Raskin, J.-F.: Algorithms for omega-regular games of incomplete information. Logical Methods in Computer Science 3(3:4) (2007)Google Scholar
  9. 9.
    Chatterjee, K., Doyen, L., Henzinger, T.A.: Qualitative analysis of Partially-observable Markov decision processes. CoRR, abs/0909.1645 (2009)Google Scholar
  10. 10.
    Chatterjee, K., Jurdziński, M., Henzinger, T.A.: Quantitative stochastic parity games. In: Proc. of SODA, pp. 114–123 (2004)Google Scholar
  11. 11.
    de Alfaro, L.: Formal Verification of Probabilistic Systems. PhD thesis, Stanford University. Technical Report STAN-CS-TR-98-1601 (1997)Google Scholar
  12. 12.
    de Alfaro, L.: The verification of probabilistic systems under memoryless partial-information policies is hard. In: Proc. of ProbMiV: Probabilistic Methods in Verification (1999)Google Scholar
  13. 13.
    De Wulf, M., Doyen, L., Raskin, J.-F.: A lattice theory for solving games of imperfect information. In: Hespanha, J.P., Tiwari, A. (eds.) HSCC 2006. LNCS, vol. 3927, pp. 153–168. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Gripon, V., Serre, O.: Qualitative concurrent stochastic games with imperfect information. In: Albers, S., Marchetti-Spaccamela, A., Matias, Y., Nikoletseas, S., Thomas, W. (eds.) ICALP 2009, Part II. LNCS, vol. 5556, pp. 200–211. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  15. 15.
    Kechris, A.: Classical Descriptive Set Theory. Springer, Heidelberg (1995)zbMATHGoogle Scholar
  16. 16.
    Littman, M.L.: Algorithms for sequential decision making. PhD thesis, Brown University (1996)Google Scholar
  17. 17.
    Madani, O., Hanks, S., Condon, A.: On the undecidability of probabilistic planning and related stochastic optimization problems. Artif. Intell. 147(1-2) (2003)Google Scholar
  18. 18.
    Papadimitriou, C.H., Tsitsiklis, J.N.: The complexity of Markov decision processes. Mathematics of Operations Research 12, 441–450 (1987)zbMATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Paz, A.: Introduction to probabilistic automata. Academic Press, London (1971)zbMATHGoogle Scholar
  20. 20.
    Reif, J.: The complexity of two-player games of incomplete information. Journal of Computer and System Sciences 29, 274–301 (1984)zbMATHCrossRefMathSciNetGoogle Scholar
  21. 21.
    Segala, R.: Modeling and Verification of Randomized Distributed Real-Time Systems. PhD thesis, MIT. Technical Report MIT/LCS/TR-676 (1995)Google Scholar
  22. 22.
    Thomas, W.: Languages, automata, and logic. In: Handbook of Formal Languages, Beyond Words, ch. 7, vol. 3, pp. 389–455. Springer, Heidelberg (1997)Google Scholar
  23. 23.
    Vardi, M.Y.: Automatic verification of probabilistic concurrent finite-state systems. In: Proc. of FOCS, pp. 327–338. IEEE Computer Society Press, Los Alamitos (1985)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Krishnendu Chatterjee
    • 1
  • Laurent Doyen
    • 2
  • Thomas A. Henzinger
    • 1
  1. 1.IST Austria (Institute of Science and TechnologyAustria)
  2. 2.LSV, ENS Cachan & CNRSFrance

Personalised recommendations