Faster Algorithms for Markov Decision Processes with Low Treewidth

  • Krishnendu Chatterjee
  • Jakub Łącki
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8044)


We consider two core algorithmic problems for probabilistic verification: the maximal end-component decomposition and the almost-sure reachability set computation for Markov decision processes (MDPs). For MDPs with treewidth k, we present two improved static algorithms for both the problems that run in time O(n ·k 2.38 ·2 k ) and O(m ·logn ·k), respectively, where n is the number of states and m is the number of edges, significantly improving the previous known \(O(n\cdot k \cdot \sqrt{n\cdot k})\) bound for low treewidth. We also present decremental algorithms for both problems for MDPs with constant treewidth that run in amortized logarithmic time, which is a huge improvement over the previously known algorithms that require amortized linear time.


Markov Decision Process Cover Vertex Transitive Closure Tree Decomposition Algorithmic Problem 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Berwanger, D., Dawar, A., Hunter, P., Kreutzer, S., Obdrzálek, J.: The DAG-width of directed graphs. J. Comb. Theory, Ser. B 102(4), 900–923 (2012)CrossRefzbMATHGoogle Scholar
  2. 2.
    Bianco, A., de Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Thiagarajan, P.S. (ed.) FSTTCS 1995. LNCS, vol. 1026, pp. 499–513. Springer, Heidelberg (1995)CrossRefGoogle Scholar
  3. 3.
    Bodlaender, H.L.: A linear-time algorithm for finding tree-decompositions of small treewidth. SIAM J. Comput. 25(6), 1305–1317 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Bodlaender, H.L.: Treewidth: Algorithmic techniques and results. In: Privara, I., Ružička, P. (eds.) MFCS 1997. LNCS, vol. 1295, pp. 19–36. Springer, Heidelberg (1997)CrossRefGoogle Scholar
  5. 5.
    Brázdil, T., Brozek, V., Chatterjee, K., Forejt, V., Kucera, A.: Two views on multiple mean-payoff objectives in Markov decision processes. In: LICS, pp. 33–42 (2011)Google Scholar
  6. 6.
    Chatterjee, K., Henzinger, M.: Faster and dynamic algorithms for maximal end-component decomposition and related graph problems in probabilistic verification. In: SODA, pp. 1318–1336 (2011)Google Scholar
  7. 7.
    Chatterjee, K., Henzinger, M.: An O(n 2) time algorithm for alternating büchi games. In: SODA, pp. 1386–1399 (2012)Google Scholar
  8. 8.
    Chatterjee, K., Henzinger, T.A.: Probabilistic systems with limsup and liminf objectives. In: Archibald, M., Brattka, V., Goranko, V., Löwe, B. (eds.) ILC 2007. LNCS, vol. 5489, pp. 32–45. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  9. 9.
    Chatterjee, K., Henzinger, T.A., Jobstmann, B., Singh, R.: Measuring and synthesizing systems in probabilistic environments. In: Touili, T., Cook, B., Jackson, P. (eds.) CAV 2010. LNCS, vol. 6174, pp. 380–395. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Chatterjee, K., Jurdziński, M., Henzinger, T.: Quantitative stochastic parity games. In: SODA 2004, pp. 121–130. SIAM (2004)Google Scholar
  11. 11.
    Chatterjee, K., Łącki, J.: Faster algorithms for Markov decision processes with low treewidth CoRR abs/1304.0084 (2013),
  12. 12.
    Coppersmith, D., Winograd, S.: Matrix multiplication via arithmetic progressions. J. Symb. Comput. 9(3), 251–280 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Courcoubetis, C., Yannakakis, M.: The complexity of probabilistic verification. Journal of the ACM 42(4), 857–907 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    de Alfaro, L.: Formal Verification of Probabilistic Systems. PhD thesis, Stanford University (1997)Google Scholar
  15. 15.
    de Alfaro, L., Faella, M., Majumdar, R., Raman, V.: Code-aware resource management. In: EMSOFT 2005. ACM (2005)Google Scholar
  16. 16.
    Fearnley, J., Lachish, O.: Parity games on graphs with medium tree-width. In: Murlak, F., Sankowski, P. (eds.) MFCS 2011. LNCS, vol. 6907, pp. 303–314. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  17. 17.
    Fearnley, J., Schewe, S.: Time and parallelizability results for parity games with bounded treewidth. In: Czumaj, A., Mehlhorn, K., Pitts, A., Wattenhofer, R. (eds.) ICALP 2012, Part II. LNCS, vol. 7392, pp. 189–200. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  18. 18.
    Filar, J., Vrieze, K.: Competitive Markov Decision Processes. Springer (1997)Google Scholar
  19. 19.
    Hinton, A., Kwiatkowska, M., Norman, G., Parker, D.: PRISM: A tool for automatic verification of probabilistic systems. In: Hermanns, H., Palsberg, J. (eds.) TACAS 2006. LNCS, vol. 3920, pp. 441–444. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  20. 20.
    Howard, H.: Dynamic Programming and Markov Processes. MIT Press (1960)Google Scholar
  21. 21.
    Kloks, T.: Treewidth, Computations and Approximations. LNCS, vol. 842. Springer (1994)Google Scholar
  22. 22.
    Kwiatkowska, M., Norman, G., Parker, D.: Verifying randomized distributed algorithms with prism. In: WAVE 2000 (2000)Google Scholar
  23. 23.
    Łącki, J.: Improved deterministic algorithms for decremental transitive closure and strongly connected components. In: SODA, pp. 1438–1445 (2011)Google Scholar
  24. 24.
    Obdržálek, J.: Fast mu-calculus model checking when tree-width is bounded. In: Hunt Jr., W.A., Somenzi, F. (eds.) CAV 2003. LNCS, vol. 2725, pp. 80–92. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  25. 25.
    Pogosyants, A., Segala, R., Lynch, N.: Verification of the randomized consensus algorithm of Aspnes and Herlihy: a case study. Dist. Comp. 13(3), 155–186 (2000)CrossRefGoogle Scholar
  26. 26.
    Robertson, N., Seymour, P.D.: Graph minors. iii. planar tree-width. J. Comb. Theory, Ser. B 36(1), 49–64 (1984)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Stoelinga, M.: Fun with FireWire: Experiments with verifying the IEEE1394 root contention protocol. In: Formal Aspects of Computing (2002)Google Scholar
  28. 28.
    Thomas, W.: Languages, automata, and logic. In: Handbook of Formal Languages, vol. 3, ch. 7, pp. 389–455. Springer, Heidelberg (1997)CrossRefGoogle Scholar
  29. 29.
    Thorup, M.: All structured programs have small tree-width and good register allocation. Inf. Comput. 142(2), 159–181 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Williams, V.V.: Multiplying matrices faster than coppersmith-winograd. In: STOC, pp. 887–898 (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Krishnendu Chatterjee
    • 1
  • Jakub Łącki
    • 2
  1. 1.IST Austria (Institute of Science and Technology Austria)Austria
  2. 2.Institute of InformaticsUniversity of WarsawPoland

Personalised recommendations