Automata, Probability, and Recursion

  • Mihalis Yannakakis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5148)


We discuss work on the modeling and analysis of systems with probabilistic and recursive features. Recursive Markov chains extend ordinary finite state Markov chains with the ability to invoke other Markov chains in a potentially recursive manner. The equivalent model of Probabilistic Pushdown Automata extends ordinary pushdown automata with probabilistic actions. Both of these are natural abstract models for probabilistic programs with procedures, and related systems. They generalize other classical well-studied stochastic models, e.g. Stochastic Context-free Grammars and (Multi-type) Branching Processes, that arise in a variety of areas. More generally, Recursive Markov Decision Processes and Recursive Stochastic Games can be used to model recursive systems that have both probabilistic and nonprobabilistic, controllable actions. In recent years there has been substantial work on the algorithmic analysis of these models, regarding basic questions of termination, reachability, and analysis of the properties of their executions. In this talk we will present some of the basic theory, algorithmic methods, results, and challenges.


Markov Chain Model Check Markov Decision Process Stochastic Game Extinction Probability 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abney, S., McAllester, D., Pereira, F.: Relating probabilistic grammars and automata. In: Proc. 37th Ann. Meeting of Ass. for Comp. Linguistics, pp. 542–549. Morgan Kaufmann, San Francisco (1999)CrossRefGoogle Scholar
  2. 2.
    Allender, E., Bürgisser, P., Kjeldgaard-Pedersen, J., Miltersen, P.B.: On the complexity of numerical analysis. In: 21st IEEE Computational Complexity Conference (2006)Google Scholar
  3. 3.
    Alur, R., Yannakakis, M.: Model checking of hierarchical state machines. ACM Trans. Prog. Lang. Sys. 23(3), 273–303 (2001)CrossRefGoogle Scholar
  4. 4.
    Alur, R., Benedikt, M., Etessami, K., Godefroid, P., Reps, T.W., Yannakakis, M.: Analysis of recursive state machines. ACM Trans. Progr. Lang. Sys. 27, 786–818 (2005)CrossRefGoogle Scholar
  5. 5.
    Bini, D., Latouche, G., Meini, B.: Numerical methods for Structured Markov Chains. Oxford University Press, Oxford (2005)zbMATHGoogle Scholar
  6. 6.
    Bouajjani, A., Esparza, J., Maler, O.: Reachability analysis of pushdown automata: Applications to model checking. In: Mazurkiewicz, A., Winkowski, J. (eds.) CONCUR 1997. LNCS, vol. 1243, pp. 135–150. Springer, Heidelberg (1997)Google Scholar
  7. 7.
    Brázdil, T., Brozek, V., Forejt, V., Kučera, A.: Reachability in recursive Markov decision processes. In: Baier, C., Hermanns, H. (eds.) CONCUR 2006. LNCS, vol. 4137, pp. 358–374. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  8. 8.
    Brázdil, T., Kučera, A., Esparza, J.: Analysis and prediction of the long-run behavior of probabilistic sequential programs with recursion. In: Proc. of FOCS 2005, pp. 521–530 (2005)Google Scholar
  9. 9.
    Brázdil, T., Kučera, A., Stražovský, O.: Decidability of temporal properties of probabilistic pushdown automata. In: Diekert, V., Durand, B. (eds.) STACS 2005. LNCS, vol. 3404. Springer, Heidelberg (2005)Google Scholar
  10. 10.
    Canny, J.: Some algebraic and geometric computations in PSPACE. In: Proc. of 20th ACM STOC, pp. 460–467 (1988)Google Scholar
  11. 11.
    Condon, A.: The complexity of stochastic games. Inf. & Comp. 96(2), 203–224 (1992)zbMATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Courcoubetis, C., Yannakakis, M.: The complexity of probabilistic verification. Journal of the ACM 42(4), 857–907 (1995)zbMATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Courcoubetis, C., Yannakakis, M.: Markov decision processes and regular events. IEEE Trans. on Automatic Control 43(10), 1399–1418 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    de Alfaro, L., Majumdar, R.: Quantitative solution of omega-regular games. J. Comp. Sys. Sc. 68(2), 374–397 (2004)zbMATHCrossRefGoogle Scholar
  15. 15.
    Durbin, R., Eddy, S.R., Krogh, A., Mitchison, G.: Biological Sequence Analysis: Probabilistic models of Proteins and Nucleic Acids. Cambridge U. Press (1999)Google Scholar
  16. 16.
    Esparza, J., Gawlitza, T., Kiefer, S., Seidl, H.: Approximative methods for motonone systems of min-max-polynomial equations. In: Proc. 35th ICALP (2008)Google Scholar
  17. 17.
    Esparza, J., Hansel, D., Rossmanith, P., Schwoon, S.: Efficient algorithms for model checking pushdown systems. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 232–247. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  18. 18.
    Esparza, J., Kiefer, S., Luttenberger, M.: Convergence thresholds of Newton’s method for monotone polynomial equations. In: Proc. STACS (2008)Google Scholar
  19. 19.
    Esparza, J., Kučera, A., Mayr, R.: Model checking probabilistic pushdown automata. In: Proc. of 19th IEEE LICS 2004 (2004); Full version in Logical Methods in Computer Science 2(1) (2006) Google Scholar
  20. 20.
    Esparza, J., Kučera, A., Mayr, R.: Quantitative analysis of probabilistic pushdown automata: expectations and variances. In: Proc. of 20th IEEE LICS (2005)Google Scholar
  21. 21.
    Etessami, K., Wojtczak, D., Yannakakis, M.: Recursive Stochastic Games with Positive Rewards. In: Proc. 35th ICALP (2008)Google Scholar
  22. 22.
    Etessami, K., Wojtczak, D., Yannakakis, M.: Quasi-birth-death processes, tree-like QBDs, probabilistic 1-counter automata, and pushdown systems (submitted, 2008)Google Scholar
  23. 23.
    Etessami, K., Yannakakis, M.: Recursive Markov chains, stochastic grammars, and monotone systems of non-linear equations. In: Diekert, V., Durand, B. (eds.) STACS 2005. LNCS, vol. 3404, pp. 340–352. Springer, Heidelberg (2005), Google Scholar
  24. 24.
    Etessami, K., Yannakakis, M.: Algorithmic verification of recursive probabilistic state machines. In: Halbwachs, N., Zuck, L.D. (eds.) TACAS 2005. LNCS, vol. 3440, pp. 253–270. Springer, Heidelberg (2005)Google Scholar
  25. 25.
    Etessami, K., Yannakakis, M.: Recursive Markov Decision Processes and Recursive Stochastic Games. In: Caires, L., Italiano, G.F., Monteiro, L., Palamidessi, C., Yung, M. (eds.) ICALP 2005. LNCS, vol. 3580, pp. 891–903. Springer, Heidelberg (2005)Google Scholar
  26. 26.
    Etessami, K., Yannakakis, M.: Checking LTL Properties of Recursive Markov Chains. In: Proc. 2nd Intl. Conf. on Quantitative Evaluation of Systems. IEEE, Los Alamitos (2005)Google Scholar
  27. 27.
    Etessami, K., Yannakakis, M.: Efficient Qualitative Analysis of Classes of Recursive Markov Decision Processes and Simple Stochastic Games. In: Durand, B., Thomas, W. (eds.) STACS 2006. LNCS, vol. 3884, pp. 634–645. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  28. 28.
    Etessami, K., Yannakakis, M.: Recursive concurrent stochastic games. In: Bugliesi, M., Preneel, B., Sassone, V., Wegener, I. (eds.) ICALP 2006. LNCS, vol. 4052, pp. 324–335. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  29. 29.
    Etessami, K., Yannakakis, M.: On the complexity of Nash equilibria and other fixed points. In: Proc. of 48th IEEE FOCS (2007)Google Scholar
  30. 30.
    Etessami, K., Yannakakis, M.: Recursive Markov Processes (in preparation, 2008)Google Scholar
  31. 31.
    Fagin, R., Karlin, A., Kleinberg, J., Raghavan, P., Rajagopalan, S., Rubinfeld, R., Sudan, M., Tomkins, A.: Random walks with “back buttons” (extended abstract). In: ACM Symp. on Theory of Computing, pp. 484–493 (2000); Full version in Ann. of App. Prob., 11, pp 810–862 (2001) Google Scholar
  32. 32.
    Filar, J., Vrieze, K.: Competitive Markov Decision Processes. Springer, Heidelberg (1997)zbMATHGoogle Scholar
  33. 33.
    Garey, M.R., Graham, R.L., Johnson, D.S.: Some NP-complete geometric problems. In: 8th ACM Symp. on Theory of Computing, pp. 10–22 (1976)Google Scholar
  34. 34.
    Haccou, P., Jagers, P., Vatutin, V.A.: Branching Processes: Variation, Growth, and Extinction of Populations. Cambridge U. Press (2005)Google Scholar
  35. 35.
    Harris, T.E.: The Theory of Branching Processes. Springer, Heidelberg (1963)zbMATHGoogle Scholar
  36. 36.
    Jagers, P.: Branching Processes with Biological Applications. Wiley, Chichester (1975)zbMATHGoogle Scholar
  37. 37.
    Kiefer, S., Luttenberger, M., Esparza, J.: On the convergence of Newton’s method for monotone systems of polynomial equations. In: Proc. 39th Symp. on Theory of Computation (STOC), pp. 217–226 (2007)Google Scholar
  38. 38.
    Kimmel, M., Axelrod, D.E.: Branching processes in biology. Springer, Heidelberg (2002)zbMATHGoogle Scholar
  39. 39.
    Kolmogorov, A.N., Sevastyanov, B.A.: The calculation of final probabilities for branching random processes. Dokl. Akad. Nauk SSSR 56, 783–786 (1947) (Russian)zbMATHGoogle Scholar
  40. 40.
    Kwiatkowska, M.: Model checking for probability and time: from theory to practice. In: 18th IEEE LICS, pp. 351–360 (2003)Google Scholar
  41. 41.
    Latouche, G., Ramaswami, V.: Introduction to Matrix Analytic Methods in Stochastic Modeling. ASA-SIAM series on statistics and applied probability (1999)Google Scholar
  42. 42.
    Manning, C., Schütze, H.: Foundations of Statistical Natural Language Processing. MIT Press, Cambridge (1999)zbMATHGoogle Scholar
  43. 43.
    Nederhof, M.J., Satta, G.: Using Newton’s method to compute the partition function of a PCFG (unpublished manuscript, 2006)Google Scholar
  44. 44.
    Neyman, A., Sorin, S. (eds.): Stochastic Games and Applications. Kluwer, Dordrecht (2003)zbMATHGoogle Scholar
  45. 45.
    Neuts, M.F.: Stuctured Stochastic Matrices of M/G/1 Type and their applications. Marcel Dekker, New York (1989)Google Scholar
  46. 46.
    Paz, A.: Introduction to Probabilistic Automata. Academic Press, London (1971)zbMATHGoogle Scholar
  47. 47.
    Puterman, M.L.: Markov Decision Processes. Wiley, Chichester (1994)zbMATHGoogle Scholar
  48. 48.
    Renegar, J.: On the computational complexity and geometry of the first-order theory of the reals, parts I-III. J. Symb. Comp. 13(3), 255–352 (1992)zbMATHMathSciNetCrossRefGoogle Scholar
  49. 49.
    Sakakibara, Y., Brown, M., Hughey, R., Mian, I.S., Sjolander, K., Underwood, R., Haussler, D.: Stochastic context-free grammars for tRNA modeling. Nucleic Acids Research 22(23), 5112–5120 (1994)CrossRefGoogle Scholar
  50. 50.
    Sevastyanov, B.A.: The theory of branching processes. Uspehi Mathemat. Nauk 6, 47–99 (1951) (Russian)MathSciNetGoogle Scholar
  51. 51.
    Shapley, L.S.: Stochastic games. Proc. Nat. Acad. Sci. 39, 1095–1100 (1953)zbMATHCrossRefMathSciNetGoogle Scholar
  52. 52.
    Tiwari, P.: A problem that is easier to solve on the unit-cost algebraic RAM. Journal of Complexity, 393–397 (1992)Google Scholar
  53. 53.
    van Houdt, B., Blondia, C.: Tree structured QBD Markov chains and tree-like QBD processes. Stochastic Models 19(4), 467–482 (2003)zbMATHCrossRefMathSciNetGoogle Scholar
  54. 54.
    Vardi, M.: Automatic verification of probabilistic concurrent finite-state programs. In: Proc. of 26th IEEE FOCS, pp. 327–338 (1985)Google Scholar
  55. 55.
    Wojtczak, D., Etessami, K.: Premo: an analyzer for probabilistic recursive models. In: Grumberg, O., Huth, M. (eds.) TACAS 2007. LNCS, vol. 4424. Springer, Heidelberg (2007), CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Mihalis Yannakakis
    • 1
  1. 1.Department of Computer ScienceColumbia University 

Personalised recommendations