Eager Markov Chains

  • Parosh Aziz Abdulla
  • Noomene Ben Henda
  • Richard Mayr
  • Sven Sandberg
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4218)


We consider infinite-state discrete Markov chains which are eager: the probability of avoiding a defined set of final states for more than n steps is bounded by some exponentially decreasing function f(n). We prove that eager Markov chains include those induced by Probabilistic Lossy Channel Systems, Probabilistic Vector Addition Systems with States, and Noisy Turing Machines, and that the bounding function f(n) can be effectively constructed for them. Furthermore, we study the problem of computing the expected reward (or cost) of runs until reaching the final states, where rewards are assigned to individual runs by computable reward functions. For eager Markov chains, an effective path exploration scheme, based on forward reachability analysis, can be used to approximate the expected reward up-to an arbitrarily small error.


Markov Chain Model Check Transition System Reward Function State Reward 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abdulla, P.A., Baier, C., Iyer, P., Jonsson, B.: Reasoning about probabilistic lossy channel systems. In: Palamidessi, C. (ed.) CONCUR 2000. LNCS, vol. 1877, p. 320. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  2. 2.
    Abdulla, P.A., Čerāns, K., Jonsson, B., Yih-Kuen, T.: Algorithmic analysis of programs with well quasi-ordered domains. Information and Computation 160, 109–127 (2000)MATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Abdulla, P.A., Henda, N.B., Mayr, R.: Verifying infinite Markov chains with a finite attractor or the global coarseness property. In: Proc. LICS 2005, pp. 127–136 (2005)Google Scholar
  4. 4.
    Abdulla, P.A., Henda, N.B., Mayr, R., Sandberg, S.: Eager Markov chains. Technical Report 2006-009, Department of Information Technology, Uppsala University (2006)Google Scholar
  5. 5.
    Abdulla, P.A., Henda, N.B., Mayr, R., Sandberg, S.: Limiting behavior of Markov chains with eager attractors. In: Proc. QEST 2006. IEEE Computer Society Press, Los Alamitos (to appear, 2006)Google Scholar
  6. 6.
    Abdulla, P.A., Rabinovich, A.: Verification of probabilistic systems with faulty communication. In: Gordon, A.D. (ed.) FOSSACS 2003. LNCS, vol. 2620, pp. 39–53. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  7. 7.
    Asarin, E., Collins, P.: Noisy Turing machines. In: Caires, L., Italiano, G.F., Monteiro, L., Palamidessi, C., Yung, M. (eds.) ICALP 2005. LNCS, vol. 3580, pp. 1031–1042. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  8. 8.
    Aziz, A., Sanwal, K., Singhal, V., Brayton, R.: Model-checking continuous-time Markov chains. ACM Trans. on Computational Logic 1(1), 162–170 (2000)CrossRefMathSciNetGoogle Scholar
  9. 9.
    Baier, C., Bertrand, N., Schnoebelen, P.: A note on the attractor-property of infinite-state Markov chains. Information Processing Letters 97(2), 58–63 (2006)MATHMathSciNetGoogle Scholar
  10. 10.
    Baier, C., Engelen, B.: Establishing qualitative properties for probabilistic lossy channel systems: An algorithmic approach. In: Katoen, J.-P. (ed.) AMAST-ARTS 1999, ARTS 1999, and AMAST-WS 1999. LNCS, vol. 1601, pp. 34–52. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  11. 11.
    Baier, C., Haverkort, B., Hermanns, H., Katoen, J.-P.: Model checking meets performance evaluation. ACM Performance Evaluation Review 32(2), 10–15 (2005)CrossRefGoogle Scholar
  12. 12.
    Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.-P.: Automated performance and dependability evaluation using model checking. In: Calzarossa, M.C., Tucci, S. (eds.) Performance 2002. LNCS, vol. 2459, pp. 261–289. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Bertrand, N., Schnoebelen, P.: Model checking lossy channels systems is probably decidable. In: Gordon, A.D. (ed.) FOSSACS 2003. LNCS, vol. 2620, pp. 120–135. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  14. 14.
    Brázdil, T., Kučera, A.: Computing the expected accumulated reward and gain for a subclass of infinite Markov chains. In: Ramanujam, R., Sen, S. (eds.) FSTTCS 2005. LNCS, vol. 3821, pp. 372–383. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  15. 15.
    Clarke, E., Grumberg, O., Peled, D.: Model Checking. MIT Press, Cambridge (1999)Google Scholar
  16. 16.
    Courcoubetis, C., Yannakakis, M.: The complexity of probabilistic verification. Journal of the ACM 42(4), 857–907 (1995)MATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    de Alfaro, L., Kwiatkowska, M.Z., Norman, G., Parker, D., Segala, R.: Symbolic model checking of probabilistic processes using mtbdds and the Kronecker representation. In: Schwartzbach, M.I., Graf, S. (eds.) ETAPS 2000 and TACAS 2000. LNCS, vol. 1785, pp. 123–137. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  18. 18.
    Esparza, J., Etessami, K.: Verifying probabilistic procedural programs. In: Lodaya, K., Mahajan, M. (eds.) FSTTCS 2004. LNCS, vol. 3328, pp. 16–31. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  19. 19.
    Esparza, J., Kučera, A., Mayr, R.: Model checking probabilistic pushdown automata. In: Proc. LICS 2004, pp. 12–21 (2004)Google Scholar
  20. 20.
    Esparza, J., Kučera, A., Mayr, R.: Quantitative analysis of probabilistic pushdown automata: Expectations and variances. In: Proc. LICS 2005, pp. 117–126 (2005)Google Scholar
  21. 21.
    Etessami, K., Yannakakis, M.: Algorithmic verification of recursive probabilistic state machines. In: Halbwachs, N., Zuck, L.D. (eds.) TACAS 2005. LNCS, vol. 3440, pp. 253–270. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  22. 22.
    Etessami, K., Yannakakis, M.: Recursive Markov Chains, Stochastic Grammars, and Monotone Systems of Nonlinear Equations. In: Diekert, V., Durand, B. (eds.) STACS 2005. LNCS, vol. 3404, pp. 340–352. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  23. 23.
    Etessami, K., Yannakakis, M.: Recursive Markov decision processes and recursive stochastic games. In: Caires, L., Italiano, G.F., Monteiro, L., Palamidessi, C., Yung, M. (eds.) ICALP 2005. LNCS, vol. 3580, pp. 891–903. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  24. 24.
    Feller, W.: An Introduction to Probability Theory and Its Applications, 2nd edn., vol. 1. Wiley & Sons, Chichester (1966)MATHGoogle Scholar
  25. 25.
    Hart, S., Sharir, M.: Probabilistic temporal logics for finite and bounded models. In: Proc. STOC 1984, pp. 1–13 (1984)Google Scholar
  26. 26.
    Iyer, P., Narasimha, M.: Probabilistic lossy channel systems. In: Bidoit, M., Dauchet, M. (eds.) CAAP 1997, FASE 1997, and TAPSOFT 1997. LNCS, vol. 1214, pp. 667–681. Springer, Heidelberg (1997)CrossRefGoogle Scholar
  27. 27.
    Kwiatkowska, M., Norman, G., Parker, D.: Probabilistic model checking in practice: Case studies with PRISM. ACM Performance Evaluation Review 32(2), 16–21 (2005)CrossRefGoogle Scholar
  28. 28.
    Lehmann, D., Shelah, S.: Reasoning with time and chance. Information and Control 53, 165–198 (1982)MATHCrossRefMathSciNetGoogle Scholar
  29. 29.
    Rabinovich, A.: Quantitative analysis of probabilistic lossy channel systems. In: Baeten, J.C.M., Lenstra, J.K., Parrow, J., Woeginger, G.J. (eds.) ICALP 2003. LNCS, vol. 2719, pp. 1008–1021. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  30. 30.
    Stǎnicǎ, P.: Good lower and upper bounds on binomial coefficients. Journal of Inequalities in Pure and Applied Mathematics 2(3) (2001)Google Scholar
  31. 31.
    Vardi, M.: Automatic verification of probabilistic concurrent finite-state programs. In: Proc. FOCS 1985, pp. 327–338 (1985)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Parosh Aziz Abdulla
    • 1
  • Noomene Ben Henda
    • 1
  • Richard Mayr
    • 2
  • Sven Sandberg
    • 1
  1. 1.Uppsala UniversitySweden
  2. 2.NC State UniversityUSA

Personalised recommendations