Deductive Proofs of Almost Sure Persistence and Recurrence Properties

  • Aleksandar ChakarovEmail author
  • Yuen-Lam VoroninEmail author
  • Sriram SankaranarayananEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9636)


Martingale theory yields a powerful set of tools that have recently been used to prove quantitative properties of stochastic systems such as stochastic safety and qualitative properties such as almost sure termination. In this paper, we examine proof techniques for establishing almost sure persistence and recurrence properties of infinite-state discrete time stochastic systems. A persistence property \(\Diamond \Box (P)\) specifies that almost all executions of the stochastic system eventually reach P and stay there forever. Likewise, a recurrence property \(\Box \Diamond (Q)\) specifies that a target set Q is visited infinitely often by almost all executions of the stochastic system. Our approach extends classic ideas on the use of Lyapunov-like functions to establish qualitative persistence and recurrence properties. Next, we extend known constraint-based invariant synthesis techniques to deduce the necessary supermartingale expressions to partly mechanize such proofs. We illustrate our techniques on a set of interesting examples.


Temporal logic Stochastic systems Markov processes Stochastic control Sum-of-squares programming 



We thank the anonymous reviewers for their comments. This work was supported by the US National Science Foundation (NSF) under award numbers 1527075 and 1320069. All opinions expressed are those of the authors and not necessarily of the US NSF.


  1. 1.
    Abate, A.: A contractivity approach for probabilistic bisimulations of diffusion processes. In: Proceedings of CDC, pp. 2230–2235 (2009)Google Scholar
  2. 2.
    Abate, A.: Probabilistic bisimulations of switching and resetting diffusions. In: Proceedings of CDC, pp. 5918–5923 (2010)Google Scholar
  3. 3.
    Abate, A., Katoen, J.-P., Lygeros, J., Prandini, M.: Approximate model checking of stochastic hybrid systems. Eur. J. Control 16(6), 624–641 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Anjos, M.F., Lasserre, J.B.: Handbook on Semidefinite, Conic and Polynomial Optimization. International Series in Operations Research & Management Science, vol. 166. Springer, New York (2012)CrossRefzbMATHGoogle Scholar
  5. 5.
    Baier, C., Katoen, J.-P., et al.: Principles of Model Checking, vol. 26202649. MIT press, Cambridge (2008)zbMATHGoogle Scholar
  6. 6.
    Blekherman, G., Parrilo, P.A., Thomas, R.R.: Semidefinite optimization and convex algebraic geometry. In: MOS-SIAM Series on Optimization. Society for Industrial and Applied Mathematics (SIAM), vol. 13. Mathematical Optimization Society, Philadelphia, PA (2013)Google Scholar
  7. 7.
    Byl, K., Tedrake, R.: Metastable walking machines. Int. J. Robot. Res. 28(8), 1040–1064 (2009)CrossRefGoogle Scholar
  8. 8.
    Chakarov, A., Sankaranarayanan, S.: Probabilistic program analysis with martingales. In: Sharygina, N., Veith, H. (eds.) CAV 2013. LNCS, vol. 8044, pp. 511–526. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  9. 9.
    Chakarov, A., Sankaranarayanan, S.: Expectation invariants for probabilistic program loops as fixed points. In: Müller-Olm, M., Seidl, H. (eds.) Static Analysis. LNCS, vol. 8723, pp. 85–100. Springer, Heidelberg (2014)Google Scholar
  10. 10.
    Chatterjee, K., Fu, H., Hasheminezhad, R., Novotny, P.: Algorithmic analysis of qualitative and quantitative termination problems for affine probabilistic programs. In: Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POpPL , St. Petersburg, Florida, United States, 20–22 January 2016Google Scholar
  11. 11.
    Courcoubetis, C., Yannakakis, M.: The complexity of probabilistic verification. J. ACM (JACM) 42(4), 857–907 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    De Alfaro, L.: Formal verification of probabilistic systems. PhD thesis, Stanford University (1997)Google Scholar
  13. 13.
    De Alfaro, L.: How to specify and verify the long-run average behaviour of probabilistic systems. In: Proceedings of LICS, pp. 454–465. IEEE (1998)Google Scholar
  14. 14.
    Durrett, R.: Probability: theory and examples. Cambridge University Press, Cambridge (2010)CrossRefzbMATHGoogle Scholar
  15. 15.
    Fioriti, L.M.F., Hartmanns, A., Hermann, H.: Probabilistic termination: Soundness, completeness, and compositionality. In: Proceedings of POPL, pp. 489–501. ACM (2015)Google Scholar
  16. 16.
    Foster, F.G.: On the stochastic matrices associated with certain queuing processes. Ann. Math. Stat. 24(3), 355–360 (1953)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Gordon, A.D., Henzinger, T.A., Nori, A.V., Rajamani, S.K.: Probabilistic programming. In: Proceedings of FOSE 2014, pp. 167–181 (2014)Google Scholar
  18. 18.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects Comput. 6(5), 512–535 (1994)CrossRefzbMATHGoogle Scholar
  19. 19.
    Katoen, J.-P., McIver, A.K., Meinicke, L.A., Morgan, C.C.: Linear-invariant generation for probabilistic programs: In: Cousot, R., Martel, M. (eds.) SAS 2010. LNCS, vol. 6337, pp. 390–406. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  20. 20.
    Kozen, D.: Semantics of probabilistic programs. J. Comput. Syst. Sci. 22(3), 328–350 (1981)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  22. 22.
    McGeer, T.: Passive dynamic walking. Int. J. Robot. Res. 9(2), 62–82 (1990)CrossRefGoogle Scholar
  23. 23.
    McIver, A., Morgan, C.: Abstraction, Refinement And Proof For Probabilistic Systems (Monographs in Computer Science). SpringerVerlag, New York (2004)Google Scholar
  24. 24.
    Meyn, S.P., Tweedie, R.L.: Markov Chains and Stochastic Stability. Cambridge University Press, Cambridge (2009)CrossRefzbMATHGoogle Scholar
  25. 25.
    Papachristodoulou, A., Anderson, J., Valmorbida, G., Prajna, S., Seiler, P., Parrilo, P.A.: SOSTOOLS: Sum of squares optimization toolbox for MATLAB (2013).
  26. 26.
    Prajna, S., Jadbabaie, A., Pappas, G.J.: Stochastic safety verification using barrier certificates. In: 43rd IEEE Conference on Decision and Control, CDC, vol. 1, pp. 929–934. IEEE (2004)Google Scholar
  27. 27.
    Steinhardt, J., Tedrake, R.: Finite-time regional verification of stochastic non-linear systems. Int. J. Robot. Res. 31(7), 901–923 (2012)CrossRefGoogle Scholar
  28. 28.
    Tkachev, I., Abate, A.: Stability and attractivity of absorbing sets for discrete-time Markov processes. In: IEEE 51st Annual Conference on Decision and Control (CDC), pp. 7652–7657. IEEE (2012)Google Scholar
  29. 29.
    Tkachev, I., Abate, A.: Characterization and computation of infinite-horizon specifications over markov processes. Theoret. Comput. Sci. 515, 1–18 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Tkachev, I., Mereacre, A., Katoen, J.-P., Abate, A.: Quantitative model-checking of controlled discrete-time Markov processes (2014). arXiv preprint arXiv:1407.5449
  31. 31.
    Kim-Chuan, T., Todd, M.J., Tütüncü, R.H.: On the implementation and usage of SDPT3–a matlab software package for semidefinite-quadratic-linear programming, version 4.0. In: Anjos, M.F., Lasserre, J.B. (eds.) Handbook on Semidefinite, Conic and Polynomial Optimization. nternational Series in Operations Research and Management Science, vol. 166, pp. 715–754. Springer, New York (2012)CrossRefGoogle Scholar
  32. 32.
    Williams, D.: Probability with Martingales. Cambridge University Press, Cambridge (1991)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.University of Colorado BoulderBoulderUSA

Personalised recommendations