Skip to main content

Coefficients of ergodicity for Markov chains with uncertain parameters

Metrika volume 76, pages 107–133 (2013)Cite this article

Abstract

One of the central considerations in the theory of Markov chains is their convergence to an equilibrium. Coefficients of ergodicity provide an efficient method for such an analysis. Besides giving sufficient and sometimes necessary conditions for convergence, they additionally measure its rate. In this paper we explore coefficients of ergodicity for the case of imprecise Markov chains. The latter provide a convenient way of modelling dynamical systems where parameters are not determined precisely. In such cases a tool for measuring the rate of convergence is even more important than in the case of precisely determined Markov chains, since most of the existing methods of estimating the limit distributions are iterative. We define a new coefficient of ergodicity that provides necessary and sufficient conditions for convergence of the most commonly used class of imprecise Markov chains. This so-called weak coefficient of ergodicity is defined through an endowment of the structure of a metric space to the class of imprecise probabilities. Therefore we first make a detailed analysis of the metric properties of imprecise probabilities.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

References

  • Beer G (1993) Topologies on closed and closed convex sets. Kluwer, Dordrecht

    MATH  Google Scholar 

  • Birkhoff G (1957) Extensions of Jentzsch’s theorem. Trans Am Math Soc 85(1): 219–227

    MathSciNet  MATH  Google Scholar 

  • Crossman RJ, Škulj D (2010) Imprecise Markov chains with absorption. Int J Approx Reason 51: 1085–1099. doi:10.1016/j.ijar.2010.08.008

    MATH  Article  Google Scholar 

  • Crossman RJ, Coolen-Schrijner P, Coolen FPA (2009a) Time-homogeneous birth-death processes with probability intervals and absorbing state. J Stat Theory Practice 3(1): 103–118

    MathSciNet  MATH  Article  Google Scholar 

  • Crossman RJ, Coolen-Schrijner P, Škulj D, Coolen FPA (2009b) Imprecise Markov chains with an absorbing state. In: Augustin T, Coolen FPA, Moral S, Troffaes MCM (eds) ISIPTA’09: proceedings of the sixth international symposium on imprecise probability: theories and applications, SIPTA, Durham, UK, pp 119–128

  • Darroch JN, Seneta E (1965) On quasi-stationary distributions in absorbing discrete-time finite Markov chains. J Appl Probab 2(1):88–100, http://www.jstor.org/stable/3211876

    Google Scholar 

  • de Campos LD, Huete J, Moral S (1994) Probability intervals: a tool for uncertain reasoning. Int J Uncertain Fuzz Knowl Based Syst 2(2): 167–196

    MATH  Article  Google Scholar 

  • De Cooman G, Hermans F, Quaeghebeur E (2009) Imprecise Markov chains and their limit behavior. Probab Eng Inform Sci 23(4): 597–635. doi:10.1017/S0269964809990039

    MathSciNet  MATH  Article  Google Scholar 

  • Dobrushin R (1956) Central limit theorem for non-stationary Markov chains, I, II. Theory Probab Appl 1(4): 329–383

    Article  Google Scholar 

  • Dunford N, Schwartz J (1988) Linear operators. Part I: general theory. Wiley, New York

    MATH  Google Scholar 

  • Fan K (1953) Minimax theorems. Proc Natl Acad Sci USA 39: 42–47

    MATH  Article  Google Scholar 

  • Hable R (2009) Data-based decisions under complex uncertainty. PhD thesis, Ludwig-Maximilians-Universität (LMU) Munich, http://edoc.ub.uni-muenchen.de/9874/

  • Hable R (2010) Minimum distance estimation in imprecise probability models. J Stat Plan Inference 140: 461–479

    MathSciNet  MATH  Article  Google Scholar 

  • Harmanec D (2002) Generalizing Markov decision processes to imprecise probabilities. J Stat Plan Inference 105: 199–213

    MathSciNet  MATH  Article  Google Scholar 

  • Hartfiel D (1998) Markov set-chains. Springer, Berlin

    MATH  Google Scholar 

  • Hartfiel D, Rothblum U (1998) Convergence of inhomogenous products of matrices and coefficients of ergodicity. Linear Algebra Appl 277: 1–9

    MathSciNet  MATH  Article  Google Scholar 

  • Hartfiel D, Seneta E (1994) On the theory of Markov set-chains. Adv Appl Probab 26(4): 947–964

    MathSciNet  MATH  Article  Google Scholar 

  • Holmes RB (1975) Geometric functional analysis and its applications. Springer, Berlin

    MATH  Book  Google Scholar 

  • Itoh H, Nakamura K (2007) Partially observable Markov decision processes with imprecise parameters. Artif Intell 171(8–9): 453–490

    MathSciNet  MATH  Article  Google Scholar 

  • Kozine I, Utkin L (2002) Interval-valued finite Markov chains. Reliable Comput 8(2): 97–113

    MathSciNet  MATH  Article  Google Scholar 

  • Nilim A, Ghaoui LE (2005) Robust control of Markov decision processes with uncertain transition matrices. Oper Res 53: 780–798

    MathSciNet  MATH  Article  Google Scholar 

  • Paz A (1970) Ergodic theorems for infinite probabilistic tables. Ann Math Stat 41(2): 539–550

    MathSciNet  MATH  Article  Google Scholar 

  • Satia J, Lave R (1973) Markovian decision processes with uncertain transition probabilities. Oper Res 21(3): 728–740

    MathSciNet  MATH  Article  Google Scholar 

  • Seneta E (1979) Coefficients of ergodicity—structure and applications. Adv Appl Probab 11(2): 270–271

    MathSciNet  Article  Google Scholar 

  • Seneta E (2006) Non-negative matrices and Markov chains. Springer, Berlin

    MATH  Google Scholar 

  • Škulj D (2006) Finite discrete time Markov chains with interval probabilities. In: Lawry J, Miranda E, Bugarín A, Li S, Gil MA, Grzegorzewski P, Hryniewicz O (eds) SMPS. Advances in soft computing. Springer, Berlin, vol 37, pp 299–306

  • Škulj D (2007) Regular finite Markov chains with interval probabilities. In: De Cooman G, Zaffalon M, Vejnarová J (eds) ISIPTA’07—proceedings of the fifth international symposium on imprecise probability: theories and applications, SIPTA, pp 405–413

  • Škulj D (2009) Discrete time Markov chains with interval probabilities. Int J Approx Reason 50(8): 1314–1329. doi:10.1016/j.ijar.2009.06.007

    MATH  Article  Google Scholar 

  • Škulj D, Hable R (2009) Coefficients of ergodicity for imprecise Markov chains. In: Augustin T, Coolen FPA, Moral S, Troffaes MCM (eds) ISIPTA’09: proceedings of the sixth international symposium on imprecise probability: theories and applications, SIPTA, Durham, UK, pp 377–386

  • Walley P (1991) Statistical reasoning with imprecise probabilities. Chapman and Hall, London

    MATH  Google Scholar 

  • Walley P (2000) Towards a unified theory of imprecise probability. Int J Approx Reason 24: 125–148

    MathSciNet  MATH  Article  Google Scholar 

  • Weichselberger K (2001) Elementare Grundbegriffe einer allgemeineren Wahrscheinlichkeitsrechnung. I: Intervallwahrscheinlichkeit als umfassendes Konzept. Physica-Verlag, Heidelberg

    MATH  Book  Google Scholar 

  • White C, Eldeib H (1994) Markov decision processes with imprecise transition probabilities. Oper Res 42(4): 739–749

    MathSciNet  MATH  Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Social Sciences, Kardeljeva pl. 5, 1000, Ljubljana, Slovenia

    D. Škulj

  2. Department of Mathematics, 95440, Bayreuth, Germany

    R. Hable

Authors
  1. D. Škulj
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. R. Hable
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to D. Škulj.

Additional information

An earlier version of this paper, Škulj and Hable (2009), was presented at the ISIPTA’09 conference in Durham, UK, 2009.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Škulj, D., Hable, R. Coefficients of ergodicity for Markov chains with uncertain parameters. Metrika 76, 107–133 (2013). https://doi.org/10.1007/s00184-011-0378-0

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00184-011-0378-0

Keywords

  • Markov chain
  • Imprecise Markov chain
  • Coefficient of ergodicity
  • Weak coefficient of ergodicity
  • Uniform coefficient of ergodicity
  • Lower expectation
  • Upper expectation
  • Hausdorff metric
  • Convergence of Markov chains