Computational Probability pp 81-111 | Cite as

# Numerical Methods for Computing Stationary Distributions of Finite Irreducible Markov Chains

## Abstract

In this chapter our attention will be devoted to computational methods for computing stationary distributions of finite irreducible Markov chains. We let *q* _{ ij } denote the rate at which an *n*-state Markov chain moves from state *i* to state *j*. The *n* × *n* matrix *Q* whose off-diagonal elements are *q* _{ ij } and whose *i* ^{ th } diagonal element is given by \( - \sum\limits_{j = 1}^n , j \ne i\) *q* _{ ij } is called the *infinitesimal generator* of the Markov chain. It may be shown that the stationary probability vector *π*, a row vector whose *k*-th element denotes the stationary probability of being in state *k*, can be obtained by solving the homogeneous system of equations *πQ* = 0. Alternatively, the problem may be formulated as an eigenvalue problem *πP* = *π*, where *P = Q*Δ*t+I* is the stochastic matrix of transition probabilities, (Δ*t* must be chosen sufficiently small so that the probability of two or more transitions occurring in time Δ*t* is small, i.e., of order *o*(*t*)). Mathematically, the problem is therefore quite simple. Unfortunately, problems arise from the computational point of view because of the large number of states which many systems may occupy. As indicated in Chapters 1 and 2, it is not uncommon for thousands of states to be generated even for simple applications.

## Keywords

Markov Chain Iterative Method Diagonal Block Infinitesimal Generator Computational Probability## Preview

Unable to display preview. Download preview PDF.

## References

- [Aho et al., 1974]Aho, A., Hoperoft, J., and Ullman, J. (1974).
*The Design and Analysis of Computer Algorithms*. Addison-Wesley, Reading, Mass.Google Scholar - [Alexsson, 1985]Alexsson, 0. (1985). A survey of preconditioned iterative methods for linear systems of algebraic equations.
*BIT*, 25: 166–187.Google Scholar - [Arnoldi, 1951]Arnoldi, W. (1951). The principle of minimized iteration in the solution of the matrix eigenvalue problem.
*Quart. Appl*. Math, 9: 17–29.Google Scholar - [Balsamo and Pandolfi, 1988]Balsamo, S. and Pandolfi, B. (1988). Bounded aggregation in Markovian networks. In
*Computer Performance and Reliability*, pages 73–92. North Holland, Amsterdam.Google Scholar - [Barker and Plemmons, 1986]Barker, G. and Plemmons, R. (1986). Convergent iterations for computing stationary distributions of Markov chains.
*SIAM J. Alg. Disc. Meth*, 7: 390–398.CrossRefGoogle Scholar - [Berman and Plemmons, 1979]Berman, A. and Plemmons, R. (1979).
*Non-negative Matrices in the Mathematical Sciences*. Academic Press, New York.Google Scholar - [Cao and Stewart, 1985]Cao, W. and Stewart, W. (1985). Iterative aggregation/disaggregation techniques for nearly uncoupled Markov chains.
*J. Assoc. Comp. Mach*, 32 3: 702–719.CrossRefGoogle Scholar - [Chatelin, 1984a]Chatelin, F. (1984a). Iterative aggregation/disaggregation methods. In Iazeolla, G., Courtois, P., and Hordijk, A., editors,
*Mathematical Computer Performance and Reliability*, pages 199–207. North Holland, Amsterdam.Google Scholar - [Chatelin, 1984b]Chatelin, F. (1984b).
*Spectral Approximation of Linear Operators*. Academic Press, New York.Google Scholar - [Chatelin and Miranker, 1982]Chatelin, F. and Miranker, W. (1982). Acceleration by aggregation of successive approximation methods.
*Linear Algebra Appl*, 43: 17–47.CrossRefGoogle Scholar - [Courtois, 1977]Courtois, P. (1977).
*Decomposability; Queueing and Computer System Applications*. Academic Press, Orlando, Florida.Google Scholar - [Courtois and Semal, 1984]Courtois, P. and Semal, P. (1984). Bounds for the positive eigenvectors of nonnegative matrices and their approximation by decomposition.
*J. Assoc. Comp. Mach.*, 31: 804–825.CrossRefGoogle Scholar - [Courtois and Semal, 1986]Courtois, P. and Semal, P. (1986). Block iterative algorithms for stochastic matrices.
*Linear Algebra Appl*, 76: 59–70.CrossRefGoogle Scholar - [Dayar and Stewart, 1996]Dayar, T. and Stewart, W. (1996). On the effects of using the Grassmann-Taksar-Heyman method in iterative aggregationdisaggregation.
*SIAM Journal on Scientific Computing*, 17: 1–17.CrossRefGoogle Scholar - [Dayar and Stewart, 1997]Dayar, T. and Stewart, W. (1997). Quasilumpability, lower bounding coupling matrices and nearly completely decomposable Markov chains.
*SIAM Journal on Matrix Analysis and Applications*, 18: 482–498.CrossRefGoogle Scholar - [Feinberg and Chiu, 1987]Feinberg, B. and Chiu, S. (1987). A method to calculate steady state distributions of large Markov chains by aggregating states.
*Operations Research*, 35: 282–290.CrossRefGoogle Scholar - [Funderlic and Meyer, 1986]Funderlic, R. and Meyer, C. (1986). Sensitivity of the stationary distribution vector for an ergodic Markov chain.
*Linear Algebra Appl*, 76: 1–17.CrossRefGoogle Scholar - [Funderlic and Plemmons, 1984]Funderlic, R. and Plemmons, R. (1984). A combined direct-iterative method for certain M-matrix linear systems.
*SIAM J. Alg. Disc. Meth*, 5 (1): 32–42.CrossRefGoogle Scholar - [Funderlic and Plemmons, 1986]Funderlic, R. and Plemmons, R. (1986). Updating LU factorizations for computing stationary distributions.
*SIAM J. Alg. Disc. Meth*, 7: 30–42.CrossRefGoogle Scholar - [Golub and Meyer, 1986]Golub, G. and Meyer, C. (1986). Using the QR factorization and group inversion to compute, differentiate and estimate the sensitivity of stationary distributions for Markov chains. SIAM J. Alg. Disc. Meth, 7: 273–281.Google Scholar
- [Grassmann et al., 1985]Grassmann, W., Taksar, M., and Heyman, D. (1985). Regenerative analysis and steady state distributions for Markov chains. Operations Research, 33: 1107–1116.CrossRefGoogle Scholar
- [Harrod and Plemmons, 1984]Harrod, W. and Plemmons, R. (1984). Comparisons of some direct methods for computing stationary distributions of Markov chains. SIAM J. Sci. Comput, 5: 453–469.Google Scholar
- [Haviv, 1987]Haviv, M. (1987). Aggregation/disagregation methods for computing the stationary distribution of a Markov chain.
*SIAM J. Numer. Anal*, 24: 952–966.CrossRefGoogle Scholar - [Haviv, 1989]Haviv, M. (1989). More on a Rayleigh-Ritz refinement technique for nearly uncoupled stochastic matrices.
*SIAM J. Matrix Anal. Appl*, 10: 287–293.CrossRefGoogle Scholar - [Heyman, 1987]Heyman, D. (1987). Further comparisons of direct methods for computing stationary distributions of Markov chains. SIAM J. Alg. Disc. Meth, 8: 226–232.Google Scholar
- [Hoperoft and Tarjan, 1973]Hoperoft, J. and Tarjan, R. (1973). Efficient algorithms for graph manipulation.
*CA CM*, 16 (6): 372–378.Google Scholar - [Jennings and Stewart, 1975]Jennings, A. and Stewart, W. (1975). Simultaneous iteration for partial eigensolution of real matrices.
*J. IMA*, 15: 351–361.Google Scholar - [Jennings and Stewart, 1981]Jennings, A. and Stewart, W. (1981). A simultaneous iteration algorithm for real matrices.
*ACM Trans. of Math. Software*, 7: 184–198.CrossRefGoogle Scholar - [Kaufman, 1983]Kaufman, L. (1983). Matrix methods for queueing problems.
*SIAM J. Sci. Comput*, 4: 525–552.CrossRefGoogle Scholar - [Kim and Smith, 1991]Kim, D. and Smith, R. (1991). An exact aggregation algorithm for mandatory set decomposable Markov chains. In Stewart, W. J., editor,
*Numerical Solution of Markov Chains*. Marcel Dekker, New York, NY.Google Scholar - [Koury et al., 1984]Koury, R., McAllister, D., and Stewart, W. (1984). Iterative methods for computing stationary distributions of nearly completely decomposable Markov chains.
*SIAM J. Alg. Disc. Math*, 5 (2): 164–186.CrossRefGoogle Scholar - [Krieger, 1995]Krieger, U. (1995). Numerical solution of large finite Markov chains by algebraic multigrid techniques. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
- [Leutenegger and Horton, 1995]Leutenegger, S. and Horton, G. (1995). On the utility of the multi-level algorithm for the solution of nearly completely decomposable Markov chains. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
- [Lubachevsky and Mitra, 1986]Lubachevsky, B. and Mitra, D. (1986). A chaotic asynchronous algorithm for computing the fixed point of a non-negative matrix of unit spectral radius.
*J. Asoc. Comput. Mach*, 33: 130–150.CrossRefGoogle Scholar - [McAllister et al., 1984]McAllister, D., Stewart, G., and Stewart, W. (1984). On a Raleigh-Ritz refinement technique for nearly uncoupled stochastic matrices.
*Linear Alg. Applications*, 60: 1–25.CrossRefGoogle Scholar - [Meyer, 1989]Meyer, C. (1989). Stochastic complementation, uncoupling Markov chains and the theory of nearly reducible systems.
*SIAM Rev*, 31: 240–272.CrossRefGoogle Scholar - [Mitra and Tsoucas, 1988]Mitra, D. and Tsoucas, P. (1988). Relaxations for the numerical solutions of some stochastic problems.
*Stochastic Models*, 4 (3): 387–419.CrossRefGoogle Scholar - Philippe et al., 1992] Philippe, B., Saad, Y., and Stewart, W. (1992). Numerical methods in Markov chain modelling.
*Operations Research*,40(6):11561179.Google Scholar - [Rieders, 1995]Rieders, M. (1995). State space decomposition for large Markov chains. In Stewart, W. J., editor,
*Computations with Markov Chains*. Kluwer Academic Publishers, Boston.Google Scholar - [Saad, 1980]Saad, Y. (1980). Variations on Arnoldi’s method for computing eigenelements of large unsymmetric matrices.
*Lin. Alg. Appl*, 34: 269–295.CrossRefGoogle Scholar - [Saad, 1981]Saad, Y. (1981). Krylov subspace methods for solving large un-symmetric linear systems.
*Math. Comp*, 37: 105–126.CrossRefGoogle Scholar - [Saad, 1982]Saad, Y. (1982). Projection methods for solving large sparse eigen-value problems. In Kagstrom, B. and Ruhe, A., editors,
*Matrix Pencils*,*Proceedings*,*Pitea Haysbad*, pages 121–144. University of Umea, Sweden, Springer Verlag, Berlin.CrossRefGoogle Scholar - [Saad, 1984]Saad, Y. (1984). Chebyshev acceleration techniques for solving non-symmetric eigenvalue problems.
*Mathematics of Computation*, 42: 567588.Google Scholar - [Saad, 1995a]Saad, Y. (1995a).
*Iterative Methods for Sparse Linear Systems*. PWS Publishing Company, Boston, MA.Google Scholar - [Saad, 1995b]Saad, Y. (1995b). Preconditioned Krylov subspace methods for the numerical solution of Markov chains. In Stewart, W. J., editor,
*Computations with Markov Chains*. Kluwer Academic Publishers, Boston.Google Scholar - [Saad and Schultz, 1986]Saad, Y. and Schultz, M. (1986). GMRES: A generalized minimal residual algorithm for solving non-symmetric linear systems.
*SIAM J. Sci. Stat. Comput*, 7: 856–869.CrossRefGoogle Scholar - [Schweitzer, 1983]Schweitzer, P. (1983). Aggregation methods for large Markov chains. In
*Models of Computer Communication Systems*, pages 225234. University of Pisa, Italy. International Workshop on Applied Mathematics and Performance Reliability.Google Scholar - [Schweitzer, 1986]Schweitzer, P. (1986). Perturbation series expansions for nearly completely decomposable Markov chains. In Boxma, O., Cohen, J., and Tijms,
**H**., editors,*Teletraffic Analysis and Computer Performance Evaluation*, pages 319–328. Elsevier North-Holland, Amsterdam.Google Scholar - [Schweitzer and Kindle, 1986]Schweitzer, P. and Kindle, K. (1986). An iterative aggregation-disaggregation algorithm for solving linear systems.
*Applied Math. and Comp*, 18: 313–353.CrossRefGoogle Scholar - [Sheskin, 1985]Sheskin, T. (1985). A Markov chain partitioning algorithm for computing steady state probabilities.
*Operations Research*, 33: 228–235.CrossRefGoogle Scholar - [Simon and Ando, 1961]Simon, H. and Ando, A. (1961). Aggregation of variables in dynamic systems.
*Econometrica*, 29: 111–138.CrossRefGoogle Scholar - [Stewart, 1976]Stewart, G. (1976). Simultaneous iteration for computing in- variant subspaces of non-Hermitian matrices.
*Numer. Mat*, 25: 123–136.CrossRefGoogle Scholar - [Stewart, 1983]Stewart, G. (1983). Computable error bounds for aggregated Markov chains.
*J. Assoc. Comp. Mach.*, 30: 271–285.CrossRefGoogle Scholar - [Stewart, 1991]Stewart, G. (1991). On the sensitivity of nearly uncoupled Markov chains. In Stewart, W. J., editor,
*Numerical Solution of Markov Chains*. Marcel Dekker, New York, NY.Google Scholar - [Stewart et al., 1993]Stewart, G., Stewart, W., and McAllister, D. (1993). A two stage iteration for solving nearly uncoupled Markov chains. In
*Recent Advances in Iterative Methods*, volume 60 of*IMA Volumes in Mathematics and its Applications*, pages 201–216. Springer Verlag, New York.Google Scholar - [Stewart, 1978]Stewart, W. (1978). A comparison of numerical techniques in Markov modelling.
*Comm. ACM*, 21: 144–151.CrossRefGoogle Scholar - Stewart, 1994] Stewart, W. (1994).
*An Introduction to the Numerical Solution of Markov Chains*. Princeton University Press„ New Jersey.Google Scholar - [Stewart and Jennings, 1981]Stewart, W. and Jennings, A. (1981). A simultaneous iteration algorithm for real matrices.
*ACM Trans. Math. Software*, 7: 184–198.CrossRefGoogle Scholar - [Stewart and Wu, 1992]Stewart, W. and Wu, W. (1992). Numerical experiments with iteration and aggregation for Markov chains.
*ORSA Journal on Computing*, 4: 336–350.CrossRefGoogle Scholar - [Sumita and Rieders, 1991]Sumita, U. and Rieders, M. (1991). A comparison of the replacement process with aggregation-disaggregation. In Stewart, W. J., editor,
*Numerical Solution of Markov Chains*. Marcel Dekker, New York, NY.Google Scholar - [Takahashi, 1975]Takahashi, Y. (1975). A lumping method for numerical calculation of stationary distributions of Markov chains. Technical report, Department of Information Sciences, Tokyo Institute of Technology, Tokyo, Japan.Google Scholar
- [Tarjan, 1972]Tarjan, R. (1972). Depth first search and linear graph algorithms.
*SIAM J. Comput.*, 1 (2): 146–160.CrossRefGoogle Scholar - [Vantilborgh, 1985]Vantilborgh, H. (1985). Agreggation with an error of o(e2).
*JACM*, 32: 161–190.CrossRefGoogle Scholar - [Varga, 1962]Varga, R. (1962).
*Matrix Iterative Analysis*. Prentice Hall, Englewood Cliffs, NJ.Google Scholar - Wallace, 1973] Wallace, V. (1973). Towards an algebraic theory of Markovian networks. In
*Proc. Symp. Computer-Communication Networks and Teletraffic.*Polytechnic Press, New York.Google Scholar - Wallace and Rosenberg, 1966a] Wallace, V. and Rosenberg, R. (1966a). Markovian models and numerical analysis of computer system behavior. In
*Proc. AFIPS Spring Joint Computer Conference.*AFIPS Press, New Jersey.Google Scholar - [Wallace and Rosenberg, 1966b]Wallace, V. and Rosenberg, R. (1966b). RQA1: The recursive queue analyzer. Technical Report 2, Systems Engineering Laboratory, University of Michigan, Ann Arbor, Michigan.Google Scholar
- [Young, 1971]Young, D. (1971).
*Iterative solution of large linear systems*. Academic Press, New York.Google Scholar - [Zarling, 1976]Zarling, R. (1976).
*Numerical solutions of nearly completely decomposable queueing networks*. PhD thesis, Department of Computer Science, University of North Carolina.Google Scholar