Numerical Methods for Computing Stationary Distributions of Finite Irreducible Markov Chains

  • William J. Stewart
Chapter
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 24)

Abstract

In this chapter our attention will be devoted to computational methods for computing stationary distributions of finite irreducible Markov chains. We let q ij denote the rate at which an n-state Markov chain moves from state i to state j. The n × n matrix Q whose off-diagonal elements are q ij and whose i th diagonal element is given by \( - \sum\limits_{j = 1}^n , j \ne i\) q ij is called the infinitesimal generator of the Markov chain. It may be shown that the stationary probability vector π, a row vector whose k-th element denotes the stationary probability of being in state k, can be obtained by solving the homogeneous system of equations πQ = 0. Alternatively, the problem may be formulated as an eigenvalue problem πP = π, where P = QΔt+I is the stochastic matrix of transition probabilities, (Δt must be chosen sufficiently small so that the probability of two or more transitions occurring in time Δt is small, i.e., of order o(t)). Mathematically, the problem is therefore quite simple. Unfortunately, problems arise from the computational point of view because of the large number of states which many systems may occupy. As indicated in Chapters 1 and 2, it is not uncommon for thousands of states to be generated even for simple applications.

Keywords

Markov Chain Iterative Method Diagonal Block Infinitesimal Generator Computational Probability 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [Aho et al., 1974]
    Aho, A., Hoperoft, J., and Ullman, J. (1974). The Design and Analysis of Computer Algorithms. Addison-Wesley, Reading, Mass.Google Scholar
  2. [Alexsson, 1985]
    Alexsson, 0. (1985). A survey of preconditioned iterative methods for linear systems of algebraic equations. BIT, 25: 166–187.Google Scholar
  3. [Arnoldi, 1951]
    Arnoldi, W. (1951). The principle of minimized iteration in the solution of the matrix eigenvalue problem. Quart. Appl. Math, 9: 17–29.Google Scholar
  4. [Balsamo and Pandolfi, 1988]
    Balsamo, S. and Pandolfi, B. (1988). Bounded aggregation in Markovian networks. In Computer Performance and Reliability, pages 73–92. North Holland, Amsterdam.Google Scholar
  5. [Barker and Plemmons, 1986]
    Barker, G. and Plemmons, R. (1986). Convergent iterations for computing stationary distributions of Markov chains. SIAM J. Alg. Disc. Meth, 7: 390–398.CrossRefGoogle Scholar
  6. [Berman and Plemmons, 1979]
    Berman, A. and Plemmons, R. (1979). Non-negative Matrices in the Mathematical Sciences. Academic Press, New York.Google Scholar
  7. [Cao and Stewart, 1985]
    Cao, W. and Stewart, W. (1985). Iterative aggregation/disaggregation techniques for nearly uncoupled Markov chains. J. Assoc. Comp. Mach, 32 3: 702–719.CrossRefGoogle Scholar
  8. [Chatelin, 1984a]
    Chatelin, F. (1984a). Iterative aggregation/disaggregation methods. In Iazeolla, G., Courtois, P., and Hordijk, A., editors, Mathematical Computer Performance and Reliability, pages 199–207. North Holland, Amsterdam.Google Scholar
  9. [Chatelin, 1984b]
    Chatelin, F. (1984b). Spectral Approximation of Linear Operators. Academic Press, New York.Google Scholar
  10. [Chatelin and Miranker, 1982]
    Chatelin, F. and Miranker, W. (1982). Acceleration by aggregation of successive approximation methods. Linear Algebra Appl, 43: 17–47.CrossRefGoogle Scholar
  11. [Courtois, 1977]
    Courtois, P. (1977). Decomposability; Queueing and Computer System Applications. Academic Press, Orlando, Florida.Google Scholar
  12. [Courtois and Semal, 1984]
    Courtois, P. and Semal, P. (1984). Bounds for the positive eigenvectors of nonnegative matrices and their approximation by decomposition. J. Assoc. Comp. Mach., 31: 804–825.CrossRefGoogle Scholar
  13. [Courtois and Semal, 1986]
    Courtois, P. and Semal, P. (1986). Block iterative algorithms for stochastic matrices. Linear Algebra Appl, 76: 59–70.CrossRefGoogle Scholar
  14. [Dayar and Stewart, 1996]
    Dayar, T. and Stewart, W. (1996). On the effects of using the Grassmann-Taksar-Heyman method in iterative aggregationdisaggregation. SIAM Journal on Scientific Computing, 17: 1–17.CrossRefGoogle Scholar
  15. [Dayar and Stewart, 1997]
    Dayar, T. and Stewart, W. (1997). Quasilumpability, lower bounding coupling matrices and nearly completely decomposable Markov chains. SIAM Journal on Matrix Analysis and Applications, 18: 482–498.CrossRefGoogle Scholar
  16. [Feinberg and Chiu, 1987]
    Feinberg, B. and Chiu, S. (1987). A method to calculate steady state distributions of large Markov chains by aggregating states. Operations Research, 35: 282–290.CrossRefGoogle Scholar
  17. [Funderlic and Meyer, 1986]
    Funderlic, R. and Meyer, C. (1986). Sensitivity of the stationary distribution vector for an ergodic Markov chain. Linear Algebra Appl, 76: 1–17.CrossRefGoogle Scholar
  18. [Funderlic and Plemmons, 1984]
    Funderlic, R. and Plemmons, R. (1984). A combined direct-iterative method for certain M-matrix linear systems. SIAM J. Alg. Disc. Meth, 5 (1): 32–42.CrossRefGoogle Scholar
  19. [Funderlic and Plemmons, 1986]
    Funderlic, R. and Plemmons, R. (1986). Updating LU factorizations for computing stationary distributions. SIAM J. Alg. Disc. Meth, 7: 30–42.CrossRefGoogle Scholar
  20. [Golub and Meyer, 1986]
    Golub, G. and Meyer, C. (1986). Using the QR factorization and group inversion to compute, differentiate and estimate the sensitivity of stationary distributions for Markov chains. SIAM J. Alg. Disc. Meth, 7: 273–281.Google Scholar
  21. [Grassmann et al., 1985]
    Grassmann, W., Taksar, M., and Heyman, D. (1985). Regenerative analysis and steady state distributions for Markov chains. Operations Research, 33: 1107–1116.CrossRefGoogle Scholar
  22. [Harrod and Plemmons, 1984]
    Harrod, W. and Plemmons, R. (1984). Comparisons of some direct methods for computing stationary distributions of Markov chains. SIAM J. Sci. Comput, 5: 453–469.Google Scholar
  23. [Haviv, 1987]
    Haviv, M. (1987). Aggregation/disagregation methods for computing the stationary distribution of a Markov chain. SIAM J. Numer. Anal, 24: 952–966.CrossRefGoogle Scholar
  24. [Haviv, 1989]
    Haviv, M. (1989). More on a Rayleigh-Ritz refinement technique for nearly uncoupled stochastic matrices. SIAM J. Matrix Anal. Appl, 10: 287–293.CrossRefGoogle Scholar
  25. [Heyman, 1987]
    Heyman, D. (1987). Further comparisons of direct methods for computing stationary distributions of Markov chains. SIAM J. Alg. Disc. Meth, 8: 226–232.Google Scholar
  26. [Hoperoft and Tarjan, 1973]
    Hoperoft, J. and Tarjan, R. (1973). Efficient algorithms for graph manipulation. CA CM, 16 (6): 372–378.Google Scholar
  27. [Jennings and Stewart, 1975]
    Jennings, A. and Stewart, W. (1975). Simultaneous iteration for partial eigensolution of real matrices. J. IMA, 15: 351–361.Google Scholar
  28. [Jennings and Stewart, 1981]
    Jennings, A. and Stewart, W. (1981). A simultaneous iteration algorithm for real matrices. ACM Trans. of Math. Software, 7: 184–198.CrossRefGoogle Scholar
  29. [Kaufman, 1983]
    Kaufman, L. (1983). Matrix methods for queueing problems. SIAM J. Sci. Comput, 4: 525–552.CrossRefGoogle Scholar
  30. [Kim and Smith, 1991]
    Kim, D. and Smith, R. (1991). An exact aggregation algorithm for mandatory set decomposable Markov chains. In Stewart, W. J., editor, Numerical Solution of Markov Chains. Marcel Dekker, New York, NY.Google Scholar
  31. [Koury et al., 1984]
    Koury, R., McAllister, D., and Stewart, W. (1984). Iterative methods for computing stationary distributions of nearly completely decomposable Markov chains. SIAM J. Alg. Disc. Math, 5 (2): 164–186.CrossRefGoogle Scholar
  32. [Krieger, 1995]
    Krieger, U. (1995). Numerical solution of large finite Markov chains by algebraic multigrid techniques. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
  33. [Leutenegger and Horton, 1995]
    Leutenegger, S. and Horton, G. (1995). On the utility of the multi-level algorithm for the solution of nearly completely decomposable Markov chains. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
  34. [Lubachevsky and Mitra, 1986]
    Lubachevsky, B. and Mitra, D. (1986). A chaotic asynchronous algorithm for computing the fixed point of a non-negative matrix of unit spectral radius. J. Asoc. Comput. Mach, 33: 130–150.CrossRefGoogle Scholar
  35. [McAllister et al., 1984]
    McAllister, D., Stewart, G., and Stewart, W. (1984). On a Raleigh-Ritz refinement technique for nearly uncoupled stochastic matrices. Linear Alg. Applications, 60: 1–25.CrossRefGoogle Scholar
  36. [Meyer, 1989]
    Meyer, C. (1989). Stochastic complementation, uncoupling Markov chains and the theory of nearly reducible systems. SIAM Rev, 31: 240–272.CrossRefGoogle Scholar
  37. [Mitra and Tsoucas, 1988]
    Mitra, D. and Tsoucas, P. (1988). Relaxations for the numerical solutions of some stochastic problems. Stochastic Models, 4 (3): 387–419.CrossRefGoogle Scholar
  38. Philippe et al., 1992] Philippe, B., Saad, Y., and Stewart, W. (1992). Numerical methods in Markov chain modelling. Operations Research,40(6):11561179.Google Scholar
  39. [Rieders, 1995]
    Rieders, M. (1995). State space decomposition for large Markov chains. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
  40. [Saad, 1980]
    Saad, Y. (1980). Variations on Arnoldi’s method for computing eigenelements of large unsymmetric matrices. Lin. Alg. Appl, 34: 269–295.CrossRefGoogle Scholar
  41. [Saad, 1981]
    Saad, Y. (1981). Krylov subspace methods for solving large un-symmetric linear systems. Math. Comp, 37: 105–126.CrossRefGoogle Scholar
  42. [Saad, 1982]
    Saad, Y. (1982). Projection methods for solving large sparse eigen-value problems. In Kagstrom, B. and Ruhe, A., editors, Matrix Pencils, Proceedings, Pitea Haysbad, pages 121–144. University of Umea, Sweden, Springer Verlag, Berlin.CrossRefGoogle Scholar
  43. [Saad, 1984]
    Saad, Y. (1984). Chebyshev acceleration techniques for solving non-symmetric eigenvalue problems. Mathematics of Computation, 42: 567588.Google Scholar
  44. [Saad, 1995a]
    Saad, Y. (1995a). Iterative Methods for Sparse Linear Systems. PWS Publishing Company, Boston, MA.Google Scholar
  45. [Saad, 1995b]
    Saad, Y. (1995b). Preconditioned Krylov subspace methods for the numerical solution of Markov chains. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
  46. [Saad and Schultz, 1986]
    Saad, Y. and Schultz, M. (1986). GMRES: A generalized minimal residual algorithm for solving non-symmetric linear systems. SIAM J. Sci. Stat. Comput, 7: 856–869.CrossRefGoogle Scholar
  47. [Schweitzer, 1983]
    Schweitzer, P. (1983). Aggregation methods for large Markov chains. In Models of Computer Communication Systems, pages 225234. University of Pisa, Italy. International Workshop on Applied Mathematics and Performance Reliability.Google Scholar
  48. [Schweitzer, 1986]
    Schweitzer, P. (1986). Perturbation series expansions for nearly completely decomposable Markov chains. In Boxma, O., Cohen, J., and Tijms, H., editors, Teletraffic Analysis and Computer Performance Evaluation, pages 319–328. Elsevier North-Holland, Amsterdam.Google Scholar
  49. [Schweitzer and Kindle, 1986]
    Schweitzer, P. and Kindle, K. (1986). An iterative aggregation-disaggregation algorithm for solving linear systems. Applied Math. and Comp, 18: 313–353.CrossRefGoogle Scholar
  50. [Sheskin, 1985]
    Sheskin, T. (1985). A Markov chain partitioning algorithm for computing steady state probabilities. Operations Research, 33: 228–235.CrossRefGoogle Scholar
  51. [Simon and Ando, 1961]
    Simon, H. and Ando, A. (1961). Aggregation of variables in dynamic systems. Econometrica, 29: 111–138.CrossRefGoogle Scholar
  52. [Stewart, 1976]
    Stewart, G. (1976). Simultaneous iteration for computing in- variant subspaces of non-Hermitian matrices. Numer. Mat, 25: 123–136.CrossRefGoogle Scholar
  53. [Stewart, 1983]
    Stewart, G. (1983). Computable error bounds for aggregated Markov chains. J. Assoc. Comp. Mach., 30: 271–285.CrossRefGoogle Scholar
  54. [Stewart, 1991]
    Stewart, G. (1991). On the sensitivity of nearly uncoupled Markov chains. In Stewart, W. J., editor, Numerical Solution of Markov Chains. Marcel Dekker, New York, NY.Google Scholar
  55. [Stewart et al., 1993]
    Stewart, G., Stewart, W., and McAllister, D. (1993). A two stage iteration for solving nearly uncoupled Markov chains. In Recent Advances in Iterative Methods, volume 60 of IMA Volumes in Mathematics and its Applications, pages 201–216. Springer Verlag, New York.Google Scholar
  56. [Stewart, 1978]
    Stewart, W. (1978). A comparison of numerical techniques in Markov modelling. Comm. ACM, 21: 144–151.CrossRefGoogle Scholar
  57. Stewart, 1994] Stewart, W. (1994). An Introduction to the Numerical Solution of Markov Chains. Princeton University Press„ New Jersey.Google Scholar
  58. [Stewart and Jennings, 1981]
    Stewart, W. and Jennings, A. (1981). A simultaneous iteration algorithm for real matrices. ACM Trans. Math. Software, 7: 184–198.CrossRefGoogle Scholar
  59. [Stewart and Wu, 1992]
    Stewart, W. and Wu, W. (1992). Numerical experiments with iteration and aggregation for Markov chains. ORSA Journal on Computing, 4: 336–350.CrossRefGoogle Scholar
  60. [Sumita and Rieders, 1991]
    Sumita, U. and Rieders, M. (1991). A comparison of the replacement process with aggregation-disaggregation. In Stewart, W. J., editor, Numerical Solution of Markov Chains. Marcel Dekker, New York, NY.Google Scholar
  61. [Takahashi, 1975]
    Takahashi, Y. (1975). A lumping method for numerical calculation of stationary distributions of Markov chains. Technical report, Department of Information Sciences, Tokyo Institute of Technology, Tokyo, Japan.Google Scholar
  62. [Tarjan, 1972]
    Tarjan, R. (1972). Depth first search and linear graph algorithms. SIAM J. Comput., 1 (2): 146–160.CrossRefGoogle Scholar
  63. [Vantilborgh, 1985]
    Vantilborgh, H. (1985). Agreggation with an error of o(e2). JACM, 32: 161–190.CrossRefGoogle Scholar
  64. [Varga, 1962]
    Varga, R. (1962). Matrix Iterative Analysis. Prentice Hall, Englewood Cliffs, NJ.Google Scholar
  65. Wallace, 1973] Wallace, V. (1973). Towards an algebraic theory of Markovian networks. In Proc. Symp. Computer-Communication Networks and Teletraffic. Polytechnic Press, New York.Google Scholar
  66. Wallace and Rosenberg, 1966a] Wallace, V. and Rosenberg, R. (1966a). Markovian models and numerical analysis of computer system behavior. In Proc. AFIPS Spring Joint Computer Conference. AFIPS Press, New Jersey.Google Scholar
  67. [Wallace and Rosenberg, 1966b]
    Wallace, V. and Rosenberg, R. (1966b). RQA1: The recursive queue analyzer. Technical Report 2, Systems Engineering Laboratory, University of Michigan, Ann Arbor, Michigan.Google Scholar
  68. [Young, 1971]
    Young, D. (1971). Iterative solution of large linear systems. Academic Press, New York.Google Scholar
  69. [Zarling, 1976]
    Zarling, R. (1976). Numerical solutions of nearly completely decomposable queueing networks. PhD thesis, Department of Computer Science, University of North Carolina.Google Scholar

Copyright information

© Springer Science+Business Media New York 2000

Authors and Affiliations

  • William J. Stewart
    • 1
  1. 1.Department of Computer ScienceNorth Carolina State UniversityRaleighUSA

Personalised recommendations