Skip to main content
SpringerLink
Log in

Singular Perturbations of Markov Chains and Decision Processes

  • Chapter
Handbook of Markov Decision Processes

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 40))

Abstract

In this survey we present a unified treatment of both singular and regular perturbations in finite Markov chains and decision processes. The treatment is based on the analysis of series expansions of various important entities such as the perturbed stationary distribution matrix, the deviation matrix, the mean-passage times matrix and others.

This research was supported in part by the ARC grant #A49906132

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 299.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 379.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 379.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. M. Abbad and J.A. Filar, “Perturbation and stability theory for Markov control problems”, IEEE Trans. Auto. Contr., AC-37, no. 9, pp. 1415–1420, 1992.

    Article  Google Scholar 

  2. M. Abbad and J.A. Filar, “Algorithms for singularly perturbed Markov control problems: A survey”, in Techniques in discrete-time stochastic control systems (ed.) C.T. Leondes, Series: Control and Dynamic Systems, v. 73, Academic Press, New York, 1995.

    Chapter  Google Scholar 

  3. M. Abbad, J.A. Filar and T.R. Bielecki, “Algorithms for singularly perturbed limiting average Markov control problems,” IEEE Trans. Auto. Contr. AC-37, pp. 1421–1425, 1992.

    Article  Google Scholar 

  4. E. Altman and V.G. Gaitsgory, “Stability and Singular Perturbations in Constrained Markov Decision Problems”, IEEE Trans. Auto. Control, v. 38, pp. 971–975, 1993.

    Article  Google Scholar 

  5. E. Altman, K.E. Avrachenkov, and J.A. Filar, “Asymptotic linear programming and policy improvement for singularly perturbed Markov decision processes”, ZOR: Math. Meth. Oper. Res., v. 49, pp. 97–109, 1999.

    Google Scholar 

  6. E. Altman, E. Feinberg, J.A. Filar, and V.A. Gaitsgory, “Perturbed Zero-sum Games with Applications to Dynamic Games,” in Proc. 8th International Symposium on Dynamic Games and Aplications, pp. 45–51, Maastricht, The Netherlands, 1998 (to appear in the Annals of Dynamic Games; Birkhauser).

    Google Scholar 

  7. E. Altman, A. Hordijk, and L.C.M. Kallenberg, “On the value function in constrained control of Markov chains”, ZOR: Math. Meth. Oper. Res., v. 44, pp. 387–399, 1996.

    Article  Google Scholar 

  8. J.A. Filar, E. Altman and K.E. Avrachenkov, “An asymptotic simplex method for singularly perturbed linear programs”, submitted to Operations Research Letters, 1999.

    Google Scholar 

  9. M. Andramonov, J. Filar, A. Rubinov and P. Pardalos, “Hamiltonian Cycle Problem via Markov Chains and Min-type Approaches” in Approximation and Complexity in Numerical Optimization: Continuous and Discrete Problems, Ed. P.M. Pardalos, Kluwer Academic Publishers, 2000.

    Google Scholar 

  10. K.E. Avrachenkov, “Analytic perturbation theory and its applications”, PhD Thesis, University of South Australia, 1999.

    Google Scholar 

  11. K.E. Avrachenkov and M. Haviv, “The highest singular coefficients in singular perturbation of stochastic matrices,” INRIA Sophia Antipolis, (in preparation).

    Google Scholar 

  12. K.E. Avrachenkov, M. Haviv and P.G. Howlett, “Inversion of analytic matrix functions that are singular at the origin”, SIAM J. Matrix Anal. Appl., (to appear).

    Google Scholar 

  13. K.E. Avrachenkov and J.B. Lasserre, “The fundamental matrix of singu- larly perturbed Markov chains,” Advances in Applied Probability, v. 31, (to appear).

    Google Scholar 

  14. T.R. Bielecki and J.A. Filar, “Singularly perturbed Markov control problem: Limiting average cost”, Annals of O.R., v. 28, pp. 153–168, 1991.

    Article  Google Scholar 

  15. T.R. Bielecki and L. Stettner, “Ergodic control of singularly perturbed Markov process in discrete time with general state and compact action spaces”, preprint, Mathematics Department, Northeastern Illinois University, 1996.

    Google Scholar 

  16. D. Blackwell, “Discrete dynamic programming”, Ann. Math. Stat., v. 33, pp. 719–726, 1962.

    Article  Google Scholar 

  17. M. Chen and J.A. Filar (1992),“Hamiltonian Cycles, Quadratic Programming and Ranking of Extreme Points” in Global Optimization, C. Floudas and P. Pardalos, eds. Princeton University Press.

    Google Scholar 

  18. M. Cordech, A.S. Willsky, S.S. Sastry and D.A. Castanon, “Hierarchical aggregation of linear systems with multiple time scales,” IEEE Trans. Autom. Contr., AC-28, pp. 1029–1071, 1083.

    Google Scholar 

  19. M. Cordech, A.S. Willsky, S.S. Sastry, and D.A. Castanon, “Hierarchical aggregation of singularly perturbed finite state Markov processes,” Stochastics. v. 8, pp. 259–289, 1983.

    Article  Google Scholar 

  20. P.J. Courtois, Decomposability: queueing and computer system applications, Academic Press, New York, 1977.

    Google Scholar 

  21. P.J. Courtois and G. Louchard, “Approximation of eigencharacteristics in nearly-completely decomposable stochastic systems”, Stoch. Process. Appl., v. 4, pp. 283–296, 1976.

    Article  Google Scholar 

  22. P.J. Courtois and P. Semel, “Bounds for the positive eigenvectors of nonnegative matrices and their approximation by decomposition”, JACM, v. 31, pp. 804–825, 1984.

    Article  Google Scholar 

  23. F. Delebecque and J.P. Quadrat, “Optimal control of Markov chains admitting strong and weak interactions”, Automatica, v. 17, pp. 281–296, 1981.

    Article  Google Scholar 

  24. F. Delebecque, “A reduction process for perturbed Markov chain,” SIAM J. Appl. Math., v. 43, pp. 325–350, 1983.

    Article  Google Scholar 

  25. C. Derman, Finite state Markovian decision processes, Academic Press, New York, 1970.

    Google Scholar 

  26. V.V. Ejov, J.A. Filar and M.T. Nguyen, “Hamiltonian cycles and singularly perturbed Markov chains”, School of Mathematics, University of South Australia, (in preparation).

    Google Scholar 

  27. E.A. Feinberg, “Constrained discounted Markov decision processes and Hamiltonian cycles”, Math. Oper. Res., v. 25, pp. 130–140, 2000.

    Article  Google Scholar 

  28. J.A. Filar and D. Krass, “Hamiltonian cycles and Markov chains,” Math. Oper. Res., v. 19, pp. 223–237, 1994.

    Article  Google Scholar 

  29. J.A. Filar and Ke Liu, “Hamiltonian cycle problem and singularly per- turbed Markov decision process”, in Statistics, Probability and Game Theory: Papers in Honor of David Blackwell, IMS Lecture Notes - Monograph Series, USA, 1996.

    Google Scholar 

  30. J.A. Filar and J-B Lasserre, “A Non-Standard Branch and Bound Method for the Hamiltonian Cycle Problem”, ANZIAM J. (on line), v. 42, 2000 (to appear).

    Google Scholar 

  31. J.A. Filar and K. Vrieze, Competitive Markov Decision Processes, Springer-Verlag, N.Y., 1996.

    Book  Google Scholar 

  32. V.G. Gaitsgori and A.A. Pervozvanskii, “Aggregation of states in a Markov chain with weak interactions”, Cybernetics, v. 11, pp. 441–450, 1975. (Translation of Russian original in Kibernetika, v. 11, pp. 91–98, 1975.)

    Google Scholar 

  33. V.G. Gaitsgori and A.A. Pervozvanskii, Theory of Suboptimal Decisions, Kluwer Academic Publishers, 1988.

    Google Scholar 

  34. R. Hassin, and M. Haviv, “Mean passage times and nearly uncoupled Markov chains,” SIAM Journal of Discrete Mathematics, v. 5, pp. 386–397, 1992.

    Article  Google Scholar 

  35. M. Haviv, “An approximation to the stationary distribution of a nearly completely decomposable Markov chain and its error analysis,” SIAM Journal on Algebraic and Discrete Methods, v. 7, pp. 589–594, 1986.

    Article  Google Scholar 

  36. M. Haviv, “Aggregation/disaggregation methods for computing the stationary distribution of a Markov chain,” SIAM Journal on Numerical Analysis, v. 24, pp. 952–966, 1987.

    Article  Google Scholar 

  37. M. Haviv, “More on the Rayleigh-Ritz refinement technique for nearly uncoupled matrices,” SIAM Journal of Matrix Analysis and Application, v. 10, pp. 287–293, 1989.

    Article  Google Scholar 

  38. M. Haviv, “An aggregation/disaggregation algorithm for computing the stationary distribution of a large Markov chain,” Communications in Statistics - Stochastic Models, v. 8, pp. 565–575, 1992.

    Article  Google Scholar 

  39. M. Haviv and Y. Ritov, “Series expansions for stochastic matrices,” unpublished manuscript, Department of Statistics, The Hebrew University of Jerusalem, 1989.

    Google Scholar 

  40. M. Haviv and Y. Ritov, “On series expansions for stochastic matrices,” SIAM Journal on Matrix Analysis and Applications, v. 14, pp. 670–677, 1993.

    Article  Google Scholar 

  41. M. Haviv and L. Van der Heyden, “Perturbation bounds for the stationary probabilities of a finite Markov chain,” Advances in Applied Probability, v. 16, pp. 804–818, 1984.

    Article  Google Scholar 

  42. O. Hernandez-Lerma and J.B. Lasserre, Discrete- time Markov control processes: basic optimality criteria, Springer-Verlag, New York, 1996.

    Book  Google Scholar 

  43. A. Hordijk, R. Dekker, and L.C.M. Kallenberg, “Sensitivity analysis in discounted Markovian decision problems”, OR Spectrum, v. 7, pp. 143–151, 1985.

    Article  Google Scholar 

  44. R.A. Howard, Dynamic programming and Markov processes, Cambridge, MA: MIT Press, 1960.

    Google Scholar 

  45. Y. Huang and A.F. Veinott Jr., “Markov branching decision chains with interest-rate-dependent rewards”, Probability in the Engineering and Information Sciences, v. 9, pp. 99–121, 1995.

    Article  Google Scholar 

  46. R.G. Jeroslow, “Asymptotic Linear Programming”, Oper. Res., v. 21, pp. 1128–1141, 1973.

    Article  Google Scholar 

  47. R.G. Jeroslow, “Linear Programs Dependent on a Single Parameter”, Disc. Math., v. 6, pp. 119–140, 1973.

    Article  Google Scholar 

  48. T. Kato, Perturbation theory for linear operators, Springer-Verlag, Berlin, 1966.

    Google Scholar 

  49. L. C. M. Kallenberg, Linear programming and finite Markovian control problems, Mathematical Centre Tracts 148, Amsterdam, 1983.

    Google Scholar 

  50. L. C. M. Kallenberg, “Survey of linear programming for standard and nonstandard Markovian control problems, Part I: Theory”, ZOR - Methods and Models in Operations Research, v. 40, pp. 1–42, 1994.

    Google Scholar 

  51. J.G. Kemeny and J.L. Snell, Finite Markov Chains, Von Nostrand, New York, 1960.

    Google Scholar 

  52. V.S. Korolyuk and A.F. Turbin, Mathematical foundations of the state lumping of large systems, Naukova Dumka, Kiev, 1978, (in Russian), translated by Kluwer Academic Publishers, Dordrecht, The Netherlands, 1996.

    Google Scholar 

  53. B.F. Lamond, “A generalized inverse method for asymptotic linear programming”, Math. Programming, v. 43, pp. 71–86, 1989.

    Article  Google Scholar 

  54. B.F. Lamond, “An efficient basis update for asymptotic linear programming”, Lin. Alg. Appl., v. 184, pp. 83–102, 1993.

    Article  Google Scholar 

  55. B. F. Lamond and M. L. Puterman, “Generalized inverses in discrete time Markov decision processes”, SIAM J. Matrix Anal. Appl., v. 10, pp. 118–134, 1989.

    Article  Google Scholar 

  56. CD. Meyer, “The role of the group generalized inverse in the theory of finite Markov chains,” SIAM Review, v. 17, pp. 443–464, 1975.

    Article  Google Scholar 

  57. B. L. Miller and A. F. Veinott Jr., “Discrete dynamic programming with a small interest rate”, Ann. Math. Stat. v. 40, pp. 366–370, 1969.

    Article  Google Scholar 

  58. A.A. Pervozvanski and V.G. Gaitsgori, Theory of suboptimal decisions, Kluwer Academic Publishers, Dordrecht, The Netherlands, 1988, (Translation from the Russian original: Decomposition, aggregation and approximate optimization, Nauka, Moscow, 1979.)

    Google Scholar 

  59. A.A. Pervozvanskii and I.N. Smirnov, “Stationary-state evaluation for a complex system with slowly varying couplings”, Cybernetics, v. 10, pp. 603–611, 1974. (Translation of Russian original in Kibernetika, v. 10, pp. 45–51, 1974.)

    Google Scholar 

  60. R.G. Phillips and P.V. Kokotovic, “A singular perturbation approach to modeling and control of Markov chains”, IEEE Trans. Auto. Contr., AC-26, no. 5, pp. 1087–1094, 1981.

    Article  Google Scholar 

  61. M. L. Puterman, Markov decision processes, John Wiley & Sons, New York, 1994.

    Book  Google Scholar 

  62. J.P. Quadrat, “Optimal control of perturbed Markov chains: the multitime scale case”, in Singular perturbations in systems and control, ed. M.D. Ardema, CISM Courses and Lectures no. 280, Springer-Verlag, Wien - New York, 1983.

    Google Scholar 

  63. J.R. Rohlicek and A.S. Willsky, “The reduction of Markov generators: An algorithm exposing the role of transient states”, JACM, v. 35, pp. 675–696, 1988.

    Article  Google Scholar 

  64. U. Rothblum, “Resolvent expansions of matrices and applications”, Linear Algebra Appl., v. 38, pp. 33–49, 1981.

    Article  Google Scholar 

  65. P.J. Schweitzer, “Perturbation theory and finite Markov chains,” J. Appl. Probability, v. 5, pp. 401–413, 1968.

    Article  Google Scholar 

  66. P.J. Schweitzer, “Perturbation series expansion of nearly completely-decomposable Markov chains,” Working Paper Series No. 8122, The Graduate School of Management, The University of Rochester, 1981.

    Google Scholar 

  67. P.J. Schweitzer, “Aggregation methods for large Markov chains,” in Mathematical Computer Performance and Reliability, by G. Iazeolla, P.J. Courtios and A. Hordijk (editors), Elsevier Science Publishers B.V. (North Holland), pp. 275–286, 1984.

    Google Scholar 

  68. P.J. Schweitzer, “Perturbation series expansion of nearly completely-decomposable Markov chains,” in Teletraffic Analysis and Computer Performance Evaluation, by O.J. Boxma, J.W. Cohen and H.C. Tijms (editors), pp. 319–328, Elsevier Science Publishers B.V. (North Holland), 1986.

    Google Scholar 

  69. P.J. Schweitzer, “A survey of aggregation-disaggregation in large Markov chains,” in Proceedings of the First International Workshop on Numerical Solution for Markov Chains by W.J. Stewart (editor), pp. 63–87, 1991.

    Google Scholar 

  70. H.A. Simon and A. Ando, “Aggregation of variables in dynamic systems”, Econometrica, v. 29, pp. 111–138, 1961.

    Article  Google Scholar 

  71. U. Sumita and M. Rieders, “Numerical Comparison for the replacement process approach with the aggregation-disaggregation algorithm for row-continuous Markov chains” in Proceedings of the First International Workshop on Numerical Solution for Markov Chains by W.J. Stewart (editor), pp. 287–302, 1991.

    Google Scholar 

  72. Z. U. Syed, “Algorithms for stochastic games and related topics”, PhD Thesis in Mathematics, University of Illinois at Chicago, 1999.

    Google Scholar 

  73. H. Vantilborgh, “Aggregation with an error of O(ε 2)”, Journal of the Association for Computing Machinery, v. 32, pp. 161–190, 1985.

    Article  Google Scholar 

  74. A. F. Veinott Jr., “Discrete dynamic programming with sensitive discount optimality criteria”, Ann. Math. Stat. v. 40, pp. 1635–1660, 1969.

    Article  Google Scholar 

  75. A. F. Veinott Jr., “Markov decision chains”, in Studies in Optimization, eds. G. B. Dantzig and B. C. Eaves, pp. 124–159, 1974.

    Google Scholar 

  76. G.G. Yin and Q. Zhang, “Continuous-time Markov chains and applications: A singular perturbation approach”, Series: Applications of Mathematics, v. 37, Springer-Verlag, New York, 1998.

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. INRIA Sophia Antipolis, 2004 route des Lucioles, B.P. 93, 06902, France

    Konstantin E. Avrachenkov

  2. Department of Mathematics, The University of South Australia, The Levels, 5095, South Australia, Australia

    Jerzy Filar

  3. Department of Statistics, The Hebrew University, 91905, Jerusalem, Israel

    Moshe Haviv

  4. Department of Econometrics, The University of Sydney, Sydney, NSW 2006, Australia

    Moshe Haviv

Authors
  1. Konstantin E. Avrachenkov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jerzy Filar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Moshe Haviv
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. State University of New York at Stony Brook, USA

    Eugene A. Feinberg

  2. Technion—Israel Institute of Technology, Israel

    Adam Shwartz

Rights and permissions

Reprints and permissions

Copyright information

© 2002 Springer Science+Business Media New York

About this chapter

Cite this chapter

Avrachenkov, K.E., Filar, J., Haviv, M. (2002). Singular Perturbations of Markov Chains and Decision Processes. In: Feinberg, E.A., Shwartz, A. (eds) Handbook of Markov Decision Processes. International Series in Operations Research & Management Science, vol 40. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-0805-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-1-4615-0805-2_4

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4613-5248-8

  • Online ISBN: 978-1-4615-0805-2

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation