Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Discrete-time Singularly Perturbed Markov Chains

  • Chapter
Stochastic Modeling and Optimization

Abstract

This chapter provides a survey on recent developments for the study of singularly perturbed Markov chains in discrete time. We illustrate the use of singular perturbation techniques to treat large-scale discrete-time hybrid stochastic dynamic systems involving discrete event interventions. To reduce the complexity of the systems, time-scale separation and singularly perturbed Markov chains are used in the modeling and analysis. Asymptotic expansions of probability vectors and the structural properties of these Markov chains are provided.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. M. Abbad, J. A. Filar, and T. R. Bielecki ,Algorithms for singularly perturbed limiting average Markov control problems, IEEE Trans. Automat. Control, AC-37 (1992), 1421–1425.

    Article  MathSciNet  MATH  Google Scholar 

  2. G. Badowski, G. Yin, and Q. Zhang, Nearly optimal controls of discrete-time dynamic systems driven by singularly perturbed Markov chains, preprint, 2001.

    Google Scholar 

  3. D. Bertsekas, Dynamic Programming and Optimal Control, Vol. I - II, Athena Scientific, Belmont, MA, 1995.

    Google Scholar 

  4. P. Billingsley, Convergence of Probability Measures, J. Wiley, New York, 1968.

    Google Scholar 

  5. G. Blankenship, Singularly perturbed difference equations in optimal control problems, IEEE Trans. Automat. Control, T-AC 26 (1981), 911–917.

    Article  MathSciNet  MATH  Google Scholar 

  6. P. J. Courtois, Decomposability: Queueing and Computer System Applications, Academic Press, New York, 1977.

    MATH  Google Scholar 

  7. K. L. Chung, Markov Chains with Stationary Transition Probabilities, Second Edition, Springer-Verlag, New York, 1967.

    MATH  Google Scholar 

  8. M. H. A. Davis, Markov Models and Optimization, Chapman - Hall, London, 1993.

    MATH  Google Scholar 

  9. F. Delebecque and J. Quadrat, Optimal control for Markov chains admitting strong and weak interactions, Automatica, 17 (1981), 281–296.

    Article  MathSciNet  MATH  Google Scholar 

  10. A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, Springer-Verlag, New York, 1998.

    MATH  Google Scholar 

  11. G. B. Di Masi and Yu. M. Kabanov, A first order approximation for the convergence of distributions of the Cox processes with fast Markov switchings, Stochastics Stochastics Rep, 54 (1995), 211–219.

    MathSciNet  Google Scholar 

  12. S. N. Etbier and T. G. Kurtz, Markov Processes: Characterization and Convergence, J. Wiley, New York, 1986.

    Google Scholar 

  13. W. H. Fleming, Generalized solution in optimal stochastic control, Proc. URI Conf. on Control, 1982, 147–165.

    Google Scholar 

  14. W. H. Fleming and R. W. Rishel, Deterministic and Stochastic Optimal Control, Springer-Verlag, New York, 1975.

    MATH  Google Scholar 

  15. S. B. Gershwin, Manufacturing Systems Engineering, Prentice Hall, Englewood Cliffs, 1994.

    Google Scholar 

  16. W.-B. Gong, P. Kelly, and W. Zhai, Comparison schemes for discrete optimization with estimation algorithms, Proc. IEEE Conf. Decision Control, 1993, 2211–2216.

    Google Scholar 

  17. F. S. Hillier and G. J. Lieberman, Introduction to Operations Research, McGraw-Hill, New York, 6th Ed., 1995.

    Google Scholar 

  18. J. D. Hamilton and R. Susmel. Autoregressive conditional heteroskedasticity and changes in regime, J. Econometrics, 64 (1994), 307–333.

    Article  MATH  Google Scholar 

  19. B. E. Hansen, The likelihood ratio test under nonstandard conditions: Testing Markov trend model of GNP, J. Appl. Economics, 7 (1992), S61 - S82.

    Article  Google Scholar 

  20. F. C. Hoppensteadt and W. L. Miranker, Multitime methods for systems of difference equations, Studies Appl. Math., 56 (1977), 273–289.

    MathSciNet  Google Scholar 

  21. A. Il’in, R.Z. Khasminskii, and G. Yin, Singularly perturbed switching diffusions: Rapid switchings and fast diffusions, J. Optim. Theory Appl., 102 (1999), 555–591.

    Article  MathSciNet  Google Scholar 

  22. A. Il’in, R.Z. Khasminskii, and G. Yin, Asymptotic expansions of solutions of integro-differential equations for transition densities of singularly perturbed switching diffusions: Rapid switchings, J. Math. Anal. Appl., 238 (1999), 516–539.

    Article  MathSciNet  Google Scholar 

  23. M. Iosifescu, Finite Markov Processes and Their Applications, Wiley, Chichester, 1980.

    Google Scholar 

  24. S. Karlin and H. M. Taylor, A First Course in Stochastic Processes, 2nd Ed., Academic Press, New York, 1975.

    MATH  Google Scholar 

  25. S. Karlin and H. M. Taylor, A Second Course in Stochastic Processes, Academic Press, New York, 1981.

    MATH  Google Scholar 

  26. D. Kendrick, On the Leontief dynamic inverse, Quart. J. Economics, 86 (1972), 693–696.

    Article  Google Scholar 

  27. R. Z. Khasminskh and G. Yin, On transition densities of singularly perturbed diffusions with fast and slow components, SIAM J. Appl. Math., 56 (1996), 1794–1819.

    Google Scholar 

  28. R. Z. Khasminskii, G. Yin and Q. Zhang, Asymptotic expansions of singularly perturbed systems involving rapidly fluctuating Markov chains, SIAM J. Appl. Math., 56 (1996), 277–293.

    MathSciNet  MATH  Google Scholar 

  29. R. Z. Khasminskii, G. Yin and Q. Zhang, Constructing asymptotic series for probability distribution of Markov chains with weak and strong interactions, Quart. Appl. Math., LV (1997), 177–200.

    Google Scholar 

  30. P. R. Kumar and P. Varaiya, Stochastic Systems: Estimation, Identification and Adaptive Control, Prentice-Hall, Englewood Cliffs, N.J., 1984.

    Google Scholar 

  31. H. J. Kushner, Introduction to Stochastic Control Theory, Holt, Rinehart and Winston, New York, 1972.

    Google Scholar 

  32. H. J. Kushner, Weak Convergence Methods and Singularly Perturbed Stochastic Control and Filtering Problems, Birkhäuser, Boston, 1990.

    Google Scholar 

  33. H. J. Kushner and G. Yin, Stochastic Approximation Algorithms and Applications, Springer-Verlag, New York, 1997.

    MATH  Google Scholar 

  34. R. H. Liu, Q. Zhang, and G. Yin, Nearly optimal control of singularly perturbed Markov decision processes in discrete time, Appl. Math. Optim., 44 (2001), 105–129.

    Article  MathSciNet  MATH  Google Scholar 

  35. D. S. Naidu, Singular Perturbation Methodology in Control Systems, Peter Peregrinus Ltd., Stevenage Herts, UK, 1988.

    Book  MATH  Google Scholar 

  36. Z. G. Pan and T. Bassar, H°°-control of Markovian jump linear systems and solutions to associated piecewise-deterministic differential games, in New Trends in Dynamic Games and Applications, G. J. Olsder (Ed.), 61–94, Birkhäuser, Boston, 1995.

    Chapter  Google Scholar 

  37. A. A. Pervozvanskii and V. G. Gaitsgori, Theory of Suboptimal Decisions: Decomposition and Aggregation, Kluwer, Dordrecht, 1988.

    MATH  Google Scholar 

  38. R. G. Phillips and P. V. Koxotovig, A singular perturbation approach to modelling and control of Markov chains, IEEE Trans. Automat. Control, 26 (1981), 1087–1094.

    Article  MATH  Google Scholar 

  39. S. Ross, Stochastic Processes, J. Wiley - Sons, New York, 1983.

    Google Scholar 

  40. E. Seneta, Non-negative Matrices and Markov Chains, Springer-Verlag, New York, 1981.

    MATH  Google Scholar 

  41. S. P. Sethi and Q. Zhang, Hierarchical Decision Making in Stochastic Manufacturing Systems, Birkhäuser, Boston, 1994.

    Google Scholar 

  42. H. A. Simon and Aando, Aggregation of variables in dynamic systems, Econometrica, 29 (1961), 111–138.

    Article  MATH  Google Scholar 

  43. W. A. Thompson, Jr., Point Process Models with Applications to Safety and Reliability, Chapman and Hall, New York, 1988.

    Book  MATH  Google Scholar 

  44. D. N. C. Tse, R. G. Gallager, and J. N. Tsitsiklis, Statistical multiplexing of multiple time-scale Markov streams, IEEE J. Selected Areas Comm., 13 (1995), 1028–1038.

    Article  Google Scholar 

  45. D. J. White, Markov Decision Processes, Wiley, New York, 1992.

    Google Scholar 

  46. H. Yang, G. Yin, K. Yin, and Q. Zhang, Control of singularly perturbed Markov chains: A numerical study, to appear in J. Australian Math. Soc. Ser. B: Appl. Math.

    Google Scholar 

  47. G. Yin, On limit results for a class of singularly perturbed switching diffusions, J. Theoretical Probab., 14 (2001), 673–697.

    Article  MATH  Google Scholar 

  48. G. Yin and M. Kniazeva, Singularly perturbed multidimensional switching diffusions with fast and slow switchings, J. Math. Anal. Appl., 229 (1999), 605–630.

    Article  MathSciNet  MATH  Google Scholar 

  49. G. Yin and J.F. Zhang, Hybrid singular systems of differential equations, to appear in Scientia Sinica.

    Google Scholar 

  50. G. Yin and Q. Zhang, Control of dynamic systems under the influence of singularly perturbed Markov chains, J. Math. Anal. Appl., 216 (1997), 343–367.

    Article  MathSciNet  MATH  Google Scholar 

  51. G. Yin and Q. Zhang, Continuous-time Markov Chains and Applications: A Singular Perturbation Approach, Springer-Verlag, New York, 1998.

    MATH  Google Scholar 

  52. G. Yin and Q. Zhang, Singularly perturbed discrete-time Markov chains, SIAM J. Appl. Math., 61 (2000), 834–854.

    Article  MathSciNet  MATH  Google Scholar 

  53. G. Yin, Q. Zhang, and G. Badowski, Asymptotic properties of a singularly perturbed Markov chain with inclusion of transient states, Ann. Appl. Probab., 10 (2000), 549–572.

    Article  MathSciNet  MATH  Google Scholar 

  54. G. Yin, Q. Zhang, and G. Badowski, Singularly perturbed Markov chains: Convergence and aggregation, J. Multivariate Anal., 72 (2000), 208–229.

    Article  MathSciNet  MATH  Google Scholar 

  55. G. Yin, Q. Zhang, and G. Badowski, Occupation measures of singularly perturbed Markov chains with absorbing states, Acta Math. Sinica, 16 (2000), 161–180.

    Article  MathSciNet  MATH  Google Scholar 

  56. G. Yin, Q. Zhang, and G. Badowski, Decomposition and aggregation of large-dimensional Markov chains in discrete time, Proc. 40th Conf. Decision Control, 2001.

    Google Scholar 

  57. G. Yin, Q. Zhang, and Q.G. Liu, Error bounds for occupation measure of singularly perturbed Markov chains including transient states, Probab. Eng. Informational Sei., 14 (2000), 511–531.

    Article  MathSciNet  MATH  Google Scholar 

  58. G. Yin, Q. Zhang, H. Yang, and K. Yin, Discrete-time dynamic systems arising from singularly perturbed Markov chains, Nonlinear Anal., Theory, Methods Appl., 47 (2001), 4763–4774.

    MathSciNet  MATH  Google Scholar 

  59. Q. Zhang, Risk sensitive production planning of stochastic manufacturing systems: A singular perturbation approach, SIAM J. Control Optim., 33 (1995), 498–527.

    Article  MathSciNet  MATH  Google Scholar 

  60. Q. Zhang, Finite state Markovian decision processes with weak and strong interactions, Stochastics Stochastics Rep., 59 (1996), 283–304.

    MathSciNet  MATH  Google Scholar 

  61. Q. Zhang, R.H. Liu, and G. Yin, Nearly optimal controls of Markovian systems, in Stochastic Models and Optimization,D. Yao, H.Q. Zhang, and X.Y. Zhou eds., Springer-Verlag.

    Google Scholar 

  62. Q. Zhang and G. Yin, A central limit theorem for singularly perturbed nonstationary finite state Markov chains, Ann. Appl. Probab., 6 (1996), 650–670.

    Article  MathSciNet  MATH  Google Scholar 

  63. Q. Zhang and G. Yin, Structural properties of Markov chains with weak and strong interactions, Stochastic Process Appl., 70 (1997), 181–197.

    Article  MathSciNet  MATH  Google Scholar 

  64. Q. Zhang, G. Yin, and E. K. Boukas, Controlled Markov chains with weak and strong interactions: Asymptotic optimality and application in manufacturing, J. Optim. Theory Appl., 94 (1997), 169–194.

    MathSciNet  MATH  Google Scholar 

  65. Q. Zhang and G. Yin, On nearly optimal controls of hybrid LQG problems, IEEE Trans. Automat. Control, 44 (1999), 2271–2282.

    MathSciNet  MATH  Google Scholar 

Download references

Authors
  1. G. Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Q. Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2003 Springer-Verlag New York, Inc.

About this chapter

Cite this chapter

Yin, G., Zhang, Q. (2003). Discrete-time Singularly Perturbed Markov Chains. In: Stochastic Modeling and Optimization. Springer, New York, NY. https://doi.org/10.1007/978-0-387-21757-4_1

Download citation

  • DOI: https://doi.org/10.1007/978-0-387-21757-4_1

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4419-3065-1

  • Online ISBN: 978-0-387-21757-4

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation