Singularly Perturbed Markov Chains and Applications to Large-Scale Systems under Uncertainty

  • G. Yin
  • Q. Zhang
  • K. Yin
  • H. Yang
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 46)


This chapter is concerned with large-scale hybrid stochastic systems, in which the dynamics involve both continuously evolving components and discrete events. Corresponding to different discrete states, the dynamic behavior of the underlying system could be markedly different. To reduce the complexity of these systems, singularly perturbed Markov chains are used to characterize the system. Asymptotic expansions of probability vectors and the structural properties of these Markov chains are provided. The ideas of decomposition and aggregation are presented using two typical optimal control problems. Such an approach leads to control policies that are simple to obtain and perform nearly as well as the optimal ones with substantially reduced complexity.

Key words

singularly perturbation Markov chain near optimality optimal control LQG MDP 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Abbad, M., J.A. Filar, and T.R. Bielecki. (1992). Algorithms for singularly perturbed limiting average Markov control problems, IEEE Trans. Automat. ControlAC-37, 1421–1425.MathSciNetzbMATHGoogle Scholar
  2. Bertsekas, D. (1995). Dynamic Programming and Optimal Control, Vol. I & II, Athena Scientific, Belmont, MA.zbMATHGoogle Scholar
  3. Billingsley, P. (1999). Convergence of Probability Measures, 2nd Ed., J. Wiley, New York.zbMATHGoogle Scholar
  4. Blankenship, G. (1981). Singularly perturbed difference equations in optimal control problems, IEEE Trans. Automat. ControlT-AC 26, 911–917.MathSciNetzbMATHGoogle Scholar
  5. Courtois, P.J. (1977). Decomposability: Queueing and Computer System Applications, Academic Press, New York.zbMATHGoogle Scholar
  6. Chung, K.L. (1967). Markov Chains with Stationary Transition Probabilities, Second Edition, Springer-Verlag, New York.zbMATHGoogle Scholar
  7. Davis, M.H.A. (1993). Markov Models and Optimization, Chapman & Hall, London.zbMATHGoogle Scholar
  8. Delebecque, F. and J. Quadrat. (1981). Optimal control for Markov chains admitting strong and weak interactions, Automatica17, 281–296.MathSciNetzbMATHGoogle Scholar
  9. Dembo, A. and O. Zeitouni. (1998). Large Deviations Techniques and Applications, Springer-Verlag, New York.zbMATHGoogle Scholar
  10. Di Masi, G.B. and Yu. M. Kabanov. (1995). A first order approximation for the convergence of distributions of the cox processes with fast Markov switchings, Stochastics Stochastics Rep.54, 211–219.MathSciNetzbMATHGoogle Scholar
  11. Ethier, S.N. and T.G. Kurtz. (1986). Markov Processes: Characterization and Convergence, J. Wiley, New York.zbMATHGoogle Scholar
  12. Fleming, W.H. and R.W. Rishel. (1975). Deterministic and Stochastic Optimal Control, Springer-Verlag, New York.zbMATHGoogle Scholar
  13. Fox, G. and P. Glynn. (1990). Discrete-time conversion for simulating finitehorizon Markov processes, SIAM J. Appl. Math., 50, 1457–1473.MathSciNetzbMATHGoogle Scholar
  14. Gershwin, S.B. (1994). Manufacturing Systems Engineering, Prentice Hall, Englewood Cliffs, 1994.Google Scholar
  15. Hillier, F.S. and G.J. Lieberman. (1995). Introduction to Operations Research, McGraw-Hill, New York, 6th Ed.zbMATHGoogle Scholar
  16. Hamilton, J.D. and R. Susmel. (1994). Autoregressive conditional heteroskedasticity and changes in regime, J. Econometrics, 64, 307–333.zbMATHGoogle Scholar
  17. Hansen, B.E. (1992). The likelihood ratio test under nonstandard conditions: Testing Markov trend model of GNP, J. Appl. Economics, 7, S61–S82.Google Scholar
  18. Hoppensteadt, F.C. and W.L. Miranker. (1977). Multitime methods for systems of difference equations, Studies Appl. Math.56, 273–289.MathSciNetzbMATHGoogle Scholar
  19. Il’in, A., R.Z. Khasminskii, and G. Yin. (1999a). Singularly perturbed switching diffusions: Rapid switchings and fast diffusions, J. Optim. Theory Appl.102, 555–591.MathSciNetzbMATHGoogle Scholar
  20. Il’in, A., R.Z. Khasminskii,and G. Yin. (1999b). Asymptotic expansions of solutions of integro-differential equations for transition densities of singularly perturbed switching diffusions: Rapid switchings, J. Math. Anal. Appl.238, 516–539.MathSciNetzbMATHGoogle Scholar
  21. Iosifescu, M. (1980). Finite Markov Processes and Their Applications, Wiley, Chichester.zbMATHGoogle Scholar
  22. Karlin, S. and H.M. Taylor. (1975). A First Course in Stochastic Processes, 2nd Ed., Academic Press, New York.zbMATHGoogle Scholar
  23. Karlin, S. and H.M. Taylor. (1981). A Second Course in Stochastic Processes, Academic Press, New York.zbMATHGoogle Scholar
  24. Kendrick, D. (1972). On the Leontief dynamic inverse, Quart. J. Economics, 86, 693–696.Google Scholar
  25. Khasminskii, R.Z. and G. Yin. (1996). On transition densities of singularly perturbed diffusions with fast and slow components, SIAM J. Appl. Math., 56, 1794–1819.MathSciNetzbMATHGoogle Scholar
  26. Khasminskii, R.Z., G. Yin, and Q. Zhang. (1996). Asymptotic expansions of singularly perturbed systems involving rapidly fluctuating Markov chains, SIAM J. Appl. Math.56, 277–293.MathSciNetzbMATHGoogle Scholar
  27. Khasminskii, R.Z., G. Yin, and Q. Zhang. (1997). Constructing asymptotic series for probability distribution of Markov chains with weak and strong interactions, Quart, Appl. Math.LV, 177–200.MathSciNetzbMATHGoogle Scholar
  28. Kumar, P.R. and P. Varaiya. (1984). Stochastic Systems: Estimation, Identification and Adaptive Control, Prentice-Hall, Englewood Cliffs, N.J.zbMATHGoogle Scholar
  29. Kushner, H.J. (1972). Introduction to Stochastic Control Theory, Holt, Rinehart and Winston, New York.Google Scholar
  30. Kushner, H.J. (1990). Weak Convergence Methods and Singularly Perturbed Stochastic Control and Filtering Problems, Birkhäuser, Boston.zbMATHGoogle Scholar
  31. Kushner, H.J. and G. Yin. (1997). Stochastic Approximation Algorithms and Applications, Springer-Verlag, New York.zbMATHGoogle Scholar
  32. Lai, T.-L., and S. Yakowitz. (1995). Machine learning and nonparametric bandit theory, IEEE Tran. Automatic Control40, 1199–1210.MathSciNetzbMATHGoogle Scholar
  33. Liu, R.H., Q. Zhang, and G. Yin. (2001). Nearly optimal control of singularly perturbed Markov decision processes in discrete time, to appear in Appl. Math. Optim.Google Scholar
  34. Pan, Z.G. and T. Başar. (1995). H∞-control of Markovian jump linear systems and solutions to associated piecewise-deterministic differential games, in New Trends in Dynamic Games and Applications, G. J. Olsder (Ed.), 61–94, Birkhäuser, Boston.zbMATHGoogle Scholar
  35. Pervozvanskii, A.A. and V.G. Gaitsgori. (1988). Theory of Suboptimal Decisions: Decomposition and Aggregation, Kluwer, Dordrecht.Google Scholar
  36. Phillips, R.G. and P.V. Kokotovic. (1981). A singular perturbation approach to modelling and control of Markov chains, IEEE Trans. Automat. Control26, 1087–1094.zbMATHGoogle Scholar
  37. Ross, S. (1983). Introduction to Stochastic Dynamic Programming, Academic Press, New York.zbMATHGoogle Scholar
  38. Seneta, E. (1981). Non-negative Matrices and Markov Chains, Springer-Verlag, New York.zbMATHGoogle Scholar
  39. Sethi, S.P. and Q. Zhang. (1994). Hierarchical Decision Making in Stochastic Manufacturing Systems, Birkhäuser, Boston.zbMATHGoogle Scholar
  40. Simon, H.A. and A. Ando. (1961). Aggregation of variables in dynamic systems, Econometrica29, 111–138.zbMATHGoogle Scholar
  41. Thompson, W.A. Jr. (1988). Point Process Models with Applications to Safety and Reliability, Chapman and Hall, New York.zbMATHGoogle Scholar
  42. Tse, D.N.C., R.G. Gallager, and J.N. Tsitsiklis. (1995). Statistical multiplexing of multiple time-scale Markov streams, IEEE J. Selected Areas Comm.13, 1028–1038.Google Scholar
  43. White, D.J. (1992). Markov Decision Processes, Wiley, New York.zbMATHGoogle Scholar
  44. Yakowitz, S. (1969). Mathematics of Adaptive Control Processes, Elsevier, New York.zbMATHGoogle Scholar
  45. Yakowitz, S., R. Hayes, and J. Gani. (1992). Automatic learning for dynamic Markov fields with application to epidemiology, Oper. Res.40, 867–876.zbMATHGoogle Scholar
  46. Yakowitz, S., P. L’Ecuyer, and F. Vázquez-Abad. (2000). Global stochastic optimization with low-dispersion point sets, Oper. Res., 48, 939–950.MathSciNetzbMATHGoogle Scholar
  47. Yang, H., G. Yin, K. Yin, and Q. Zhang. (2001). Control of singularly perturbed Markov chains: A numerical study, to appear in J. Australian Math. Soc. Ser. B: Appl. Math.Google Scholar
  48. Yin, G. (2001). On limit results for a class of singularly perturbed switching diffusions, to appear in J. TheoreticalProbab.Google Scholar
  49. Yin, G. and M. Kniazeva. (1999). Singularly perturbed multidimensional switching diffusions with fast and slow switchings, J. Math. Anal. Appl., 229, 605–630.MathSciNetzbMATHGoogle Scholar
  50. Yin, G. and J.F. Zhang. (2001). Hybrid singular systems of differential equations, to appear in Scientia Sinica.Google Scholar
  51. Yin, G. and Q. Zhang. (1997a). Control of dynamic systems under the influence of singularly perturbed Markov chains, J. Math. Anal. Appl.216, 343–367.MathSciNetzbMATHGoogle Scholar
  52. Yin, G. and Q. Zhang (Eds.) (1997b). Mathematics of Stochastic Manufacturing Systems, Proc. 1996 AMS-SIAM Summer Seminar in Applied Mathematics, Lectures in Applied Mathematics, LAM 33, Amer. Math. Soc., Providence, RI, Providence, RI.Google Scholar
  53. Yin, G. and Q. Zhang. (1998). Continuous-time Markov Chains and Applications: A Singular Perturbation Approach, Springer-Verlag, New York.zbMATHGoogle Scholar
  54. Yin, G. and Q. Zhang. (2000). Singularly perturbed discrete-time Markov chains, SIAM J. Appl. Math., 61, 834–854.MathSciNetzbMATHGoogle Scholar
  55. Yin, G., Q. Zhang, and G. Badowski. (2000a). Asymptotic properties of a singularly perturbed Markov chain with inclusion of transient states, Ann. Appl. Probab., 10, 549–572.MathSciNetzbMATHGoogle Scholar
  56. Yin, G., Q. Zhang, and G. Badowski. (2000b). Singularly perturbed Markov chains: Convergence and aggregation, J. Multivariate Anal.72, 208–229.MathSciNetzbMATHGoogle Scholar
  57. Yin, G., Q. Zhang, and G. Badowski. (2000c). Occupation measures of singularly perturbed Markov chains with absorbing states, Acta Math. Sinica, 16, 161–180.MathSciNetzbMATHGoogle Scholar
  58. Yin, G., Q. Zhang, and G. Badowski. (2000d). Decomposition and aggregation of large-dimensional Markov chains in discrete time, preprint.Google Scholar
  59. Yin, G., Q. Zhang, and Q.G. Liu. (2000e). Error bounds for occupation measure of singularly perturbed Markov chains including transient states, Probab. Eng. Informational Sci.14, 511–531.MathSciNetzbMATHGoogle Scholar
  60. Yin, G., Q. Zhang, H. Yang, and K. Yin. (2001). Discrete-time dynamic systems arising from singularly perturbed Markov chains, to appear in Nonlinear Anal., Theory, Methods Appl.MathSciNetzbMATHGoogle Scholar
  61. Zhang, Q. (1995). Risk sensitive production planning of stochastic manufacturing systems: A singular perturbation approach, SIAM J. Control Optim.33, 498–527.MathSciNetzbMATHGoogle Scholar
  62. Zhang, Q. (1996). Finite state Markovian decision processes with weak and strong interactions, Stochastics Stochastics Rep.59, 283–304.MathSciNetzbMATHGoogle Scholar
  63. Zhang, Q. and G. Yin. (1996). A central limit theorem for singularly perturbed nonstationary finite state Markov chains, Ann. Appl. Probab.6, 650–670.MathSciNetzbMATHGoogle Scholar
  64. Zhang, Q. and G. Yin. (1997). Structural properties of Markov chains with weak and strong interactions, Stochastic Process Appl.70, 181–197.MathSciNetzbMATHGoogle Scholar
  65. Zhang, Q. and G. Yin. (1999). On nearly optimal controls of hybrid LQG problems, IEEE Trans. Automat. Control, 44, 2271–2282.MathSciNetzbMATHGoogle Scholar
  66. Zhang, Q., G. Yin, and E.K. Boukas. (1997). Controlled Markov chains with weak and strong interactions: Asymptotic optimality and application in manufacturing, J. Optim. Theory Appl.94, 169–194.MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer Science + Business Media, Inc. 2002

Authors and Affiliations

  • G. Yin
    • 1
  • Q. Zhang
    • 2
  • K. Yin
    • 3
  • H. Yang
    • 3
  1. 1.Department of MathematicsWayne State UniversityDetroitUSA
  2. 2.Department of MathematicsUniversity of GeorgiaAthensUSA
  3. 3.Department of Wood and Paper ScienceUniversity of MinnesotaSt. PaulUSA

Personalised recommendations