Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Perturbation Analysis

  • Chapter
Stochastic Learning and Optimization

Abstract

Perturbation analysis (PA) is the core of the gradient-based (or policy gradient) learning and optimization approach. The basic principle of PA is that the derivative of a system’s performance with respect to a parameter of the system can be decomposed into the sum of many small building blocks, each of which measures the effect of a single perturbation on the system’s performance, and this effect can be estimated on a sample path of the system.

To climb steep hills requires slow pace at first.

William Shakespeare, English poet and playwright (1564 – 1818)

Don’t buy the house; buy the neighborhood.

Russian Proverb

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. X. R. Cao, Realization Probabilities: The Dynamics of Queueing Systems, Springer-Verlag, New York, 1994.

    Book  MATH  Google Scholar 

  2. C. G. Cassandras and S. Lafortune, Introduction to Discrete Event Systems, Kluwer Academic Publishers, Boston, 1999.

    MATH  Google Scholar 

  3. M. C. Fu and J. Q. Hu, Conditional Monte Carlo: Gradient Estimation and Optimization Applications, Kluwer Academic Publishers, Boston, 1997.

    MATH  Google Scholar 

  4. Y. C. Ho and X. R. Cao, Perturbation Analysis of Discrete-Event Dynamic Systems, Kluwer Academic Publisher, Boston, 1991.

    MATH  Google Scholar 

  5. X. R. Cao and H. F. Chen, “Perturbation Realization, Potentials and Sensitivity Analysis of Markov Processes,” IEEE Transactions on Automatic Control, Vol. 42, 1382-1393, 1997.

    Article  MATH  MathSciNet  Google Scholar 

  6. X. R. Cao, X. M. Yuan, and L. Qiu, “A Single Sample Path-Based Performance Sensitivity Formula for Markov Chains,” IEEE Transactions on Automatic Control, Vol. 41, 1814-1817, 1996.

    Article  MATH  MathSciNet  Google Scholar 

  7. E. Çinlar, Introduction to Stochastic Processes, Prentice Hall, Englewood Cliffs, New Jersey, 1975.

    Google Scholar 

  8. C. D. Meyer, “The Role of the Group Generalized Inverse in the Theory of Finite Markov Chains,” SIAM Review, Vol. 17, 443-464, 1975.

    Article  MATH  MathSciNet  Google Scholar 

  9. G. Hooghiemstra, M. Keane, and S. Van De Ree, “Power Series for Stationary Distribution of Coupled Processor Models,” SIAM Journal of Applied Mathematics, Vol. 48, 1159-1166, 1988.

    Article  MATH  Google Scholar 

  10. Y. Zhu and H. Li, “The MacLaurin Expansion for a G/G/1 Queue with Markov-Modulated Arrivals and Services,” Queueing Systems: Theory and Applications, Vol. 14, 125-134, 1993.

    Article  MATH  MathSciNet  Google Scholar 

  11. W. B. Gong and J. Q. Hu, “The Maclaurin Series for the GI/G/1 Queue,” Journal of Applied Probability, Vol. 29, 176-184, 1992.

    Article  MATH  MathSciNet  Google Scholar 

  12. J. P. C. Blanc, “A Numerical Approach to Cyclic-Server Queueing Models,” Queueing Systems: Theory and Applications, Vol. 6, 173-188, 1990.

    Article  MATH  MathSciNet  Google Scholar 

  13. J. Q. Hu, S. Nananukul, and W. B. Gong, “A New Approach to (s, S) Inventory Systems,” Journal of Applied Probability, Vol. 30, 898-912, 1993.

    Article  MATH  MathSciNet  Google Scholar 

  14. T. Kailath, Linear Systems, Prentice Hall, Englewood Cliffs, New Jersey, 1980.

    MATH  Google Scholar 

  15. P. Lancaster and M. Tismenetsky, The Theory of Matrices with Applications, Second Edition, Academic Press, Orlando, 1985.

    MATH  Google Scholar 

  16. X. R. Cao, “Semi-Markov Decision Problems and Performance Sensitivity Analysis,” IEEE Transactions on Automatic Control, Vol. 48, 758-769, 2003.

    Article  Google Scholar 

  17. L. Kleinrock, Queueing Systems, Volume I: Theory, John Wiley & Sons, New York, 1975.

    Google Scholar 

  18. H. C. Tijms, Stochastic Models: An Algorithmic Approach, John Wiley & Sons, New York, 1994.

    MATH  Google Scholar 

  19. X. R. Cao, “Realization Probability in Closed Jackson Queueing Networks and Its Application,” Advances in Applied Probability, Vol. 19, 708-738, 1987.

    Article  MATH  MathSciNet  Google Scholar 

  20. X. R. Cao, “Realization Probability and Throughput Sensitivity in a Closed Jackson Network,” Journal of Applied Probability, Vol. 26, 615-624, 1989.

    Article  MATH  MathSciNet  Google Scholar 

  21. X. R. Cao, “Realization Factors and Sensitivity Analysis of Queueing Networks with State-Dependent Service Rates,” Advances in Applied Probability, Vol. 22, 178-210, 1990.

    Article  MATH  MathSciNet  Google Scholar 

  22. P. Glasserman, “The Limiting Value of Derivative Estimates Based on Perturbation Analysis,” Communications in Statistics: Stochastic Models, Vol. 6, 229-257, 1990.

    Article  MATH  MathSciNet  Google Scholar 

  23. Y. C. Ho and X. R. Cao, “Perturbation Analysis and Optimization of Queueing Networks,” Journal of Optimization Theory and Applications, Vol. 40, 559-582, 1983.

    Article  MATH  MathSciNet  Google Scholar 

  24. X. R. Cao, “Convergence of Parameter Sensitivity Estimates in a Stochastic Experiment,” IEEE Transactions on Automatic Control, Vol. 30, 845-853, 1985.

    Article  MATH  Google Scholar 

  25. X. R. Cao, “A Sample Performance Function of Jackson Queueing Networks,” Operations Research, Vol. 36, 128-136, 1988.

    Article  MATH  MathSciNet  Google Scholar 

  26. M. C. Fu and J. Q. Hu, “Smoothed Perturbation Analysis Derivative Estimation for Markov Chains,” Operations Research Letters, Vol. 15, 241-251, 1994.

    Article  MATH  MathSciNet  Google Scholar 

  27. P. Glasserman and W. B. Gong, “Smoothed Perturbation Analysis for A Class of Discrete Event System,” IEEE Transactions on Automatic Control, Vol. 35, 1218-1230, 1990.

    Article  MATH  MathSciNet  Google Scholar 

  28. W. B. Gong and Y. C. Ho, “Smoothed Perturbation Analysis for Discrete Event Dynamic Systems,” IEEE Transactions on Automatic Control, Vol. 32, 858-866, 1987.

    Article  MATH  MathSciNet  Google Scholar 

  29. Y. C. Ho, X. R. Cao, and C. Cassandras, “Infinitesimal and Finite Perturbation Analysis for Queueing Networks,” Automatica, Vol. 19, 439-445, 1983.

    Article  MATH  Google Scholar 

  30. P. Bremaud, “Maximal Coupling and Rare Perturbation Sensitivity Analysis,” Queueing Systems: Theory and Applications, Vol. 11, 307-333, 1992.

    Article  MATH  MathSciNet  Google Scholar 

  31. P. Bremaud and W. B. Gong, “Derivatives of Likelihood Ratios and Smoothed Perturbation Analysis for Routing Problem,” ACM Transactions on Modeling and Computer Simulation, Vol. 3, 134-161, 1993.

    Article  MATH  Google Scholar 

  32. P. Bremaud and L. Massoulie, “Maximal Coupling and Rare Perturbation Analysis with a Random Horizon,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 5, 319-342, 1995.

    Article  MATH  Google Scholar 

  33. P. Bremaud and F. J. Vazquez-Abad, “On the Pathwise Computation of Derivatives with Respect to the Rate of A Point Process: The Phantom RPA Method,” Queueing Systems: Theory and Applications, Vol. 10, 249-269, 1992.

    Article  MATH  MathSciNet  Google Scholar 

  34. C. G. Cassandras and S. G. Strickland, “On-Line Sensitivity Analysis of Markov Chains,” IEEE Transactions on Automatic Control, Vol. 34, 76-86, 1989.

    Article  MATH  MathSciNet  Google Scholar 

  35. L. Dai, and Y. C. Ho, “Structural Infinitesimal Perturbation Analysis (SIPA) for Derivative Estimation of Discrete Event Dynamic Systems,” IEEE Transactions on Automatic Control, Vol. 40, 1154-1166, 1995.

    Article  MATH  MathSciNet  Google Scholar 

  36. M. Freimer and L. Schruben, “Graphical Representation of IPA Estimation,” Proceedings of the 2001 Winter Simulation Conference, Arlington, Virginia, U.S.A, Vol. 1, 422-427, December 2001.

    Google Scholar 

  37. M. C. Fu and J. Q. Hu, “Efficient Design and Sensitivity Analysis of Control Charts Using Monte Carlo Simulation,” Management Science, Vol. 45, 395-413, 1999.

    Article  Google Scholar 

  38. A. A. Gaivoronski, L. Y. Shi, and R. S. Sreenivas, “Augmented Infinitesimal Perturbation Analysis: An Alternate Explanation,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 2, 121-138, 1992.

    Article  MATH  Google Scholar 

  39. B. Heidergott, “Infinitesimal Perturbation Analysis for Queueing Networks with General Service Time Distributions,” Queueing Systems: Theory and Applications, Vol. 31, 43-58, 1999.

    Article  MATH  MathSciNet  Google Scholar 

  40. Y. C. Ho and S. Li, “Extensions of Infinitesimal Perturbation Analysis,” IEEE Transactions on Automatic Control, Vol. 33, 427-438, 1988.

    Article  MATH  MathSciNet  Google Scholar 

  41. J. Q. Hu, “Convexity of Sample Path Performance and Strong Consistency of Infinitesimal Perturbation Analysis Estimates,” IEEE Transactions on Automatic Control, Vol. 37, 258-262, 1992.

    Article  Google Scholar 

  42. Q. L. Li, and L. M. Liu, “An Algorithmic Approach on Sensitivity Analysis of Perturbed QBD Processes,” Queueing Systems, Vol. 48, 365-397, 2004.

    Article  MATH  MathSciNet  Google Scholar 

  43. E. L. Plambeck, B. R. Fu, S. M. Robinson, and R. Suri, “Sample-Path Optimization of Convex Stochastic Performance Functions,” Mathematical Programming, Vol. 75, 137-176, 1996.

    MathSciNet  Google Scholar 

  44. R. Suri, “Infinitesimal Perturbation Analysis for General Discrete Event Systems,” Journal of the ACM, Vol. 34, 686-717, 1987.

    Article  MathSciNet  Google Scholar 

  45. Q. Y. Tang, P. L’Ecuyer, and H. F. Chen, “Central Limit Theorems for Stochastic Optimization Algorithms Using Infinitesimal Perturbation Analysis,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 10, 5-32, 2000.

    Article  MATH  MathSciNet  Google Scholar 

  46. S. Uryasev, “Analytic Perturbation Analysis for DEDS with Discontinuous Sample Path Functions,” Communications in Statistics: Stochastic Models, Vol. 13, 457-490, 1997.

    Article  MATH  MathSciNet  Google Scholar 

  47. F. J. Vazquez-Abad and J. H. Kushner, “Estimation of the Derivative of a Stationary Measure with Respect to a Control Parameter,” Journal of Applied Probability, Vol. 29, 343-352, 1992.

    Article  MATH  MathSciNet  Google Scholar 

  48. Y. Wardi, M. W. McKinnon, and R. Schuckle, “On Perturbation Analysis of Queueing Networks with Finitely Supported Service Time Distributions,” IEEE Transactions on Automatic Control, Vol. 36, 863-867, 1991.

    Article  MathSciNet  Google Scholar 

  49. P. Glasserman, Gradient Estimation Via Perturbation Analysis, Kluwer Academic Publishers, Boston, 1991.

    MATH  Google Scholar 

  50. C. G. Cassandras, G. Sun, C. G. Panayiotou, and Y. Wardi, “Perturbation Analysis and Control of Two-Class Stochastic Fluid Models for Communication Networks,” IEEE Transactions on Automatic Control, Vol. 48, 770-782, 2003.

    Article  MathSciNet  Google Scholar 

  51. C. G. Cassandras, Y. Wardi, B. Melamed, G. Sun, and C. G. Panayiotou, “Perturbation Analysis for Online Control and Optimization of Stochastic Fluid Models,” IEEE Transactions on Automatic Control, Vol. 47, 1234-1248, 2002.

    Article  MathSciNet  Google Scholar 

  52. Y. Liu and W. B. Gong, “Perturbation Analysis for Stochastic Fluid Queueing Systems,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 12, 391-416, 2002.

    Article  MATH  MathSciNet  Google Scholar 

  53. C. Panayiotou and C. G. Cassandras, “Infinitesimal Perturbation Analysis and Optimization for Make-to-Stock Manufacturing Systems Based on Stochastic Fluid Models,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 16, 109-142, 2006.

    Article  MATH  MathSciNet  Google Scholar 

  54. C. Panayiotou, C. G. Cassandras, G. Sun, and Y. Wardi, “Control of Communication Networks Using Infinitesimal Perturbation Analysis of Stochastic Fluid Models,” Advances in Communication Control Networks, Lecture Notes in Control and Information Sciences, Vol. 308, 1-26, 2004.

    MathSciNet  Google Scholar 

  55. G. Sun, C. G. Cassandras, and C. G. Panayiotou, “Perturbation Analysis of Multiclass Stochastic Fluid Models,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 14, 267-307, 2004.

    Article  MATH  MathSciNet  Google Scholar 

  56. Y. Wardi, B. Melamed, C. G. Cassandras, and C. G. Panayiotou, “Online IPA Gradient Estimators in Stochastic Continuous Fluid Models,” Journal of Optimization Theory and Applications, Vol. 115, 369-405, 2002.

    Article  MATH  MathSciNet  Google Scholar 

  57. H. Yu and C. G. Cassandras, “Perturbation Analysis of Feedback-Controlled Stochastic Flow Systems,” IEEE Transactions on Automatic Control, Vol. 49, 1317-1332, 2004.

    Article  MathSciNet  Google Scholar 

  58. H. Yu and C. G. Cassandras, “Perturbation Analysis of Communication Networks with Feedback Control Using Stochastic Hybrid Models,” Nonlinear Analysis - Theory Methods and Applications, Vol. 65, 1251-1280, 2006.

    Article  MATH  MathSciNet  Google Scholar 

  59. P. W. Glynn, “Regenerative Structure of Markov Chains Simulated Via Common Random Numbers,” Operations Research Letters, Vol. 4, 49-53, 1985.

    Article  MATH  MathSciNet  Google Scholar 

  60. P. W. Glynn, “Likelihood Ratio Gradient Estimation: An Overview,” Proceedings of the 1987 Winter Simulation Conference, Atlanta, Georgia, U.S.A, 366-375, December 1987.

    Google Scholar 

  61. P. W. Glynn, “Optimization of Stochastic Systems Via Simulation,” Proceedings of the 1989 Winter Simulation Conference, Washington, U.S.A, 90-105, December 1989.

    Google Scholar 

  62. P. W. Glynn and P. L’Ecuyer, “Likelihood Ratio Gradient Estimation for Stochastic Recursions,” Advances in Applied Probability, Vol. 27, 1019-1053, 1995.

    Article  MATH  MathSciNet  Google Scholar 

  63. B. Heidergott and X. R. Cao, “A Note on the Relation Between Weak Derivatives and Perturbation Realization,” IEEE Transactions on Automatic Control, Vol. 47, 1112-1115, 2002.

    MathSciNet  Google Scholar 

  64. P. L’Ecuyer, “A Unified View of the IPA, SF, and LR Gradient Estimation Techniques,” Management Science, Vol. 36, 1364-1383, 1990.

    Article  MATH  Google Scholar 

  65. P. L’Ecuyer, “Convergence Rate for Steady-State Derivative Estimators,” Annals of Operations Research, Vol. 39, 121-136, 1992.

    Article  MATH  MathSciNet  Google Scholar 

  66. P. L’Ecuyer, “On the Interchange of Derivative and Expectation for Likelihood Ratio Derivative Estimators,” Management Science, Vol. 41, 738-748, 1995.

    Article  MATH  Google Scholar 

  67. P. L’Ecuyer and G. Perron, “On the Convergence Rates of IPA and FDC Derivative Estimators,” Operations Research, Vol. 42, 643-656, 1994.

    Article  MATH  MathSciNet  Google Scholar 

  68. M. K. Nakayama and P. Shahabuddin, “Likelihood Ratio Derivative Estimation for Finite-Time Performance Measures in Generalized Semi-Markov Processes,” Management Science, Vol. 44, 1426-1441, 1998.

    Article  MATH  Google Scholar 

  69. M. I. Reiman and A. Weiss, “Sensitivity Analysis for Simulations Via Likelihood Ratios,” Operations Research, Vol. 37, 830-844, 1989.

    Article  MATH  MathSciNet  Google Scholar 

  70. R. V. Rubinstein, Monte Carlo Optimization, Simulation, and Sensitivity Analysis of Queueing Networks, John Wiley & Sons, New York, 1986.

    Google Scholar 

  71. R. V. Rubinstein and A. Shapiro, Sensitivity Analysis and Stochastic Optimization by the Score Function Method, John Wiley & Sons, New York, 1993.

    MATH  Google Scholar 

  72. X. R. Cao, “Sensitivity Estimates Based on One Realization of a Stochastic System,” Journal of Statistical Computation and Simulation, Vol. 27, 211-232, 1987.

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong

    Xi-Ren Cao PhD (Professor)

Authors
  1. Xi-Ren Cao PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xi-Ren Cao PhD .

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Cao, XR. (2007). Perturbation Analysis. In: Stochastic Learning and Optimization. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-69082-7_2

Download citation

  • DOI: https://doi.org/10.1007/978-0-387-69082-7_2

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-0-387-36787-3

  • Online ISBN: 978-0-387-69082-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation