Stochastic Gradient Estimation

Part of the International Series in Operations Research & Management Science book series (ISOR, volume 216)


This chapter reviews simulation-based methods for estimating gradients, which are central to gradient-based simulation optimization algorithms such as stochastic approximation and sample average approximation. We begin by describing approaches based on finite differences, including the simultaneous perturbation method. The remainder of the chapter then focuses on the direct gradient estimation techniques of perturbation analysis, the likelihood ratio/score function method, and the use of weak derivatives (also known as measure-valued differentiation). Various examples are provided to illustrate the different estimators—for a single random variable, a stochastic activity network, and a single-server queue. Recent work on quantile sensitivity estimation is summarized, and several newly proposed approaches for using stochastic gradients in simulation optimization are discussed.



This work was supported in part by the National Science Foundation under Grants CMMI-0856256 and ECCS-0901543, and by the Air Force Office of Scientific Research under Grant FA9550-10-10340.


  1. 1.
    B. Ankenman, B. L. Nelson, and J. Staum. Stochastic kriging for simulation metamodeling. Operations Research, 58(2):371–382, 2010.CrossRefGoogle Scholar
  2. 2.
    S. Asmussen and P. Glynn. Stochastic Simulation: Algorithms and Analysis. Springer, New York, 2007.Google Scholar
  3. 3.
    S. Bhatnagar, M. C. Fu, S. I. Marcus, and I. J. Wang. Two-timescale simultaneous perturbation stochastic approximation using deterministic perturbation sequences. ACM Transactions on Modeling and Computer Simulation, 13(2):180–209, 2003.CrossRefGoogle Scholar
  4. 4.
    P. Brémaud and F. J. Vázquez-Abad. On the pathwise computation of derivatives with respect to the rate of a point process: The phantom RPA method. Queueing Systems: Theory and Applications, 10:249–270, 1992.CrossRefGoogle Scholar
  5. 5.
    M. Broadie and P. Glasserman. Estimating security price derivatives using simulation. Management Science, 42(2):269–285, 1996.CrossRefGoogle Scholar
  6. 6.
    X.-R. Cao. Realization Probabilities: The Dynamics of Queuing Systems. Springer-Verlag, Boston, Massachusetts, 1994.CrossRefGoogle Scholar
  7. 7.
    C. G. Cassandras and S. G. Strickland. On-line sensitivity analysis of Markov chains. IEEE Transactions on Automatic Control, AC-34:76–86, 1989.CrossRefGoogle Scholar
  8. 8.
    M. Chau, M. C. Fu, and H. Qu. Multivariate stochastic approximation using a Secant-Tangents AveRaged (STAR) gradient estimator. Technical report, Working paper, University of Maryland, College Park, 2014.Google Scholar
  9. 9.
    M. Chau, H. Qu, and M. C. Fu. A new hybrid stochastic approximation algorithm. In Proceedings of the 12th International Workshop on Discrete Event Systems, 2014.Google Scholar
  10. 10.
    X. Chen. Enhancing Stochastic Kriging Metamodels for Computer Simulation. PhD thesis, Northwestern University, 2012.Google Scholar
  11. 11.
    X. Chen, B. Ankenman, and B. L. Nelson. Enhancing stochastic kriging metamodels with gradient estimators. Operations Research, 61(2):512–528, 2013.CrossRefGoogle Scholar
  12. 12.
    L. Y. Dai and Y. C. Ho. Structural infinitesimal perturbation analysis for derivative estimation in discrete event dynamic systems. IEEE Transactions on Automatic Control, 40:1154–1166, 1995.CrossRefGoogle Scholar
  13. 13.
    M. C. Fu. Sample path derivatives for (s, S) inventory systems. Operations Research, 42(2):351–364, 1994.Google Scholar
  14. 14.
    M. C. Fu. Gradient estimation. In S. G. Henderson and B. L. Nelson, editors, Handbooks in Operations Research and Management Science: Simulation, chapter 19, pages 575–616. Elsevier, 2006.Google Scholar
  15. 15.
    M. C. Fu. Sensitivity analysis in Monte Carlo simulation of stochastic activity networks. In F. B. Alt, M. C. Fu, and B. L. Golden, editors, Perspectives in Operations Research: Papers in Honor of Saul Gass’ 80th Birthday, pages 351–366. Springer, 2006.Google Scholar
  16. 16.
    M. C. Fu, L. J. Hong, and J. Q. Hu. Conditional Monte Carlo estimation of quantile sensitivities. Management Science, 55(12):2019–2027, 2009.CrossRefGoogle Scholar
  17. 17.
    M. C. Fu and J. Q. Hu. Sensitivity analysis for Monte Carlo simulation of option pricing. Probability in the Engineering and Informational Sciences, 9(3):417–446, 1995.CrossRefGoogle Scholar
  18. 18.
    M. C. Fu and J. Q. Hu. Conditional Monte Carlo: Gradient Estimation and Optimization Applications. Kluwer Academic Publishers, 1997.Google Scholar
  19. 19.
    M. C. Fu and H. Qu. Regression models augmented with direct stochastic gradient estimators. INFORMS Journal on Computing, 26(3):484–499, 2014.CrossRefGoogle Scholar
  20. 20.
    A. Gaivoronski, L. Y. Shi, and R. S. Sreenivas. Augmented infinitesimal perturbation analysis: An alternate explanation. Discrete Event Dynamic Systems: Theory and Applications, 2:121–138, 1992.CrossRefGoogle Scholar
  21. 21.
    P. Glasserman. Gradient Estimation via Perturbation Analysis. Kluwer Academic Publishers, Boston, Massachusetts, 1991.Google Scholar
  22. 22.
    P. Glasserman. Monte Carlo Methods in Financial Engineering. Springer, New York, 2004.Google Scholar
  23. 23.
    P. Glasserman and S. Tayur. Sensitivity analysis for base-stock levels in multi-echelon production-inventory systems. Management Science, 41:263–281, 1995.CrossRefGoogle Scholar
  24. 24.
    B. Heidergott, F. Vazquez-Abad, G. Pflug, and T. Farenhorst-Yuan. Gradient estimation for discrete-event systems by measure-valued differentiation. ACM Transactions on Modeling and Computer Simulation, 20(1):5.1–5.28, 2010.CrossRefGoogle Scholar
  25. 25.
    B. Heidergott, F. Vazquez-Abad, and W. Volk-Makarewicz. Sensitivity estimation for Gaussian systems. European Journal of Operational Research, 187(1):193–207, 2008.CrossRefGoogle Scholar
  26. 26.
    Y. C. Ho and X. R. Cao. Perturbation Analysis and Discrete Event Dynamic Systems. Kluwer Academic, 1991.Google Scholar
  27. 27.
    Y. C. Ho, M. A. Eyler, and T. T. Chien. A gradient technique for general buffer storage design in a serial production line. International Journal of Production Research, 17:557–580, 1979.CrossRefGoogle Scholar
  28. 28.
    Y. C. Ho and S. Li. Extensions of infinitesimal perturbation analysis. IEEE Transactions on Automatic Control, AC-33:827–838, 1988.Google Scholar
  29. 29.
    L. J. Hong. Estimating quantile sensitivities. Operations Research, 57(1):118–130, 2009.CrossRefGoogle Scholar
  30. 30.
    G. Jiang and M. C. Fu. On estimating quantile sensitivities via infinitesimal perturbation analysis. Operations Research, under review, 2014.Google Scholar
  31. 31.
    H. Kesten. Accelerated stochastic approximation. The Annals of Mathematical Statistics, 29(1):41–59, 1958.CrossRefGoogle Scholar
  32. 32.
    J. Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of a regression function. Annals of Mathematical Statistics, 23:462–266, 1952.CrossRefGoogle Scholar
  33. 33.
    H. J. Kushner and D. Clark. Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer-Verlag, 1978.Google Scholar
  34. 34.
    H. J. Kushner and G. G. Yin. Stochastic Approximation Algorithms and Applications. Springer-Verlag, 2nd edition, 2003.Google Scholar
  35. 35.
    A. M. Law and W. D. Kelton. Simulation Modeling and Analysis. McGraw-Hill, New York, 3rd edition, 2000.Google Scholar
  36. 36.
    G. C. Pflug. Sampling derivatives of probabilities. Computing, 42:315–328, 1989.CrossRefGoogle Scholar
  37. 37.
    H. Qu and M. C. Fu. Gradient extrapolated stochastic kriging. ACM Transactions on Modeling and Computer Simulation, 24(4): forthcoming, 2014.Google Scholar
  38. 38.
    H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22:400–407, 1951.CrossRefGoogle Scholar
  39. 39.
    R. Y. Rubinstein and A. Shapiro. Discrete Event Systems: Sensitivity Analysis and Stochastic Optimization by the Score Function Method. John Wiley & Sons, 1993.Google Scholar
  40. 40.
    L. Y. Shi. Discontinuous perturbation analysis of discrete event dynamic systems. IEEE Transactions on Automatic Control, 41:1676–1681, 1996.CrossRefGoogle Scholar
  41. 41.
    J. C. Spall. Multivariate stochastic approximation using simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37:332–341, 1992.CrossRefGoogle Scholar
  42. 42.
    R. Suri and M. A. Zazanis. Perturbation analysis gives strongly consistent sensitivity estimates for the MG∕1 queue. Management Science, 34:39–64, 1988.CrossRefGoogle Scholar
  43. 43.
    P. Vakili. Using a standard clock technique for efficient simulation. Operations Research Letters, 10(8):445–452, 1991.CrossRefGoogle Scholar
  44. 44.
    W. Volk-Makarewicz. Advances in Derivative Estimation: Ranked Data, Quantiles, and Options. PhD thesis, Vrije (Free) University of Amsterdam, 2014.Google Scholar
  45. 45.
    Y. Wang, M. C. Fu, and S. I. Marcus. A new stochastic derivative estimator for discontinuous payoff functions with application to financial derivatives. Operations Research, 60(2):447–460, 2012.CrossRefGoogle Scholar
  46. 46.
    X. Xiong, I. J. Wang, and M. C. Fu. An asymptotic analysis of stochastic approximation with deterministic perturbation sequences. In Proceedings of the 2002 Winter Simulation Conference, pages 285–291. IEEE, Piscataway, NJ, 2002.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.University of MarylandCollege ParkUSA

Personalised recommendations