Abstract
Stochastic approximation algorithms have been one of the main focus areas of research on solution methods for stochastic optimization problems. The Robbins-Monro algorithm [17] is a basic stochastic approximation scheme that has been found to be applicable in a variety of settings that involve finding the roots of a function under noisy observations. We first review in this chapter the Robbins-Monro algorithm and its convergence. In cases where one is interested in optimizing the steady-state system performance, i.e., the objective is a long-run average cost function, multi-timescale variants of the Robbins-Monro algorithm have been found useful. We also review multi-timescale stochastic approximation in this chapter since many of the schemes presented in the later chapters shall involve such algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Benaim, M.: A dynamical systems approach to stochastic approximations. SIAM Journal on Control and Optimization 34(2), 437–472 (1996)
Benveniste, A., Métivier, M., Priouret, P.: Adaptive Algorithms and Stochastic Approximations. Springer, Berlin (1990)
Benveniste, A., Priouret, P., Métivier, M.: Adaptive algorithms and stochastic approximations. Springer-Verlag New York, Inc. (1990)
Borkar, V.S.: Stochastic approximation with two timescales. Systems and Control Letters 29, 291–294 (1997)
Borkar, V.S.: Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press and Hindustan Book Agency (Jointly Published), Cambridge and New Delhi (2008)
Borkar, V.S., Meyn, S.P.: The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM Journal of Control and Optimization 38(2), 447–469 (2000)
Chen, H.: Stochastic approximation and its applications, vol. 64. Kluwer Academic Pub. (2002)
Chen, H.F., Duncan, T.E., Pasik-Duncan, B.: A Kiefer-Wolfowitz algorithm with randomized differences. IEEE Trans. Auto. Cont. 44(3), 442–453 (1999)
Dai, J.G.: On positive Harris recurrence for multiclass queueing networks: A unified approach via fluid limit models. Annals of Applied Probability 5, 49–77 (1995)
Dai, J.G., Meyn, S.P.: Stability and convergence of moments for multiclass queueing networks via fluid limit models. IEEE Transactions on Automatic Control 40, 1889–1904 (1995)
Duflo, M.: Random iterative models, vol. 34. Springer (1997)
Kushner, H.J., Clark, D.S.: Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer, New York (1978)
Kushner, H.J., Yin, G.G.: Stochastic Approximation Algorithms and Applications. Springer, New York (1997)
Ljung, L.: Analysis of recursive stochastic algorithms. IEEE Transactions on Automatic Control AC-22, 551–575 (1977)
Polyak, B.T., Juditsky, A.B.: Acceleration of stochastic approximation by averaging. SIAM J. Control and Optim. 30(4), 838–855 (1992)
Renotte, C., Wouwer, A.V., Remy, M.: Neural modeling and control of a heat exchanger based on SPSA techniques. In: Proceedings of the American Control Conference, Chicago, IL, pp. 3299–3303 (2000)
Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Statist. 22, 400–407 (1951)
Spall, J.C., Cristion, J.A.: Nonlinear adaptive control using neural networks: estimation with a smoothed form of simultaneous perturbation gradient approximation. Statistica Sinica 4, 1–27 (1994)
Wouwer, A.V., Renotte, C., Remy, M.: Application of stochastic approximation techniques in neural modelling and control. International Journal of Systems Science 34, 851–863 (2003)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2013 Springer-Verlag London
About this chapter
Cite this chapter
Bhatnagar, S., Prasad, H., Prashanth, L. (2013). Stochastic Approximation Algorithms. In: Stochastic Recursive Algorithms for Optimization. Lecture Notes in Control and Information Sciences, vol 434. Springer, London. https://doi.org/10.1007/978-1-4471-4285-0_3
Download citation
DOI: https://doi.org/10.1007/978-1-4471-4285-0_3
Publisher Name: Springer, London
Print ISBN: 978-1-4471-4284-3
Online ISBN: 978-1-4471-4285-0
eBook Packages: EngineeringEngineering (R0)