Abstract
Consider the stochastic approximation algorithm (*) % MathType!MTEF!2!1!+- % feaagaart1ev2aaatCvAUfKttLearuqr1ngBPrgarmWu51MyVXgatC % vAUfeBSjuyZL2yd9gzLbvyNv2CaeHbd9wDYLwzYbItLDharyavP1wz % ZbItLDhis9wBH5garqqtubsr4rNCHbGeaGqiVu0Je9sqqrpepC0xbb % L8F4rqqrFfpeea0xe9Lq-Jc9vqaqpepm0xbba9pwe9Q8fs0-yqaqpe % pae9pg0FirpepeKkFr0xfr-xfr-xb9adbaqaaeGaciGaaiaabeqaam % aaeaqbaaGcbiGacWefca0fcqWGybawdaWgaaWcbaGaemOBa4Maey4k % aSIaeGymaedabeaakiabg2da9iabdIfaynaaBaaaleaacqWGUbGBae % qaaOGaey4kaSIaemyyae2aaSbaaSqaaiabd6gaUbqabaGccqWGNbWz % cqGGOaakcqWGybawdaWgaaWcbaGaemOBa4gabeaakiabcYcaSiabe6 % 7a4jabcMcaPiabc6caUaaa!5136! \[ X_{n + 1} = X_n + a_n g(X_n ,\xi ). \]
The problem of selecting the gain or step size sequences a n has been a serious handicap in applications. In a fundamental paper, Polyak and Juditsky [17] showed that (loosely speaking) if the coefficients a n go to zero slower than O(l/n), then the averaged sequence % MathType!MTEF!2!1!+- % feaagaart1ev2aaatCvAUfKttLearuqr1ngBPrgarmWu51MyVXgatC % vAUfeBSjuyZL2yd9gzLbvyNv2CaeHbd9wDYLwzYbItLDharyavP1wz % ZbItLDhis9wBH5garqqtubsr4rNCHbGeaGqiVu0Je9sqqrpepC0xbb % L8F4rqqrFfpeea0xe9Lq-Jc9vqaqpepm0xbba9pwe9Q8fs0-yqaqpe % pae9pg0FirpepeKkFr0xfr-xfr-xb9adbaqaaeGaciGaaiaabeqaam % aaeaqbaaGcbiGacWefca0fdaaeWaqaaiabdIfaybWcbaGaemyAaKMa % eyypa0JaeGymaedabaGaemOBa4ganiabggHiLdGccqWGPbqAcqGGVa % WlcqWGUbGBaaa!472B! \[ \sum\nolimits_{i = 1}^n X i/n \] converged to its limit at an optimum rate, for any coefficient sequence. This result implies that we should use “larger” than usual” gains, and let the off line averaging take care of the increased noise effects, with substantial overall improvement. Here we give a simpler proof under weaker conditions. Basically, it is shown that the averaging works whenever there is a “classical” rate of convergence theorem. I.e., results of this type are generic to stochastic approximation. Intuitive insight is provided by relating the behavior to that of a two time scale discrete algorithm. The value of the method has been supported by simulations. Since the averaged estimate is “off line,” it is not the actual value used in the SA iteration (*) itself. We show how the averaged value can be partially fed back into the actual operating algorithm for improved performance. Numerical data are presented to support the theoretical conclusions. An error in the tightness part of the proof in [14] is corrected.
Supported by AFOSR Contract F 49620-92-0081 and NSF grant ECS-8913351.
Supported by AFOSR Contract F 49620-92-0081.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
A. Benveniste, M. Metivier, AND P. Prioret, Adaptive Algorithms and Stochastic Approximation, Springer-Verlag, New York, Berlin 1990.
P. Billingsley, Convergence of Probability Measures, John Wiley, New York 1968.
G. Blankenship AND G.C. Papanicolaou, Stability and control of systems with wide band noise disturbances, SIAM J. Appl. Math., 34 (1978), pp. 437–476.
B. Deylon AND A. Juditsky, Stochastic optimization with averaging of trajectories, Stochastics, 39 (1992), pp. 107–118.
S.N. Ethier AND T.G. Kurtz, Markov Processes: Characterization and Convergence, Wiley, New York 1986.
H.J. Kushner AND Hai Huang, Averaging methods for the asymptotic analysis of learning and adaptive systems, SIAM J. on Control and Optimization, 19 (1981), pp. 635–650.
H.J. Kushner, Stochastic approximation with discontinuous dynamics and state dependent noise, J. Math. Analysis and Applications, 82 (1981), pp. 527–542.
H.J. Kushner, Approximation and Weak Convergence Methods for Random Processes with Applications to Stochastic System Theory, MIT Press, Cambridge, MA 1984.
H.J. Kushner, Weak Convergence Methods and Singularly Perturbed Stochastic Control and Filtering Problems, Volume 3 of Systems and Control, Birkhauser, Boston 1990.
H.J. Kushner AND D.S. Clark, Stochastic Approximation for Constrained and Unconstrained Syetems, Springer-Verlag, Berlin and New York 1978.
H.J. Kushner AND A. Shwartz, An invariant measure approach to the convergence of stochastic approximations with state dependent noise, SIAM J. on Optimization and Control, 22 (1984), pp. 13–27.
H.J. Kushner AND J. Yang, A montecario method for the sensitivity analysis and parametric optimization of nonlinear stochastic systems, SIAM J. on Control and Optimization, 29 (1992), pp. 1216–1249.
H.J. Kushner AND J. Yang, Stochastic approximation with averaging and feedback: rapidly convergent “on line” algorithms, and applications to adaptive systems, (Technical report) Brown University, Lefschetz Center for Dynamical Systems 1992.
H.J. Kushner AND J. Yang, Stochastic approximation with averaging: optimal asymptotic rates of convergence for general processes, SIAM J. on Control and Optimization 31 (1993), pp. 1045–1062.
H.J. Kushner AND G. Yin, Asymptotic properties of distributed and communicating stochastic approximation algorithms, SIAM J. on Control and Optimization, 25 (1987), pp. 1266–1290.
B.T. Polyak, New stochastic approximation type procedures, Autom. i Telemekh, 7 (1990), pp. 98–107.
B.T. Polyak AND A.B. Juditsky, Acceleration of stochastic approximation by averaging, SIAM J. on Control and Optimization, 30 (1992), pp. 838–855.
D. Ruppbrt, Efficient estimators from a slowly convergent robbins-munro process, Technical Report 787, School of Operations Research and Industrial Engineering, Cornell University 1988.
G. Yin, On extensions of polyak’s averaging approach to stochastic approximation, Stochastics, 36 (1992), pp. 245–264.
G. Yin, Stochastic approximation via averaging: Polyak’s approach revisited, in Lecture Notes in Economics and Mathematical Systems 374 (G. Pflug AND U. Dieter, editors), Springer-Verlag, Berlin 1992, pp. 119–134.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1995 Springer Science+Business Media New York
About this paper
Cite this paper
Kushner, H.J., Yang, J. (1995). Stochastic Approximation with Averaging and Feedback: Faster Convergence. In: Åström, K.J., Goodwin, G.C., Kumar, P.R. (eds) Adaptive Control, Filtering, and Signal Processing. The IMA Volumes in Mathematics and its Applications, vol 74. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-8568-2_9
Download citation
DOI: https://doi.org/10.1007/978-1-4419-8568-2_9
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4612-6439-2
Online ISBN: 978-1-4419-8568-2
eBook Packages: Springer Book Archive