, Volume 42, Issue 1, pp 153–171 | Cite as

Some results about averaging in stochastic approximation

  • Alain Le Breton
  • Alexander Novikov


The paper presents some results concerning the averaging approach in a “general” linear regression model in one dimension under suitable conditions about the martingale structure of errors. At first asymptotics of the primary and averaged estimators are discussed. Then it is shown that variances of estimators can be consistently estimated by appropriate integrated squared deviations functionals. Finally applications to the construction of confidence regions are considered.


Linear Regression Regression Model Stochastic Process Probability Theory Economic Theory 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ruppert D (1988) Efficient estimations from a slowly convergent Robbins-Monro process. Technical Report 781, Cornell UnivGoogle Scholar
  2. Polyak BT (1990) New method of stochastic approximation type. Automate Remote Control 51:937–946Google Scholar
  3. Yin G (1991) On extension of Polyak's averaging approach to stochastic approximation. Stochastics and Stochastic Reports 36:245–264Google Scholar
  4. Polyak BT, Juditsky AB (1992) Acceleration of stochastic approximation by averaging. SIAM J Cont & Optim 30 (4):838–855Google Scholar
  5. Delyon B, Juditsky AB (1992) Stochastic optimization with averaging of trajectories. Stochastics and Stochastic Reports 39:107–118Google Scholar
  6. Juditsky AB (1992) On the recursive algorithm of estimation of autoregressive parameters. Automate Remote Control 7:118–127 (in russian)Google Scholar
  7. Chen HF (1992) Continuous-time stochastic approximation: Convergence and asymptotic efficiency. (preprint)Google Scholar
  8. Chen HF (1993) Asymptotically efficient stochastic approximation. Stochastics and Stochastic Reports 45:1–16Google Scholar
  9. Le Breton A (1993) About the averaging approach in Gaussian schemes for stochastic approximation. Mathematical Methods of Statistics 2 (4):295–315Google Scholar
  10. Le Breton A (1994) About Gaussian schemes in stochastic approximation. Stochastic Processes and their Applications 50:101–115Google Scholar
  11. Le Breton A, Novikov AA (1993a) Averaging for estimating covariances in stochastic approximation. To appear in Mathematical Methods of StatisticsGoogle Scholar
  12. Le Breton A, Novikov AA (1993b) On stochastic approximation procedures with averaging. To appear in Proceedings of the Steklov Mathematical Institute.Google Scholar
  13. Lai TL, Robbins H (1979) Adaptive design and stochastic approximation. Ann Stat 7:1196–1221Google Scholar
  14. Lai TL, Robbins H (1981) Consistency and asymptotic efficiency of slope estimates in stochastic approximation schemes. Z Waht verw Gebiete 56:329–360Google Scholar
  15. Lipcer RSH, Shiryaev AN (1989) Theory of martingales. Kluwer Academic Publishers, DordrechtGoogle Scholar
  16. Chow YS, Robbins H (1965) On the asymptotic theory of fixed-width confidence intervals for the mean. Ann Math Stat 36:457–462Google Scholar
  17. Ghosh BK, Sen PK (1991) Handbook of sequential analysis. Marcel Dekker Inc, New YorkGoogle Scholar
  18. Sielken R (1973) Stopping times for stochastic approximation procedures. Z Wahr verw Gebiete 26:67–75Google Scholar
  19. Stroup DF, Braun HI (1982) On a new stopping time for stochastic approximation. Z Wahr verw Gebiete 60:535–554Google Scholar
  20. Stroup DF, Braun HI (1984) Correction to “On a new stopping time for stochastic approximation”. Z Wahr verw Gebiete 67:237Google Scholar
  21. Wei CZ (1985) Asymptotic properties of least squares estimates in stochastic regression models. Ann Stat 13:1498–1508Google Scholar
  22. Pflug GCH (1985) Stepsize rules, stopping times and their implementation in stochastic quasigradient algorithms. In: Numerical Techniques for stochastic optimization Wets R Ermoliev Yu (Eds.). Springer Berlin-New YorkGoogle Scholar
  23. Pflug GCH (1990) Non-asymptotic confidence bounds for stochastic approximation algorithms with constant step size. Monatshefte fur Math 110:297–314Google Scholar
  24. Yin G (1988) A stopped stochastic approximation algorithm. Systems & Control Letters 11:107–115Google Scholar
  25. Yin G (1990) A stopping rule for the Robbins-Monro method. J of Optim Theor and Appl 67: 151–173Google Scholar

Copyright information

© Physica-Verlag 1995

Authors and Affiliations

  • Alain Le Breton
    • 1
  • Alexander Novikov
    • 2
  1. 1.Laboratoire de Modélisation et Calcul/IMAGUniversité Joseph Fourier Grenoble IGrenoble Cedex 09France
  2. 2.Steklov Mathematical InstituteMoscow, GSP1Russia

Personalised recommendations