Loss Functions and Stochastic Approximation

  • Jack Sklansky
  • Gustav N. Wassel


Gradient descent as a technique for finding the minimum of a loss function J(v) was introduced in Section 2.10. Recall that the technique consists of finding the gradient ∇ J(v) and then adjusting the parameter vector v so that the change in v is in the direction of the negative of the gradient.


Feature Vector Loss Function Gradient Descent Training Procedure Stochastic Approximation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    S. S. Yau and J. M. Schumpert, Design of pattern classifiers with the updating property using stochastic approximation techniques. IEEE Transactions on Computers, C-17 (September): 861–872 (1968).CrossRefGoogle Scholar
  2. 2.
    S. S. Viglione, Applications of pattern recognition technology. In: Adaptive Learning and Pattern Recognition Systems, J. M. Mendel and K. S. Fu (eds.), Academic Press, New York, 1970, pp. 115–162.CrossRefGoogle Scholar
  3. 3.
    B. B. Brick, Wiener’s nonlinear expansion procedure applied to cybernetic problems. IEEE Transactions on Systems Science and Cybernetics, SSC-1 (1): 67–74 (1965).CrossRefGoogle Scholar
  4. 4.
    R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. John Wiley & Sons, New York, 1973, Chapter 5.MATHGoogle Scholar
  5. 5.
    A. Papoulis, Probability, Random Variables and Stochastic Processes. McGraw- Hill, New York, 1965.MATHGoogle Scholar
  6. 6.
    H. Robbins and S. Monro, A stochastic approximation method. The Annals of Mathematical Statistics, 22: 400–407 (1951).MathSciNetMATHCrossRefGoogle Scholar
  7. 7.
    J. Kiefer and J. Wolfowitz, Stochastic estimation of the maximum of a regression function. The Annals of Mathematical Statistics, 23: 462–466 (1952).MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    J. R. Blum, Multidimensional stochastic approximation methods. The Annals of Mathematical Statistics, 25 (4): 737–744 (1954).MATHCrossRefGoogle Scholar
  9. 9.
    A. Dvoretzky, On stochastic approximation. Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, December, 1954 and June and July, 1955, pp. 39–55.Google Scholar
  10. 10.
    S. S. L. Chang, Synthesis of Optical Control Systems. McGraw-Hill, New York, 1961, pp. 289–93.Google Scholar

Copyright information

© Springer-Verlag New York Inc 1981

Authors and Affiliations

  • Jack Sklansky
    • 1
  • Gustav N. Wassel
    • 2
  1. 1.Department of Electrical EngineeringUniversity of California at IrvineIrvineUSA
  2. 2.Deparment of Electronic and Electrical EngineeringCalifornia Polytechnic State UniversitySan Luis ObispoUSA

Personalised recommendations