Skip to main content
SpringerLink
Log in

Parameter-Free On-line Deep Learning

  • Conference paper
  • First Online:
Automation 2017 (ICA 2017)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 550))

Included in the following conference series:

  • 2304 Accesses

Abstract

In this paper the classic momentum algorithm for stochastic optimization is considered. A method is introduced that adjusts coefficients for this algorithm during its operation. The method does not depend on any preliminary knowledge of the optimization problem. In the experimental study, the method is applied to on-line learning in deep auto-encoders, and outperforms manually tuned coefficients. The method makes on-line learning a fully parameter-free process and broadens the area of potential application of this technology.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Actually [17] experimented with \(\lambda _{\max } \in \{0.9, 0.99, 0.995, 0.999\}\). In all cases but for Faces the best choice was \(\lambda _{\max }=0.99\). For Faces it was the second best choice, and the best was \(\lambda _{\max }=0.995\).

References

  1. Behera, L., Kumar, S., Patnaik, A.: On adaptive learning rate that guarantees convergence in feedforward networks. IEEE Trans. Neural Networks 17(5), 1116–1125 (2006)

    Article  Google Scholar 

  2. Bhaya, A., Kaszkurewicz, E.: Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method. Neural Networks 17, 65–71 (2004)

    Article  MATH  Google Scholar 

  3. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  4. Jacobs, R.A.: Increased rates of convergence through learning rate adaptation. Neural Networks 1(4), 295–308 (1988)

    Article  Google Scholar 

  5. Kathirvalavakumar, T., Subavathi, S.J.: Neighborhood based modified backpropagation algorithm using adaptive learning parameters for training feedforward neural networks. Neurocomputing 72, 3915–3921 (2009)

    Article  Google Scholar 

  6. Kushner, H.J., Yin, G.: Stochastic Approximation Algorithms and Applications. Springer-Verlag, New York (1997)

    Book  MATH  Google Scholar 

  7. Martens, J., Grosse, R.: Optimizing neural networks with kronecker-factored approximate curvature. In: Proceedings of the 32th International Conference on Machine Learning, pp. 573–582 (2015)

    Google Scholar 

  8. Nesterov, Y.: A method of solving a convex programming problem with convergence rate o(1/sqr(k)). Sov. Math. Dokl. 27, 372–376 (1983)

    MATH  Google Scholar 

  9. Pearlmutter, B.A.: Fast exact multiplication by the hessian. Neural Comput. 6(1), 147–160 (1994)

    Article  Google Scholar 

  10. Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4, 1–17 (1964)

    Article  Google Scholar 

  11. Qian, N.: On the momentum term in gradient descent learning algorithms. Neural Networks 12, 145–151 (1999)

    Article  Google Scholar 

  12. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22, 400–407 (1951)

    Article  MathSciNet  MATH  Google Scholar 

  13. Schaul, T., Zhang, S., LeCun, Y.: No More Pesky Learning Rates. In: International Conference on Machine Learning (ICML) (2013)

    Google Scholar 

  14. Schraudolph, N.N., Giannakopoulos, X.: Online independent component analysis with local learning rate adaptation. Adv. NIPS 12, 789–795 (2000)

    Google Scholar 

  15. Silva, F.M., Almeida, L.B.: Acceleration techniques for the backpropagation algorithm. In: Almeida, L.B., Wellekens, C.J. (eds.) EURASIP 1990. LNCS, vol. 412, pp. 110–119. Springer, Heidelberg (1990). doi:10.1007/3-540-52255-7_32

    Chapter  Google Scholar 

  16. Sutskever, I.: Training recurrent neural networks. Ph.D. thesis, University of Toronto (2012)

    Google Scholar 

  17. Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: Proceedings of the 30th International Conference on Machine Learning (ICML 2013). vol. 28, JMLR Workshop and Conference Proceedings, pp. 1139–1147 (2013)

    Google Scholar 

  18. Yu, X., Wang, B., Batbayar, B., Wang, L., Man, Z.: An improved training algorithm for feedforward neural network learning based on terminal attractors. J. Global Optim. 1–14, September 2010

    Google Scholar 

  19. Zak, M.: Terminal attractors in neural networks. Neural Networks 2, 259–274 (1989)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Warsaw University of Technology, Warsaw, Poland

    Paweł Wawrzyński

Authors
  1. Paweł Wawrzyński
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paweł Wawrzyński .

Editor information

Editors and Affiliations

  1. Industrial Research Institute for Automation and Measurements PIAP, Warsaw, Poland

    Roman Szewczyk

  2. Industrial Research Institute for Automation and Measurements PIAP, Warsaw, Poland

    Cezary Zieliński

  3. Industrial Research Institute for Automation and Measurements PIAP, Warsaw, Poland

    Małgorzata Kaliczyńska

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Wawrzyński, P. (2017). Parameter-Free On-line Deep Learning. In: Szewczyk, R., Zieliński, C., Kaliczyńska, M. (eds) Automation 2017. ICA 2017. Advances in Intelligent Systems and Computing, vol 550. Springer, Cham. https://doi.org/10.1007/978-3-319-54042-9_54

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-54042-9_54

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-54041-2

  • Online ISBN: 978-3-319-54042-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation