Skip to main content
SpringerLink
Log in

On the Computational Complexity of Deep Learning Algorithms

  • Conference paper
  • First Online:
Proceedings of International Scientific Conference on Telecommunications, Computing and Control

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 220))

Abstract

The paper analyzes current research and the state of the industry to assess the complexity of machine learning algorithms. The tasks of deep learning are associated with an extremely high degree of computational complexity, which requires the use, first of all, of new algorithmic methods and an understanding of the assessment of the complexity of the calculations. This area of research is not given due attention for various reasons, but primarily because of the novelty of this paradigm, as well as the use of other advanced methods, which is briefly analyzed in this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 229.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 299.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 299.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.mpi-inf.mpg.de/departments/algorithms-complexity/.

  2. 2.

    https://developer.nvidia.com/cuda-zone.

  3. 3.

    https://www.nvidia.com/en-us/data-center/dgx-1/.

  4. 4.

    https://en.wikipedia.org/wiki/Parallel_computing

  5. 5.

    https://en.wikipedia.org/wiki/Amdahl%27s_law.

  6. 6.

    https://en.wikipedia.org/wiki/Graphics_processing_unit.

  7. 7.

    https://en.wikipedia.org/wiki/NP_(complexity)

  8. 8.

    https://en.wikipedia.org/wiki/Universal_approximation_theorem.

  9. 9.

    https://en.wikipedia.org/wiki/Unsupervised_learning.

  10. 10.

    https://en.wikipedia.org/wiki/Hyperparameter_optimization

  11. 11.

    https://www.mpi-inf.mpg.de/departments/algorithms-complexity/teaching/summer16/poly-complexity/.

References

  1. Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of Machine Learning, p. 427. The MIT Press, Cambridge, Massachusetts, London, England (2012)

    MATH  Google Scholar 

  2. Knuth D.E.: The Art of Computer Programming. 3rd ed. vol 1. Addison-Wesley (2013) 672pp

    Google Scholar 

  3. Knuth, E.D.: The Art of Computer Programming, Seminumerical Algorithms, 3rd edition ed., vol 2. Addison-Wesley, 762pp

    Google Scholar 

  4. Graham, R.L., Knuth, D.E., Patashnik, O.: Concrete Mathematics. 2nd ed. Addison-Wesley Publishing Company (1994) 604pp

    Google Scholar 

  5. Cormen, T.H., Leiserson, C.E., Rivest, R.L.: Introduction to Algorithms, 3rd ed. The MIT Press, 1292pp

    Google Scholar 

  6. Sedgewick, R., Wayne, K.: Algorithms. Addison-Wesley (2013), 848pp

    Google Scholar 

  7. Dasgupta, S.: Algorithms, 1st ed. McGraw-Hill Education, 336pp

    Google Scholar 

  8. Knuth, D.E.: The Art of Computer Programming. Generating All Trees-History of Combinatorial Generation, vol. 4 (2007), 160pp

    Google Scholar 

  9. Rieck, B., Togninalli, M., Bock, C., Moor, M.: Neural persistence: a complexity measure for deep neural networks using algebraic topology. ICLR (2019)

    Google Scholar 

  10. Williamson, D.P., Shmoys, D.B.: The Design Approximation Algorithms. Cambridge University Press (2010)

    Google Scholar 

  11. Бopecкoв, A.B., Xapлaмoв, A.A., Mapкoвcкий, H.Д., Mикyшин, Д.H., Mpoтникoв, E.B., Mыльцeв, A.A., Caxapныx, H.A., Фpoлoв, B.A.: Пapaллeльныe вычиcлeния нa GPU. Apxитeктypa и пpoгpaммнaя мoдeль CUDA, Издaтeльcтвo Mocкoвcкoгo yнивepcитeтa (2012)

    Google Scholar 

  12. Graham, R.L., Knuth, D.E., Patashnik, O.: Concrete Mathematics. Reading. Addison-Wesley Publishing Company, Inc., Massachusetts (1994, 1989), 604pp

    Google Scholar 

  13. Li, L., Abu-Mostafa, Y.S.: Data Complexity in Machine Learning, Caltech Computer Science Technical Report CaltechCSTR:2006.004, May 2006 (2006)

    Google Scholar 

  14. Widrow, B.: Adaptive Signal Processing, 1st ed. (1985), 486pp

    Google Scholar 

  15. Bengio, Y.: Learning Deep Architectures for AI, Technical Report 1312. [Online]. https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf

  16. Bengio, Y., LeCun, Y.: Scaling Learning Algorithms towards AI. In: Large-Scale Kernel Machines, P. 408. The MIT Press (2007)

    Google Scholar 

  17. Barron, A.R.: Approximation and estimation bounds for artificial neural network. Mach. Learn. 14, 115–133 (1994)

    Google Scholar 

  18. Hassibi, B., Stork, D.G.: Second order derivatives for network pruning: optimal brain surgeon. Adv. Neural Inf. Proc. Syst. 5(NIPS 1992) (1992)

    Google Scholar 

  19. Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. https://papers.nips.cc/paper/3048-greedy-layer-wise-training-of-deep-networks.pdf (дaтa oбpaщeния: 15.09.2019)

  20. Justus, D., Brennan, J., Bonner, S., McGough, A.S.: Predicting the computational cost of deep learning models. https://arxiv.org/pdf/1811.11880.pdf (дaтa oбpaщeния: 1.11.2019)

  21. Scholkopf, B., Christopher, Smola, A.J.: Advances in Kernel Methods. The MIT Press (1999)

    Google Scholar 

  22. Corinna Cortes, Vladimir Vapnik. Support-Vector Networks. http:/​/​homepages.rpi.edu/​ ~ bennek/​class/​mmld/​papers/​svn.pdf (дaтa oбpaщeния: 02.10.2019)

    Google Scholar 

  23. Dall’Anese, E., Simonetto, A., Becker, S., Madden, L.: Optimization and Learning with Information Streams: Time-varying Algorithms and Applications. https://arxiv.org/pdf/1910.08123v1.pdf (дaтa oбpaщeния: 22.10.2019)

  24. Colin, I., Dos Santos, L., Scaman, K.: Theoretical limits of pipeline parallel optimization and application to distributed deep learning http://arxiv-sanity.com/search?q = learning+and+optimization (дaтa oбpaщeния: 15.10.2019)

  25. Liu, S., Vicente, L.N.: The stochastic multi-gradient algorithm for multi-objective The stochastic multi-gradient algorithm for multi-objective. https://arxiv.org/pdf/1907.04472v2.pdf (дaтa oбpaщeния: 28.09.2019)

  26. Backurs, A., Indyk, P., Schmidt, L.: On the fine-grained complexity of empirical risk minimization:kernel methods and neural networks. https://papers.nips.cc/paper/7018-on-the-fine-grained-complexity-of-empirical-risk-minimization-kernel-methods-and-neural-networks.pdf (дaтa oбpaщeния: 02.10.2019)

  27. Shalev-Shwartz, S., Ben-David, S.: Understanding Machine Learning. Cambridge University Press, From Theory to Algorithms (2014)

    Book  Google Scholar 

  28. Fujimoto, S., Conti, E., Ghavamzadeh, M., Pineau, J.: Benchmarking Batch Deep Reinforcement Learning Algorithms. https://arxiv.org/pdf/1910.01708v1.pdf (дaтa oбpaщeния: 10.7.2019)

  29. Glasser, I., Sweke, R., Pancotti, N., Eisert, J., Cirac, J.I.: Expressive power of tensor-network factorizations for probabilistic modeling. https://arxiv.org/pdf/1907.03741v1.pdf (дaтa oбpaщeния: 03.10.2019)

  30. Sheriff, M.R., Chatterjee, D.: Dictionary Learning With Almost Sure Error Constraints. https://arxiv.org/pdf/1910.08828v1.pdf (дaтa oбpaщeния: 10.17.2019)

  31. Jungnickel, D.: Graphs, Networks and Algorithms. 3rd ed. Springer (2008)

    Google Scholar 

Download references

Acknowledgements

We thank the staff of SPbPU Peter the Great and the Institute of Computer Science and Technology for their support in the preparation of this material. We drew thoughts and ideas at joint seminars within the framework of the institute, as well as during fruitful communication with colleagues.

Author information

Authors and Affiliations

  1. Peter the Great St. Petersburg Polytechnic University, St. Petersburg, Russia

    Dmitry Baskakov & Dmitry Arseniev

Authors
  1. Dmitry Baskakov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dmitry Arseniev
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dmitry Baskakov .

Editor information

Editors and Affiliations

  1. Higher School of Software Engineering, Peter the Great St. Petersburg Polytechnic University, St. Petersburg, Russia

    Nikita Voinov

  2. Department of Computer Science and Biomedical Engineering, Graz University of Technology, Graz, Austria

    Tobias Schreck

  3. School of Maths, Computer Science and Engineering, City, University of London, London, UK

    Sanowar Khan

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Baskakov, D., Arseniev, D. (2021). On the Computational Complexity of Deep Learning Algorithms. In: Voinov, N., Schreck, T., Khan, S. (eds) Proceedings of International Scientific Conference on Telecommunications, Computing and Control. Smart Innovation, Systems and Technologies, vol 220. Springer, Singapore. https://doi.org/10.1007/978-981-33-6632-9_30

Download citation

  • DOI: https://doi.org/10.1007/978-981-33-6632-9_30

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-33-6631-2

  • Online ISBN: 978-981-33-6632-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation