Abstract
Any optimization of gradient descent methods involves selecting a learning rate. Tuning the learning rate can quickly become repetitive with deeper models of image classification, does not necessarily lead to optimal convergence. We proposed in this paper, a modification of the gradient descent algorithm in which the Nestrove step is added, and the learning rate is update in each epoch. Instead, we learn learning rate itself, either by Armijo rule, or by control step. Our algorithm called fast gradient descent (FGD) for solving image classification with neural networks problems, the quadratic convergence rate \(o(k^2)\) of FGD algorithm are proved. FGD algorithm are applicate to a MNIST dataset. The numerical experiment, show that our approach FGD algorithm is faster than gradient descent algorithms.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–27 (2009)
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., et al.: End to End Learning for Self-driving Cars (2016). arXiv:1604.07316
Botev, A., Lever, G., Barber, D.: Nesterov’s accelerated gradient and momentum as approximations to regularised update descent. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 1899–1903 (2017)
Cui, N.: Applying gradient descent in convolutional neural networks. J. Phys. Conf. Ser. 1004, 012027 (2018)
Chen, A., Xu, X., Ryu, S., Zhou, Z.: A self-adaptive Armijo stepsize strategy with application to traffic assignment models and algorithms. Transp. A Transp. Sci. 9(8), 695–712 (2013)
El Mouatasim, A., Ellaia, R., Souza de Cursi, J.E.: Stochastic perturbation of reduced gradient & GRG methods for nonconvex programming problems. J. Appl. Math. Comput. 226, 198–211 (2014)
El Mouatasim, A., Wakrim, M.: Control subgradient algorithm for image regularization. J. Signal Image Video Process. (SIViP) 9, 275–283 (2015)
El Mouatasim, A.: Control proximal gradient algorithm for \(\ell _1\) regularization image. J. Signal Image Video Process. (SIViP) 13(6), 1113–1121 (2019)
Evans, B., Al-Sahaf, H., Xue, B., Zhang, M.: Evolutionary deep learning: a genetic programming approach to image classification. In: IEEE Congress on Evolutionary Computation, pp. 1–6 (2018)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Huang, K., Hussain, A., Wang, Q., Zhang, R.: Deep Learning: Fundamentals, Theory and Applications. Springer, Berlin (2019)
LeCun, Y., Cortes, C.: MNIST Handwritten Digit Database. AT&T Labs, vol. 2. (2010). http://yann.lecun.com/exdb/mnist/
Li, P., Lee, S., Park, J.: Development of a global batch clustering with gradient descent and initial parameters in colour image classification. IET Image Process 13(1), 161–174 (2019)
Liu, G., Xiao, L., Xiong, C.: Image classification with deep belief networks and improved gradient descent. In: International Conference on Embedded and Ubiquitous Computing (2017)
MacLean, J., Tsotsos, J.: Fast Pattern Recognition Using Gradient-Descent Search in an Image Pyramid. IEEE (2000)
Nesterov, Y.: Introduction Lectures on Convex Optimization: A Basic Course, vol. 87. Springer, Berlin (2004)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Shi, B., Lyengar, S.S.: Mathematical Theories of Machine Learning: Theory and Applications. Springer, Berlin (2020)
Singh, B.K., Verma, K., Thoke, A.S.: Adaptive gradient descent backpropagation for classification of breast tumors in ultrasound imaging. Proc. Comput. Sci. 46, 1601–1609 (2015)
Uryas’ev, S.P.: New variable-metric algorithms for nondifferentiable optimization problems. J. Optim. Theory Appl. 71(2), 359–388 (1991)
Xinhua, L., Qian, Y.: Face recognition based on deep neural network. Int. J. Signal Process. Image Process. Pattern Recognit. 8(10), 29–38 (2015)
Acknowledgements
We indebted to the anonymous Reviewers and Editors for many suggestions and stimulating comments to improve the original manuscript.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
El Mouatasim, A. Fast gradient descent algorithm for image classification with neural networks. SIViP 14, 1565–1572 (2020). https://doi.org/10.1007/s11760-020-01696-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-020-01696-2