A Comparative Analysis of Gradient-Based Optimization Methods for Machine Learning Problems

Abstract

In this study, we compare and contrast the seven most widely used gradient-based optimization algorithms of the first order for machine learning problems. These methods are Stochastic Gradient Descent with momentum (SGD), Adaptive Gradient (AdaGrad), Adaptive Delta (AdaDelta), Root Mean Square Propagation (RMSProp), Adaptive Moment Estimation (Adam), Nadam (Nestrove accelerated adaptive moment estimation) and Adamax (maximum adaptive moment estimation). For model creation and comparison, three test problems based on regression, binary classification and multi-classification are addressed. Using three randomly selected datasets, we trained the model and evaluated the optimization strategy in terms of accuracy and loss function. The total experimental results demonstrate that Nadam outperformed the other optimization approach across these datasets, but only in terms of correctness, not in terms of time. Adam optimizer has the best performance in terms of time and accuracy.