Combining Stochastic Adaptive Cubic Regularization with Negative Curvature for Nonconvex Optimization

Abstract

We focus on minimizing nonconvex finite-sum functions that typically arise in machine learning problems. In an attempt to solve this problem, the adaptive cubic-regularized Newton method has shown its strong global convergence guarantees and the ability to escape from strict saddle points. In this paper, we expand this algorithm to incorporating the negative curvature method to update even at unsuccessful iterations. We call this new method Stochastic Adaptive cubic regularization with Negative Curvature (SANC). Unlike the previous method, in order to attain stochastic gradient and Hessian estimators, the SANC algorithm uses independent sets of data points of consistent size over all iterations. It makes the SANC algorithm more practical to apply for solving large-scale machine learning problems. To the best of our knowledge, this is the first approach that combines the negative curvature method with the adaptive cubic-regularized Newton method. Finally, we provide experimental results, including neural networks problems supporting the efficiency of our method.

Appendix A: Experiment Settings on Numerical Results

In “Appendix A”, we present some experimental results to show the effectiveness of the SANC algorithm for stochastic nonconvex optimization problems. In our numerical experiments, we considered three machine learning problems, namely (i) the logistic regression (ii) the multilayer perceptron, and (iii) the convolutional neural networks (CNN) with real datasets.

For the logistic regression problems, with the binary classification dataset, i.e., $\{\mathbf {x}_i,y_i\}$, $y_i\in \{0,1\}$, in order to find the optimum $\mathbf {w}^*$, we solved the following problem,

$$\begin{aligned} \min _{\mathbf {w}\in \mathbb {R}^d}&-\frac{1}{n}\sum _{i=1}^{n}y_i\log \left( \frac{1}{1+e^{-\mathbf {w}^T\mathbf {x}_i}}\right) \end{aligned}$$

(68)

$$\begin{aligned}&+\log \left( \frac{e^{\mathbf {w}^T\mathbf {x}_i}}{1+e^{-\mathbf {w}^T\mathbf {x}_i}}\right) +\lambda \Omega (\mathbf {w}) \end{aligned}$$

(69)

where $\Omega (\mathbf {w})=\sum _{i=1}^d \frac{w_i^2}{1+w_i^2}$, and $\lambda$ is a fixed regularization coefficient. We initialized all variables by $\mathbf {w}_0=1$.

For a multilayer perceptron, we used two hidden layers. We use the first layer with the size of 300, the second layer with the size of 500, and use hyperbolic tangent functions as an activation. At the outer layer, softmax functions were used. Cross-entropy loss was used as an objective function to be minimized. We also added l2 norm as a convex regularization term with a coefficient $\lambda =0.01$.

For CNN, we used two convolutional receptive filters. The first one is $5\times 5\times 32$ dimensional, and the second one is $5\times 5\times 64$ dimensional. The fully connected layer was added at the output of the convolutional layers, which has 1000 neurons. The settings for nonlinear activation functions and l2 norm are the same as those of multilayer perceptron.

All the variables, including the weights and the biases in the multilayer perceptron and the CNN, were initialized by the Xavier initialization [32]. We used real datasets from libsvm [33] for the logistic regression problem and multilayer perceptron. For CNN, we used MNIST [34] and CIFAR10 datasets [35].

The sizes of $\mathcal {S}_\mathbf {g}$ and $\mathcal {S}_\mathbf {B}$ are $\lceil \text {the number of datapoints}/20\rceil$ for the logistic regression problems. For the multilayer perceptron and CNN, we computed the stochastic gradients and Hessian estimators using independently drawn mini-batches of size 128. These batch size settings are used for SANC as well as all the baselines, which are described below.

We compared our method, SANC, with various optimization methods as follows:

• SGD Fixed step size is used. We used 0.01 for the logistic regression problems and 0.001 for the multilayer perceptron and CNN problems, which is the best choice among $10^{-3:1:3}$, respectively.

• SCR [19] We used the same parameter values for SANC. We used same sets, $\mathcal {S}_\mathbf {g}$, $\mathcal {S}_\mathbf {B}$ in SANC, which is different with the experiments in [19] where they used increasing sized sets.

• CR [14] For the fixed cubic coefficient $\sigma$, we used 5,  for all problems.

• CRM [16] For the fixed cubic coefficient $\sigma$, we used 5,  for all problems. The momentum parameter $\beta$ is set to $8\times \Vert \mathbf {s}\Vert$ as in [16].

• NCD [2] We used the same parameter values what we used for SANC.

As a default setting, we used $\gamma =2$, $\eta _1=0.2$, $\eta _2=0.8$, and $\sigma _0=1$. These parameter settings are the same as those in SCR [19]. For the neural networks problems, we used $\eta _1=0.1$ and $\eta _2=0.3$ tuned by some experiments. $L_1$ and $L_2$ parameters in SANC have tuned. The searching range for $L_1$ and $L_2$ is $10^{-3:1:3}$. The used parameter values are presented in Table 1.

Table 1 Datasets used for numerical experiments

For the Lanczos method, we truncated it at Lanczos iteration 5 for all experiments. Also, an approximate minimizer of an approximate local cubic model over a Krylov subspace was obtained by the ‘conjugate gradient method’ in scipy.optimize.minimize. To calculate the left-most eigenpair of $\mathbf {T}$, we used eigh_tridiagonal in scipy.linalg.

For SANC and SCR, we have to calculate $f(\mathbf {x}_t)$ and $f(\mathbf {x}_t+\mathbf {s}_t)$ to measure $\rho _t$. For the neural networks problems, because of the shortage of the memory storage, we made another set of dataset whose size is same as $\mathcal {S}_\mathbf {g}$ and $\mathcal {S}_\mathbf {B}$ to estimate $f(\mathbf {x}_t)$ and $f(\mathbf {x}_t+\mathbf {s}_t)$.

All baselines are our implementations using Tensorflow. One can reach out to the code at https://github.com/seonho-park/Stochastic-Adaptive-cubic-regularization-with-Negative-Curvature.