Resolving learning rates adaptively by locating stochastic non-negative associated gradient projection points using line searches

Abstract

Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. Attempts to resolve learning rates adaptively, using line searches, have proven computationally demanding. Reducing the computational cost by considering mini-batch sub-sampling (MBSS) introduces challenges due to significant variance in information between batches that may present as discontinuities in the loss function, depending on the MBSS approach. This study proposes a robust approach to adaptively resolve learning rates in dynamic MBSS loss functions. This is achieved by finding sign changes from negative to positive along directional derivatives, which ultimately converge to a stochastic non-negative associated gradient projection point. Through a number of investigative studies, we demonstrate that gradient-only line searches (GOLS) resolve learning rates adaptively, improving convergence performance over minimization line searches, ignoring certain local minima and eliminating an otherwise expensive hyperparameter. We also show that poor search directions may benefit computationally from overstepping optima along a descent direction, which can be resolved by considering improved search directions. Having shown that GOLS is a reliable line search allows for comparative investigations between static and dynamic MBSS.

Appendices

A Artificial neural networks

The single and double hidden layer feedforward neural network architectures are expressed mathematically by Eqs. (23) and (24) respectively below. The optimization vector, ${\textit{\textbf{x}}}$ is sectioned and transformed into matrices ${\textit{\textbf{X}}}^{(c)}$ for the relevant weights in connection layers c of the network. The given data observation pair ${\textit{\textbf{t}}}_b$ is separated to give the input data, ${\textit{\textbf{T}}}^i_b$, and output data, ${\textit{\textbf{T}}}^o_b$. Suppose a given dataset has an input domain, ${\textit{\textbf{T}}}^i$ with $|{\mathcal {B}}|$ observations and D dimensions (features). The respective output domain, ${\textit{\textbf{T}}}^o$, has corresponding observations $|{\mathcal {B}}|$ and output dimensions E (classes). Then for every observation b and every output dimension e, a prediction of the output data $\hat{{\textit{\textbf{T}}}}^o$ can be constructed from the original data input domain ${\textit{\textbf{T}}}^i$, given by

$$\begin{aligned} \hat{{\textit{\textbf{T}}}}_{be}^o = a_{outer} \left( \sum _{b=1}^{M_1} {\textit{\textbf{X}}}_{ej}^{(2)} a_{inner} \left( \sum _{i=1}^{D} {\textit{\textbf{X}}}_{ji}^{(1)} {\textit{\textbf{T}}}_{bi}^i + {\textit{\textbf{X}}}_{j0}^{(1)}\right) + {\textit{\textbf{X}}}_{e0}^{(2)}\right) , \end{aligned}$$

(23)

for a single hidden layer neural network and

$$\begin{aligned} \hat{{\textit{\textbf{T}}}}_{be}^o = a_{outer} \left( \sum _{l=1}^{M_2} {\textit{\textbf{X}}}_{le}^{(3)} a_{inner}^{(2)} \left( \sum _{b=1}^{M_1} {\textit{\textbf{X}}}_{ej}^{(2)} a_{inner}^{(1)} \left( \sum _{i=1}^{D} {\textit{\textbf{X}}}_{ji}^{(1)} {\textit{\textbf{T}}}_{bi}^i + {\textit{\textbf{X}}}_{j0}^{(1)} \right) + {\textit{\textbf{X}}}_{e0}^{(2)} \right) + {\textit{\textbf{X}}}_{l0}^{(3)} \right) , \end{aligned}$$

(24)

for a double hidden layer neural network.

$M_{n}$, $n \in [1,2]$ gives number of nodes in the respective hidden layers. The nodal activation function is denoted by a and ${\textit{\textbf{X}}}^{(c)}$, $c \in [1,2,3]$ denotes the set of weights connecting sequential layers in the network between the input layer and the output layer in a forward direction. Thus the single hidden layer network has two sets of weights, ${\textit{\textbf{X}}}^{(c)}$, and the double hidden layer network has three respectively [8].

The nodal weights ${\textit{\textbf{x}}}$ are optimized to a configuration which best captures the relationship between the input and output data spaces. The loss-function used is the mean squared error (MSE), determined over every b in batch size $|{\textit{\textbf{B}}}|$ and every class $e \in E$ according to the Proben1 dataset guidelines [43] as:

$$\begin{aligned} \ell ({\textit{\textbf{x}}},{\textit{\textbf{t}}}_b) = \frac{100}{E\cdot |{\textit{\textbf{B}}}|} \sum _{b = 1 }^{|{\textit{\textbf{B}}}|} \sum _{e=1}^{E}(\hat{{\textit{\textbf{T}}}}_{be}^o({\textit{\textbf{x}}}) - {\textit{\textbf{T}}}_{be}^o)^2, \end{aligned}$$

(25)

where $\hat{{\textit{\textbf{T}}}}^o({\textit{\textbf{x}}})$ is the output estimation of the current network configuration as a function of the weights, and ${\textit{\textbf{T}}}^o$ is the target output of the corresponding training dataset samples.

B Exact line search: gradient-only line search with bisection (GOLS-B)

The directional derivative values used in this method are defined as $F'_n(\alpha ) = {{\textit{\textbf{g}}}}({\textit{\textbf{x}}}_n + \alpha \cdot {\textit{\textbf{d}}}_n)^T{\textit{\textbf{d}}}_n$ and the search direction, ${\textit{\textbf{d}}}_n$, at the respective values for $\alpha$ at the different points.

C Inexact line search: gradient-only line search that is inexact (GOLS-I)

Parameters used for this method are: $\eta = 2$, $c_2 = 0.9$, $\alpha _{min} = 10^{-8}$ and $\alpha _{max} = 10^7$. $F'_n(\alpha ) = {{\textit{\textbf{g}}}}({\textit{\textbf{x}}}_n + \alpha \cdot {\textit{\textbf{d}}}_n)^T{\textit{\textbf{d}}}_n$.

D Inexact line search: gradient-only line search maximizing step size (GOLS-Max)

Parameters used for this method are: $\eta = 2$, $c_2 = 0.9$, $\alpha _{min} = 10^{-8}$ and $\alpha _{max} = 10^7$. $F'_n(\alpha ) = {{\textit{\textbf{g}}}}({\textit{\textbf{x}}}_n + \alpha \cdot {\textit{\textbf{d}}}_n)^T{\textit{\textbf{d}}}_n$.

E Inexact line search: gradient-only line search with backtracking (GOLS-Back)

Parameters used for this method are: $\eta = 2$, $c_2 = 0$, $\alpha _{min} = 10^{-8}$ and $\alpha _{max} = 10^7$. $F'_n(\alpha ) = {{\textit{\textbf{g}}}}({\textit{\textbf{x}}}_n + \alpha \cdot {\textit{\textbf{d}}}_n)^T{\textit{\textbf{d}}}_n$.