Goal-Oriented Sensitivity Analysis of Hyperparameters in Deep Learning

Abstract

Tackling new machine learning problems with neural networks always means optimizing numerous hyperparameters that define their structure and strongly impact their performances. In this work, we study the use of goal-oriented sensitivity analysis, based on the Hilbert–Schmidt independence criterion (HSIC), for hyperparameter analysis and optimization. Hyperparameters live in spaces that are often complex and awkward. They can be of different natures (categorical, discrete, boolean, continuous), interact, and have inter-dependencies. All this makes it non-trivial to perform classical sensitivity analysis. We alleviate these difficulties to obtain a robust analysis index that is able to quantify hyperparameters’ relative impact on a neural network’s final error. This valuable tool allows us to better understand hyperparameters and to make hyperparameter optimization more interpretable. We illustrate the benefits of this knowledge in the context of hyperparameter optimization and derive an HSIC-based optimization algorithm that we apply on MNIST and Cifar, classical machine learning data sets, but also on the approximation of Runge function and Bateman equations solution, of interest for scientific machine learning. This method yields neural networks that are both competitive and cost-effective.

Appendices

Appendix A Hyperparameters Spaces

In this section, we describe hyperparameters spaces used for each problem in this chapter. Note that hyperparameter n_seeds denotes the number of random repetitions of the training for each hyperparameter configuration. If a conditional hyperparameter $X_j$ is only involved for some specific values of a main hyperparameter $X_i$, it is displayed with an indent on tab lines below that of $X_i$, with the value of $X_i$ required for $X_j$ to be involved in the training.

1.1 Runge and MNIST

For Runge and MNIST, only fully connected Neural Networks are trained, and the width (n_units) is the same for every layer (Tables ).

Table 5 Hyperparameters values for Runge & MNIST

Conditional Groups: (see (iii) of Sect. 4.3) $\mathcal {G}_0$ and $\mathcal {G}_{\texttt {dropout\_rate}}$

1.2 Bateman

For Bateman, only fully connected Neural Networks are trained, and the width (n_units) is the same for every layer (Table ).

Table 6 Hyperparameters values for Bateman

Conditional groups: (see (iii) of Sect. 4.3) $\mathcal {G}_0$, $\mathcal {G}_{\texttt {dropout\_rate}}$, $\mathcal {G}_{\texttt {amsgrad}}$, $\mathcal {G}_{\texttt {centered}}$, $\mathcal {G}_{\texttt {nesterov}}$, $\mathcal {G}_{\texttt {momentum}}$ and $\mathcal {G}_{(\texttt {1st\_moment}, \texttt {2nd\_moment})}$

1.3 Cifar10

For Cifar10, we use Convolutional Neural Networks, whose width increases with the depth according to hyperparameters stages and stage_mult. The first layer has width n_filters, and then, $\texttt {stages} - 1$ times, the network is widen by a factor stage_mult. For instance, a neural network with $\texttt {n\_filters} = 20$, $\texttt {n\_layers}=3$, $\texttt {stages}=3$ and $\texttt {stage\_mult}=2$ will have a first layer with 20 filters, a second layer with $\texttt {n\_filters} \times {stage\_mult} = 40$ filters, and a third layer with $\texttt {n\_filters} \times \texttt {stage\_mult}^{\texttt {stages}-1} = 60$ filters (Table ).

Table 7 Hyperparameters values for Cifar10

Conditional groups: (see (iii) of Sect. 4.3) $\mathcal {G}_0$, $\mathcal {G}_{\texttt {dropout\_rate}}$, $\mathcal {G}_{\texttt {amsgrad}}$, $\mathcal {G}_{\texttt {centered}}$, $\mathcal {G}_{\texttt {nesterov}}$, $\mathcal {G}_{\texttt {momentum}}$ and

$\mathcal {G}_{(\texttt {1st\_moment}, \texttt {2nd\_moment})}$

Appendix B HSICs for Conditional Hyperparameters

1.1 B.1 MNIST

For MNIST, there is only one conditional hyperparameter, dropout_rate, so only one conditional group to consider in order to assess the importance of conditional hyperparameters (Fig. ).

Fig. 14

HSICs for $\mathcal {G}_{\texttt {dropout\_rate}}$ of MNIST hyperparameter analysis. Conditional hyperparameter dropout_rate is not impactful

1.2 B.2 Bateman

For Bateman, there are seven conditional hyperparameter, amsgrad, 1st_moment (beta_1), 2nd_moment (beta_2), dropout_rate, centered, momentum, and nesterov. Six conditional groups, specified in Fig. , have to be considered in order to assess their importance.

Fig. 15

a $\mathcal {G}_{\texttt {amsgrad}}$. b $\mathcal {G}_{(\texttt {1st\_moment}, \texttt {2nd\_moment})}$ c $\mathcal {G}_{\texttt {centered}}$ d $\mathcal {G}_{\texttt {dropout\_rate}}$ f $\mathcal {G}_{\texttt {momentum}}$ g $\mathcal {G}_{\texttt {nesterov}}$ HSICs for conditional groups of Bateman hyperparameter analysis. a amsgrad is not impactful (it is in the estimation noise), b 1st_moment is not impactful but 2nd_moment is the fourth most impactful hyperparameter of this group, c centered is the second most impactful hyperparameter of this group, d dropout_rate is not impactful, e momentum is not impactful, f nesterov is the most impactful hyperparameter of this group

1.3 B. 3 Cifar10

For Cifar10, there are seven conditional hyperparameter, amsgrad, 1st_moment (beta_1), 2nd_moment (beta_2), dropout_rate, centered, momentum, and nesterov. Six conditional groups, specified in Fig. , have to be considered in order to assess their importance.

Fig. 16

a $\mathcal {G}_{\texttt {amsgrad}}$ b $\mathcal {G}_{(\texttt {1st\_moment}, \texttt {2nd\_moment})}$ c $\mathcal {G}_{\texttt {centered}}$ d $\mathcal {G}_{\texttt {dropout\_rate}}$ e $\mathcal {G}_{\texttt {momentum}}$ f $\mathcal {G}_{\texttt {nesterov}}$. HSICs for conditional groups of cifar10 hyperparameter analysis. a amsgrad is not impactful, b 1st_moment, 2nd_moment are not impactful, c centered is the third most impactful hyperparameter of this group, d dropout_rate is not impactful, e momentum is not impactful, f nesterov is not impactful

Appendix C Construction of Bateman Data Set

Bateman data set is based on the resolution of the Bateman equations, which is an ODE system modeling multi species reactions:

$$\begin{aligned} \partial _t\varvec{\eta }(t) = \varvec{\Sigma _r}\big (\varvec{\eta }(t)\big ) \cdot \varvec{\eta }(t) \text {, with initial conditions } \varvec{\eta }(0) = \varvec{\eta _0}, \end{aligned}$$

and $\varvec{\eta }\in (\mathbb {R}^{+})^M$, $\varvec{\Sigma }_r \in \mathbb {R}^{M\times M}$. Here, $f: ( \varvec{\eta _0}, t) \rightarrow \varvec{\eta }(t)$, and we are interested in $\varvec{\eta }(t)$, which is the concentration of each of the species $S_k$, with $k \in \{1,\ldots ,M\}$. For physical applications, M ranges from tens to thousands. We consider the particular case $M=11$. Matrix $\varvec{\Sigma _r}\big (\varvec{\eta }(t)\big )$ depends on reaction constants. Here, 4 reactions are considered and each reaction p has constant $\sigma _p$.

$$\begin{aligned} {\left\{ \begin{array}{ll} (1): S_1 + S_2 \rightarrow S_3 + S_4 + S_6 + S_7, \\ (2): S_3 + S_4 \rightarrow S_2 + S_8 + S_{11}, \\ (3): S_2 + S_{11} \rightarrow S_3 + S_5 + S_9, \\ (4): S_3 + S_{11} \rightarrow S_2 + S_5 + S_6 + S_{10}, \end{array}\right. } \end{aligned}$$

with $\sigma _1=1$, $\sigma _2=5$, $\sigma _3=3$ and $\sigma _4=0.1$. To obtain $\varvec{\Sigma _r}\big (\varvec{\eta }(t)\big )$, the species have to be considered one by one. Here we give an example of how to construct the second row of $\varvec{\Sigma _r}\big (\varvec{\eta }(t)\big )$. The other rows are built the same way. Given the reaction equations :

$$\begin{aligned} \partial _t \eta _2 = - \sigma _1\eta _1\eta _2 + \sigma _2\eta _3\eta _4 - \sigma _3\eta _2\eta _{11} + \sigma _4\eta _3\eta _{11}, \end{aligned}$$

because $S_2$ disappears in reactions (1) and (3) involving $S_1$ and $S_{11}$ as other reactants at rate $\sigma _1$ and $\sigma _3$, respectively, and appears in reactions (2) and (4) involving $S_3$, $S_4$ and $S_3$, $S_{11}$ as reactants, at rate $\sigma _2$ and $\sigma _4$ respectively. Hence, the second row of $\varvec{\Sigma _r}\big (\varvec{\eta }(t)\big )$ is

$$\begin{aligned} {[}0, \; -\sigma _1 \eta _1, \; 0, \; \sigma _2 \eta _3, \; 0, \; 0, \; 0, \; 0, \; 0, \; 0, \; -\sigma _3 \eta _2 + \sigma _4 \eta _3 ], \end{aligned}$$

with $\varvec{\eta }(t)$ denoted by $\varvec{\eta }$ to simplify the equation and $\eta _i$ the i-th component of $\varvec{\eta }$. To construct the training, validation and test data sets, we sample uniformly $( \varvec{\eta }_0, t) \in [0,1]^{12} \times [0,5]$ 130000 times. We denote these samples $( \varvec{\eta }_0, t)_i$ for $i \in \{1,\ldots ,130000\}$. Then, we apply a first order Euler solver with a time step of $10^{-3}$ to compute $f(( \varvec{\eta }_0, t)_i)$. As a result, neural network’s input is $( \varvec{\eta }_0, t)$ and neural network’s output is $f(( \varvec{\eta }_0, t))$.