SCMP-IL: an incremental learning method with super constraints on model parameters

Abstract

Deep learning technology has played an important role in our life. Since deep learning technology relies on the neural network model, it is still plagued by the catastrophic forgetting problem, which refers to the neural network model will forget what it has learned after learning new knowledge. The neural network model learns knowledge through labeled samples, and its knowledge is stored in its parameters. Therefore, many methods try to solve this problem from the perspective of constraint parameters and stored samples. There are few ways to solve this problem from the perspective of constraining features output of neural network models. This paper proposes an incremental learning method with super constraints on model parameters. This method not only calculates the parameter similarity loss of the old and new models, but also calculates the layer output feature similarity loss of the old and new models, and finally suppresses the change of model parameters from two directions. In addition, we also propose a new strategy for selecting representative samples from dataset and tackling the imbalance between stored samples and new task samples. Finally, we utilize the neural kernel mapping support vector machine theory to increase the interpretability of the model. In order to better meet the actual situation, five sample sets with different categories and amounts were employed in experiments. Experiments show the effectiveness of our method. For example, after learning the last task, our method is at least 1.930% and 0.562% higher than other methods on the training set and test set, respectively.

Appendices

Appendix 1

Table

Table 8 Without employing any strategy to prevent catastrophic forgetting, training set Top-1 accuracies of all tasks after adding new tasks

8 and Table

Table 9 Without employing any strategy to prevent catastrophic forgetting, test set Top-1 accuracies of all tasks after adding new tasks

9 show experimental results of training set and test set of incremental learning without any strategy to prevent catastrophic forgetting, respectively. The purpose of showing two tables is to illustrate the severity of catastrophic forgetting in the neural network model. Without employing any strategy to prevent catastrophic forgetting, the neural network model will completely forget the previous knowledge.

Appendix 2

Comparative experiments of main hyperparameters of Mixer Layer is shown.

Table

Table 10 Setting different numbers of Mixer Layer and then getting training set Top-1 average accuracies of all tasks (%)

10 and Table

Table 11 Setting different numbers of Mixer Layer and then getting test set Top-1 average accuracies of all tasks (%)

11 respectively show Top-1 average accuracies of the training set and test set of all tasks after stacking different Mixer Layers. It can be seen from Table 10 and Table 11 that setting the number of Mixer Layer overlays to 8 is the most appropriate.

Table

Table 12 Setting different numbers of Patch Size and then getting training set Top-1 average accuracies of all tasks (%)

12 and Table

Table 13 Setting different numbers of Patch Size and then getting test set Top-1 average accuracies of all tasks (%)

13 represents Top-1 average accuracies of the training set and test set of all tasks after using different patch sizes. From Table 12, it can be seen that setting the patch size to 16 is the most appropriate, and the performance is the best at this time. However, from Table 13, it seems most appropriate to set the patch size to 4. In the experiment, we found that with the decrease of patch size, the time cost and the GPU memory occupied will increase rapidly. In addition, in Table 13, the performance when patch size is set to 4 is only slightly higher than that when patch size is 16. So in the final experiment, we set the Patch Size to 16.

Appendix 3

In order to reduce the use of loop statements and improve the running efficiency of the program, the formula for calculating the Euclidean distance between sample set ${d}_{i}$ and its sample center ${center}_{i}$ in a single task is given here, and the derivation process is also given. Here, we take the image set as an example for calculation and derivation. This class consists of m images, and an image is represented by ${I}_{[c,w,h]}$, where c, w and h represent the color channel, width and height of the image, respectively. The length of the flattened ${I}_{[c,w,h]}$ is $c*w*h$. Let $\mathrm{n}=\mathrm{c}*\mathrm{w}*\mathrm{h}$, m flattened images can be represented by ${M}_{[m,n]}$. Let ${{\varvec{a}}}_{[1,n]}=({a}_{1},{a}_{2},\cdots ,{a}_{n})$ and $M_{[m,n]} = \left[ {\begin{array}{*{20}c} {b_{11} } & {b_{12} } & \cdots & {b_{1n} } \\ {b_{21} } & {b_{22} } & \cdots & {b_{2n} } \\ \cdots & \cdots & \ddots & \cdots \\ {b_{m1} } & {b_{m2} } & \cdots & {b_{mn} } \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {{\varvec{b}}_{1} } \\ {{\varvec{b}}_{2} } \\ \cdots \\ {{\varvec{b}}_{m} } \\ \end{array} } \right]$,

The Euclidean distance between each row of ${{\varvec{M}}}_{[m,n]}$ and vector ${{\varvec{a}}}_{[1,n]}$ can be expressed by the following formula:

$$\begin{gathered} R_{[m,1]} = sqrt\left( {\left[ {\begin{array}{*{20}c} {\left\| {{\varvec{b}}_{1} } \right\|^{2} + \left\| {{\varvec{a}}_{[1,n]} } \right\|^{2} - 2{\varvec{b}}_{1} \cdot {\varvec{a}}_{[1,n]}^{T} } \\ {\left\| {{\varvec{b}}_{2} } \right\|^{2} + \left\| {{\varvec{a}}_{[1.n]} } \right\|^{2} - 2{\varvec{b}}_{2} \cdot {\varvec{a}}_{[1,n]}^{T} } \\ \vdots \\ {\left\| {{\varvec{b}}_{m} } \right\|^{2} + \left\| {{\varvec{a}}_{[1,n]} } \right\|^{2} - 2{\varvec{b}}_{m} \cdot {\varvec{a}}_{[1,n]}^{T} } \\ \end{array} } \right]_{[m,1]} } \right) \hfill \\ \, \hfill \\ \, = sqrt\left( {\left[ {\begin{array}{*{20}c} {\left\| {{\varvec{b}}_{1} } \right\|^{2} } \\ {\left\| {{\varvec{b}}_{2} } \right\|^{2} } \\ \cdots \\ {\left\| {{\varvec{b}}_{m} } \right\|^{2} } \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} 1 \\ 1 \\ \cdots \\ 1 \\ \end{array} } \right]_{[m,1]} \left\| {{\varvec{a}}_{[1,n]} } \right\|^{2} - 2\left[ {\begin{array}{*{20}c} {{\varvec{b}}_{1} } \\ {{\varvec{b}}_{2} } \\ \cdots \\ {{\varvec{b}}_{m} } \\ \end{array} } \right]{\varvec{a}}_{[1,n]}^{T} } \right) \hfill \\ \, = sqrt\left( {sum(M_{m*n}^{2} , \, {\mathbf{axis = 1}}) + \left[ {\begin{array}{*{20}c} 1 \\ 1 \\ \cdots \\ 1 \\ \end{array} } \right]_{[m,1]} \left\| {{\varvec{a}}_{[1,n]} } \right\|^{2} - 2M_{[m,n]} {\varvec{a}}_{[1,n]}^{T} } \right) \hfill \\ \end{gathered}$$

(13)

where ${M}_{[m,n]}^{2}$ means to calculate the second power of its each element, and $\mathrm{sum}({M}_{[m,n]}^{2},\mathrm{ axis}=1)$ represents the sum of the elements of each row of ${M}_{[m,n]}^{2}$; sqrt (‘*’) indicates that the square root operation is performed on each element of ‘*’. The element ${{\varvec{R}}}_{i1}$ of ${{\varvec{R}}}_{[m,1]}$ is the Euclidean distance between the i-th row of ${{\varvec{M}}}_{[m,n]}$ and vector ${{\varvec{a}}}_{[1,n]}$, where 1 ≤ i ≤ m.