Sparse semi-autoencoders to solve the vanishing information problem in multi-layered neural networks

Abstract

The present paper aims to propose a new neural network called “sparse semi-autoencoder” to overcome the vanishing information problem inherent to multi-layered neural networks. The vanishing information problem represents a natural tendency of multi-layered neural networks to lose information in input patterns as well as training errors, including also natural reduction in information due to constraints such as sparse regularization. To overcome this problem, two methods are proposed here, namely, input information enhancement by semi-autoencoders and the separation of error minimization and sparse regularization by soft pruning. First, we try to enhance information in input patterns to prevent the information from decreasing when going through multi-layers. The information enhancement is realized in a form of new architecture called “semi-autoencoders”, in which information in input patterns is forced to be given to all hidden layers to keep the original information in input patterns as much as possible. Second, information reduction by the sparse regularization is separated from a process of information acquisition as error minimization. The sparse regularization is usually applied in training autoencoders, and it has a natural tendency to decrease information by restricting the information capacity. This information reduction in terms of the penalties tends to eliminate even necessary and important information, because of the existence of many parameters to harmonize the penalties with error minimization. Thus, we introduce a new method of soft pruning, where information acquisition of error minimization and information reduction of sparse regularization are separately applied without a drastic change in connection weights, as is the case of the pruning methods. The two methods of information enhancement and soft pruning try jointly to keep the original information as much as possible and particularly to keep necessary and important information by enabling the making of a flexible compromise between information acquisition and reduction. The method was applied to the artificial data set, eye-tracking data set, and rebel forces participation data set. With the artificial data set, we demonstrated that the selectivity of connection weights increased by the soft pruning, giving sparse weights, and the final weights were naturally interpreted. Then, when it was applied to the real data set of eye tracking, it was confirmed that the present method outperformed the conventional methods, including the ensemble methods, in terms of generalization. In addition, for the eye-tracking data set, we could interpret the final results according to the conventional eye-tracking theory of choice process. Finally, the rebel data set showed the possibility of detailed interpretation of relations between inputs and outputs. However, it was also found that the method had the limitation that the selectivity by the soft pruning could not be increased.

Appendix: Soft pruning by layer-wise selectivity increase

We briefly explain here how to modify connection weights to have strong selectivity. The selectivity of connection weights is realized by multiplying the connection weights by their normalized importance with a fixed number of learning steps (N steps), as shown in Fig. 11. In the first learning step in Fig. 11a, the ordinary learning is performed with T learning epochs. Then, from the second step on, the normalized importance of connection weights is computed, and weights are multiplied by their importance in Fig. 11b. The effect of importance values for the weights naturally diminishes when the number of epochs increases. Thus, when the number of steps goes beyond T epochs, the normalized importance is again computed and multiplied, and so on. The actual value of T was fixed to 100 in all the experiments, because we maximally repeated the learning step ten times, amounting to a total of 1,000 epochs, which corresponded to the default learning epochs of the Matlab neural network package. As mentioned above, we tried to use the default parameter values as much as possible for easy reproduction of the present results.

Fig. 11

Weight modification taking into account the selectivity from the first a to the final N th step c with T epochs in each step. In the figure, solid lines and dotted lines respectively represent positive and negative weights

We present here how to control the selectivity step by step, and the superscript s for input patterns is omitted for simplification. In the first epoch of the first step denoted by the superscript (1, 1), we have the output from the j th hidden neuron

$$ {~}^{(1,1)}v_{j}^{(2)} = \text{tansig} \left( \sum\limits_{k = 1}^{n_{1}} {~}^{(1,1)}w_{j k}^{(2)} x_{k} \right). $$

(19)

Then, the output from the output neuron (input) is computed by

$$ {~}^{(1,1)}o_{k}^{(1)} = \sum\limits_{j = 1}^{n_{2}} {~}^{(1,1)}w_{k j}^{(1)} {~}^{(1,1)}v_{j}^{(2)}. $$

(20)

The error is computed by

$$ {~}^{(1,1)}E = \sum\limits_{k = 1}^{n_{1}} \left( x_{k} - {~}^{(1,1)}o_{k}^{(1)} \right)^{2}. $$

(21)

Then, the number of epochs is increased up to the T th epoch (T, 1) in the first step. For the final T th epoch, the importance of the first step is computed by

$$ {~}^{(T,1)}u_{j k}^{(2)}= \left| {~}^{(T,1)}w_{j k}^{(2)} \right|. $$

(22)

The normalized importance is

$$ {~}^{(T,1)}z_{j k}^{(2)}= { {{~}^{(T,1)}u_{j k}^{(2) } \over { {~}^{(T,1)}{u}_{\max}^{(2)}}}}. $$

(23)

Then, the importance in the second step in Fig. 11b is used to modify connection weights of the first step in Fig. 11a as

$$ {~}^{(1,2)}w_{j k}^{(2)}= {~}^{(T,1)}z_{j k}^{(2)} {~}^{(T,1)}w_{j k}^{(2)}. $$

(24)

Gradually, connection weights are modified to take into account the effect of importance of the connection weights. As can be seen in Fig. 11a–c, gradually, the number of strong connection weights diminishes. Finally, in the final T th epoch of the N th step in Fig. 11c3, the number of strong connection weights decreases to the final point.