K-Means Clustering Optimizing Deep Stacked Sparse Autoencoder

Abstract

Because of the large structure and long training time, the development cycle of the common depth model is prolonged. How to speed up training is a problem deserving of study. In order to accelerate training, K-means clustering optimizing deep stacked sparse autoencoder (K-means sparse SAE) is presented in this paper. First, the input features are divided into K small subsets by K-means clustering, then each subset is input into corresponding autoencoder model for training, which only has fewer nodes in the hidden layer than traditional models. After training, each autoencoder’s trained weights and biases is merged to obtain the next layer’s input features by feedforward network. The above steps are repeated till the softmax layer, then fine-tuning is carried out. Using MNIST-Rotation datasets to train the network that has three hidden layers and each layer has 800 nodes, the improved model has higher classification accuracy and shorter training time when K = 10. With K increasing, the training time is reduced to almost the same as the fine-tuning time but the recognition ability is descended. Compared with the recently stacked denoising sparse autoencoder, the recognition accuracy is improved by 1%, not only the noise factor is not selected but also the training speed is significantly increased. The trained filters from the improved model is also used to train convolutional autoencoder, and it performs better than traditional models. We find that pre-training stage doesn’t need large samples simultaneously, and small samples parallel training reduces the probability of falling into the local minimum.