Research on Visual Speech Recognition Based on Local Binary Pattern and Stacked Sparse Autoencoder

  • Yuanyao LuEmail author
  • Ke Gu
  • Shan He
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 876)


Lip feature extraction from human mouth image plays an essential role in visual speech recognition applications. This paper presents a lip feature extraction algorithm based on Local Binary Patterns (LBP) and Stacked Sparse Autoencoders (SSAE). First, LBP texture features are extracted from lip images. Then SSAE uses greedy unsupervised learning to extract high-level features. At last, we improve the performance of overall system by fine-tuning and input the extracted features into the Softmax classifier. Compared with traditional methods, the model proposed in this paper has higher classification accuracy and more applicability.


Visual speech recognition Local Binary Pattern Stacked Sparse Autoencoder 



The research was supported by the National Natural Science Foundation of China (61571013), by the Beijing Natural Science Foundation of China (4143061). The authors thank all the partners and the participants in the experiment for their help, by the Science and Technology Development Program of Beijing Municipal Education Commission (KM201710009003) and by the Great Wall Scholar Reserved Talent Program of North China University of Technology (NCUT2017XN018013).


  1. 1.
    Turk, M., Pentland, A.: Face recognition using eigenfaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–591. IEEE Computer Society Press, Maui (1991)Google Scholar
  2. 2.
    Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigen-faces vs fisherfaces: recognition using class specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 711–720 (1997)CrossRefGoogle Scholar
  3. 3.
    Huang, G.B., Lee, H., Learned-Miller, E.: Learning hierarchical representations for face verification with convolutional deep belief networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2518–2525. IEEE, Piscataway (2012)Google Scholar
  4. 4.
    Chen, D., Cao, X., Wen, F., et al.: Blessing of dimensionality: high-dimensional feature and its efficient compression for face verification. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3025–3032. IEEE, Piscataway (2013)Google Scholar
  5. 5.
    Ojala, T., Pietikäinen, M., Harwood, D.: Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. In: Proceedings of the 12th IAPR International Conference on Pattern Recognition, vol. 1, pp. 582–585 (1994)Google Scholar
  6. 6.
    Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on feature distributions. Pattern Recognit. 29, 51–59 (1996)CrossRefGoogle Scholar
  7. 7.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face recognition with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)CrossRefGoogle Scholar
  8. 8.
    Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)CrossRefGoogle Scholar
  9. 9.
    Ian, G., Courville, A., Bengion, Y.: Large scale feature learning with spike and slab sparse coding. In: International Conference on Machine Learning, pp. 1439–1446. Edinburgh, Scotland (2012)Google Scholar
  10. 10.
    Yang, P., Shan, S., Gao, W., et a1.: Face recognition using AdaBoosted gabor features. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 356–361. IEEE Computer Society (2004)Google Scholar
  11. 11.
    Chen, J., Shan, S., He, C., et al.: WLD: a robust local image descriptor. IEEE Trans. Pattern Anal. Mach. Intell. 32(9), 1705–1720 (2010)CrossRefGoogle Scholar
  12. 12.
    Itamar, A., Dereck, C.R., Thomas, P.K.: Deep machine learning-a new frontier in artificial intelligence research. IEEE Comput. Intell. Mag. 5(4), 13–18 (2010)CrossRefGoogle Scholar
  13. 13.
    Bengio, L.Y.: Learning deep architectures for AI. Found. trends Mach. Learn. 2(1), 1–127 (2009)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504 (2006)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Bengio, Y., LeCun, Y.: Scaling learning algorithms towards AI. In: Bottou, L., Chapelle, O., DeCoste, D., Weston, J. (eds.) Large-Scale Kernel Machines. MIT Press, Cambridge (2007)Google Scholar
  16. 16.
    Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 11, 625–660 (2010)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of Electronic and Information EngineeringNorth China University of TechnologyBeijingChina

Personalised recommendations