LipsID Using 3D Convolutional Neural Networks

  • Miroslav HlaváčEmail author
  • Ivan Gruber
  • Miloš Železný
  • Alexey Karpov
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11096)


This paper presents a proposition for a method inspired by iVectors for improvement of visual speech recognition in the similar way iVectors are used to improve the recognition rate of audio speech recognition. A neural network for feature extraction is presented with training parameters and evaluation. The network is trained as a classifier for a closed set of 64 speakers from the UWB-HSCAVC dataset and then the last softmax fully connected layer is removed to gain a feature vector of size 256. The network is provided with sequences of 15 frames and outputs the softmax classification to 64 classes. The training data consists of approximately 20000 sequences of grayscale images from the first 50 sentences that are common to every speaker. The network is then evaluated on the 60000 sequences created from 150 sentences from each speaker. The testing sentences are different for each speaker.


Visual speech Neural network 3D convolution Deep features 



This work was supported by the Ministry of Education of the Czech Republic, project No. LTARF18017. The work has been also supported by the grant of the University of West Bohemia, project No. SGS-2016-039. This work was supported by the Government of the Russian Federation (grant No. 08-08) and the Russian Foundation for Basic Research (project No. 18-07-01407) too. Moreover, access to computing and storage facilities owned by parties and projects contributing to the National Grid Infrastructure MetaCentrum provided under the programme “Projects of Large Research, Development, and Innovations Infrastructures” (CESNET LM2015042), is greatly appreciated.


  1. 1.
    Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). software available from
  2. 2.
    Assael, Y.M., Shillingford, B., Whiteson, S., de Freitas, N.: Lipnet: Sentence-level lipreading. arXiv preprint arXiv:1611.01599 (2016)
  3. 3.
    Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Incremental face alignment in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1859–1866 (2014)Google Scholar
  4. 4.
    Chollet, F., et al.: Keras: Deep learning library for theano and tensorflow, vol. 7, p. 8 (2015).
  5. 5.
    Chung, J.S., Senior, A.W., Vinyals, O., Zisserman, A.: Lip reading sentences in the wild. CoRR abs/1611.05358 (2016).
  6. 6.
    Chung, J., Zisserman, A.: Lip reading in the wild. In: Asian Conference on Computer Vision (2016)Google Scholar
  7. 7.
    Císař, P., Železnỳ, M., Krňoul, Z., Kanis, J., Zelinka, J., Müller, L.: Design and recording of czech speech corpus for audio-visual continuous speech recognition. In: Proceedings of the Auditory-Visual Speech Processing International Conference 2005 (2005)Google Scholar
  8. 8.
    Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: continual prediction with LSTM (1999)Google Scholar
  9. 9.
    Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376. ACM (2006)Google Scholar
  10. 10.
    Ji, S., Xu, W., Yang, M., Yu, K.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2013)CrossRefGoogle Scholar
  11. 11.
    Saon, G., Soltau, H., Nahamoo, D., Picheny, M.: Speaker adaptation of neural network acoustic models using i-vectors. In: ASRU, pp. 55–59 (2013)Google Scholar
  12. 12.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Miroslav Hlaváč
    • 1
    • 2
    • 3
    Email author
  • Ivan Gruber
    • 1
    • 2
    • 3
  • Miloš Železný
    • 1
  • Alexey Karpov
    • 3
    • 4
  1. 1.Faculty of Applied Sciences, Department of CyberneticsUWBPilsenCzech Republic
  2. 2.Faculty of Applied Sciences, NTISUWBPilsenCzech Republic
  3. 3.ITMO UniversitySt. PetersburgRussia
  4. 4.SPIIRASSt. PetersburgRussia

Personalised recommendations