Skip to main content
SpringerLink
Log in

An Experimental Analysis of Different Approaches to Audio–Visual Speech Recognition and Lip-Reading

  • Conference paper
  • First Online:
Proceedings of 15th International Conference on Electromechanics and Robotics "Zavalishin's Readings"

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 187))

Abstract

In this paper, we have analyzed different approaches to audio–visual speech recognition. We mainly focused on testing different modalities fusion techniques, rather than other parts of AVSR (e.g., feature extraction methods). Tree audio–visual modalities integration methods were under consideration, namely GMM-CHMM, DNN-HMM and end-to-end approaches, defined as the most promising and commonly found in scientific literature. The testing was performed on two different datasets: on GRID corpus for the English language and on HAVRUS corpus for the Russian. Obtained results once again confirms the superiority of neural network approaches compared to the others in conditions when we have enough data to effectively train NN models, which was demonstrated by our experiments on the GRID dataset. On a more compact in size HAVRUS database, the best recognition results were demonstrated by the traditional GMM-CHMM approach. This paper presents our vision on current state of audio–visual speech recognition field and possible directions for the further research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Katsaggelos, K., Bahaadini, S., Molina, R.: Audiovisual fusion: challenges and new approaches. Proc. IEEE 103(9), 1635–1653 (2015)

    Article  Google Scholar 

  2. McGurk, H., MacDonald, J.: Hearing lips and seeing voices. Nature 264, 746–748 (1976)

    Article  Google Scholar 

  3. Ivanko, D., et al.: Multimodal speech recognition: increasing accuracy using high speed video data. J. Multimodal User Interfaces 12(4), 319–328 (2018)

    Article  Google Scholar 

  4. Davis, S.: Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences [Text]. Proces.s IEEE Trans. Acoust. Speech Signal 28(4), 357–366 (1980)

    Article  Google Scholar 

  5. Hermansky, H.: Perceptual linear predictive (PLP) analysis of speech [Text]. J. Acoust. Soc. Am. 87(4), 1738–1752 (1990)

    Article  Google Scholar 

  6. Bundy, A., Wallen, L.: Linear predictive coding. In: Catalogue of Artificial Intelligence. Springer-Verlag (1984)

    Google Scholar 

  7. Bahaadini, S., Asaei, A., Imseng, D., Bourlard, H.: Posterior-based sparse representation for automatic speech recognition. In: Proceeding of Interspeech, pp. 2454–2458 (2014)

    Google Scholar 

  8. Konar, A., Chakraborty, A.: Emotion Recognition: A Pattern Analysis Approach. Wiley (2015)

    Google Scholar 

  9. Hong, X., Yao, H., Wan, Y., Chen, R.: A PCA Based visual DCT feature extraction method for lip-reading. In: Proceedings International Conference Intelligent Information Hiding Multimedia, Signal Process, pp. 321–326 (2006)

    Google Scholar 

  10. Ivanko, D., Ryumin, D., Kipyatkova, I., Axyonov, A., Karpov, A.: Lip-Reading Using pixel-based and geometry-based features for multimodal human–robot interfaces. In: Proceedings of 14th International Conference on Electromechanics and Robotics “Zavalishin’s Readings”, pp. 477–486 (2020)

    Google Scholar 

  11. Cetingul, H., Yemez, Y., Erzin, E., Tekalp, A.: Discriminative analysis of lip motion features for speaker identification and speech-reading. Proc. IEEE Trans. Image Process. 15(10), 2879–2891 (2006)

    Article  Google Scholar 

  12. Ivanko, D., Ryumin, D., Axyonov, A., Zelezny, M.: Designing advanced geometric features for automatic Russian visual speech recognition. In: Proceedings of International Conference on Speech and Computer, pp. 245–254 (2018)

    Google Scholar 

  13. Ngiam, J. et al.: Multimodal deep learning. In: Proceedings of 28th international Conference of Machine Learning, pp. 689–696 (2011)

    Google Scholar 

  14. Chetty, G., Wagner., M.: Audio-visual multimodal fusion for biometric person authentication and liveness verification. In: Proceedings NICTA-HCSNet Multimodal user interaction workshop, vol. 57, pp. 17–24 (2006)

    Google Scholar 

  15. Atrey, P.K., Hossain, M.A., Saddik, E., Kankanhalli, M.S.: Multimodal fusion for multimedia analysis: A survey. Multimedia Syst. 16(6), 345–379 (2010)

    Article  Google Scholar 

  16. Xu, H., Chua, T.S.: Fusion of AV features and external information sources for event detection in team sport video. ACM Trans. Multimedia Comput. Commun. Appl. 2(1), 44–67 (2006)

    Article  Google Scholar 

  17. Snoek, C.G., Worring, M., Smeulders, A.W.: Early versus late fusion in semantic video analysis. In: Proceedings 13th Annual ACM International Conference on Multimedia, pp. 399–402 (2005)

    Google Scholar 

  18. Wu, Z., Cai, L., Meng, H.: Multi-level fusion of audio and visual features for speaker identification. In: Advances in Biometrics, pp. 493–499 (2005)

    Google Scholar 

  19. Ivanko, D., et al.: Using a high-speed video Camera for robust audio-visual speech recognition in acoustically noisy conditions. SPECOM 2017, 757–766 (2017)

    Google Scholar 

  20. Chung, J.S., Zisserman, A.: Lip reading in the wild. In: Computer Vision—ACCV 2016. pp. 87–103 (2016)

    Google Scholar 

  21. He, J., Zhang, H.: Lipreading recognition based on svm and dtak. In: 2010 4th International Conference on Bioinformatics and Biomedical Engineering, pp. 1–4 (2010)

    Google Scholar 

  22. Hinton, G., et al.: IEEE signal process. Mag. 29(6), 82–97 (2012)

    Google Scholar 

  23. Assael, Y.M., Shillingford, B., Whiteson, S., Freitas, N. LipNet: End-to-End Sentence-level Lipreading. In: arXiv:1611.01599, pp. 1–13 (2016)

  24. Cooke, M., Barker, J., Cunningham, S., Shao, X.: An audio-visual corpus for speech perception and automatic speech recognition. J. Acoust. Soc. Am. 120(5), 2421–2424 (2006)

    Article  Google Scholar 

  25. Verkhodanova, V., Ronzhin, A., Kipyatkova, I., Ivanko, D., Karpov, A., Zelezny, M.: HAVRUS corpus: high-Speed recordings of audio-visual Russian speech. In: International Conference on Speech and Computer, pp. 338–345 (2016)

    Google Scholar 

  26. Ivanko, D. et al.: Using a high-speed video Camera for robust audio-visual speech recognition in acoustically noisy conditions. In: International Conference on Speech and Computer, pp. 757–766 (2017)

    Google Scholar 

  27. Meutzner, H., Ma, N., Nickel, R., Schymura, C., Kolossa, D.: Improving audio-visual speech recognition using deep neural networks with dynamic stream reliability estimates. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5320–5324 (2017)

    Google Scholar 

  28. Markovnikov, N., Kipyatkova, I.: An analytic survey of end-to-end speech recognition systems. SPIIRAS Proc. 58(3), 77–110 (2018)

    Article  Google Scholar 

Download references

Acknowledgements

This research is supported by the RFBR projects No. 19-29-09081 (Section 3) and No. 18-37-00306 (Sections 2 and 4), as well as state research No. 0073-2019-0005.

Author information

Authors and Affiliations

  1. St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, St. Petersburg, Russia

    Denis Ivanko, Dmitry Ryumin & Alexey Karpov

Authors
  1. Denis Ivanko
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dmitry Ryumin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexey Karpov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Denis Ivanko .

Editor information

Editors and Affiliations

  1. St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, St. Petersburg, Russia

    Andrey Ronzhin

  2. Saint Petersburg State University of Aerospace Instrumentation, St. Petersburg, Russia

    Vladislav Shishlakov

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ivanko, D., Ryumin, D., Karpov, A. (2021). An Experimental Analysis of Different Approaches to Audio–Visual Speech Recognition and Lip-Reading. In: Ronzhin, A., Shishlakov, V. (eds) Proceedings of 15th International Conference on Electromechanics and Robotics "Zavalishin's Readings". Smart Innovation, Systems and Technologies, vol 187. Springer, Singapore. https://doi.org/10.1007/978-981-15-5580-0_16

Download citation

Publish with us

Policies and ethics

Search

Navigation