Ultra2Speech - A Deep Learning Framework for Formant Frequency Estimation and Tracking from Ultrasound Tongue Images

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12263)


Thousands of individuals need surgical removal of their larynx due to critical diseases every year and therefore, require an alternative form of communication to articulate speech sounds after the loss of their voice box. This work addresses the articulatory-to-acoustic mapping problem based on ultrasound (US) tongue images for the development of a silent-speech interface (SSI) that can provide them with an assistance in their daily interactions. Our approach targets automatically extracting tongue movement information by selecting an optimal feature set from US images and mapping these features to the acoustic space. We use a novel deep learning architecture to map US tongue images from the US probe placed beneath a subject’s chin to formants that we call, Ultrasound2Formant (U2F) Net. It uses hybrid spatio-temporal 3D convolutions followed by feature shuffling, for the estimation and tracking of vowel formants from US images. The formant values are then utilized to synthesize continuous time-varying vowel trajectories, via Klatt Synthesizer. Our best model achieves R-squared (\(R^2\)) measure of 99.96% for the regression task. Our network lays the foundation for an SSI as it successfully tracks the tongue contour automatically as an internal representation without any explicit annotation.


Silent speech interface Ultrasound tongue contour Formant Spatio-temporal feature Deep neural network Articulatory-to-acoustics 



This work was funded by the Natural Sciences and Engineering Research Council (NSERC) of Canada and Canadian Institutes for Health Research (CIHR).


  1. 1.
    Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)Google Scholar
  2. 2.
    Csapó, T.G., Grósz, T., Gosztolya, G., Tóth, L., Markó, A.: DNN-based ultrasound-to-speech conversion for a silent speech interface. Proc. Interspeech 2017, 3672–3676 (2017)CrossRefGoogle Scholar
  3. 3.
    Dahl, G.E., Sainath, T.N., Hinton, G.E.: Improving deep neural networks for LVCSR using rectified linear units and dropout. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8609–8613. IEEE (2013)Google Scholar
  4. 4.
    Denby, B., Oussar, Y., Dreyfus, G., Stone, M.: Prospects for a silent speech interface using ultrasound imaging. In: Proceedings of the 2006 IEEE International Conference on Acoustics Speech and Signal Processing, vol. 1, p. I. IEEE (2006)Google Scholar
  5. 5.
    Denby, B., Schultz, T., Honda, K., Hueber, T., Gilbert, J.M., Brumberg, J.S.: Silent speech interfaces. Speech Commun. 52(4), 270–287 (2010)CrossRefGoogle Scholar
  6. 6.
    Denby, B., Stone, M.: Speech synthesis from real time ultrasound images of the tongue. In: 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 1, pp. I–685. IEEE (2004)Google Scholar
  7. 7.
    Gilbert, M., et al.: Restoring speech following total removal of the larynx by a learned transformation from sensor data to acoustics. J. Acoust. Soc. Am. 141(3), EL307–EL313 (2017)CrossRefGoogle Scholar
  8. 8.
    Gosztolya, G., Pintér, Á., Tóth, L., Grósz, T., Markó, A., Csapó, T.G.: Autoencoder-based articulatory-to-acoustic mapping for ultrasound silent speech interfaces. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2019)Google Scholar
  9. 9.
    Hueber, T., et al.: Eigentongue feature extraction for an ultrasound-based silent speech interface. In: 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP 2007, vol. 1, pp. I–1245. IEEE (2007)Google Scholar
  10. 10.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
  11. 11.
    Juanpere, E.M., Csapó, T.G.: Ultrasound-based silent speech interface using convolutional and recurrent neural networks. Acta Acust. United Acust. 105(4), 587–590 (2019)CrossRefGoogle Scholar
  12. 12.
    Klatt, D.H.: Software for a cascade/parallel formant synthesizer. J. Acoust. Soc. Am. 67(3), 971–995 (1980)CrossRefGoogle Scholar
  13. 13.
    Luo, C., Yuille, A.L.: Grouped spatial-temporal aggregation for efficient action recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5512–5521 (2019)Google Scholar
  14. 14.
    Mandal, M., Kumar, L.K., Saran, M.S., et al.: MotionRec: a unified deep framework for moving object recognition. In: The IEEE Winter Conference on Applications of Computer Vision, pp. 2734–2743 (2020)Google Scholar
  15. 15.
    O’Shaughnessy, D.: Formant estimation and tracking. In: Benesty, J., Sondhi, M.M., Huang, Y.A. (eds.) Springer Handbook of Speech Processing. SH, pp. 213–228. Springer, Heidelberg (2008). Scholar
  16. 16.
    Saha, P., Srungarapu, P., Fels, S.: Towards automatic speech identification from vocal tract shape dynamics in real-time MRI. Proc. Interspeech 2018, 1249–1253 (2018)CrossRefGoogle Scholar
  17. 17.
    Stavness, I., Lloyd, J.E., Fels, S.: Automatic prediction of tongue muscle activations using a finite element model. J. Biomech. 45(16), 2841–2848 (2012)CrossRefGoogle Scholar
  18. 18.
    Tóth, L., Gosztolya, G., Grósz, T., Markó, A., Csapó, T.G.: Multi-task learning of speech recognition and speech synthesis parameters for ultrasound-based silent speech interfaces. In: Interspeech, pp. 3172–3176 (2018)Google Scholar
  19. 19.
    Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6450–6459 (2018)Google Scholar
  20. 20.
    Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Electrical and Computer Engineering DepartmentUniversity of British ColumbiaVancouverCanada
  2. 2.Department of LinguisticsUniversity of British ColumbiaVancouverCanada
  3. 3.Haskins LaboratoriesNew HavenUSA

Personalised recommendations