ICCHP Keynote: Recognizing Silent and Weak Speech Based on Electromyography

  • Tanja Schultz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6179)


In the past decade, the performance of automatic speech processing systems, including speech recognition, spoken language translation, and speech synthesis, has improved dramatically. This has resulted in an increasingly widespread use of speech and language technologies in a large variety of applications. However, speech-driven interfaces based on conventional acoustic speech signals still suffer from several limitations.


Speech Recognition Automatic Speech Recognition Vocal Tract Speech Synthesis Brain Computer Interface 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bartels, J.L., Andreasen, D., Ehirim, P., Mao, H., Seibert, S., Wright, E.J., Kennedy, P.R.: Neurotrophic electrode: method of assembly and implantation into human motor speech cortex. Journal of Neuroscience Methods 174(2), 168–176 (2008)CrossRefGoogle Scholar
  2. 2.
    Birbaumer, N.: The thought translation device (TTD) for completely paralyzed patients. IEEE Transactions on Rehabilitation Engineering 8(2), 190–193 (2000)CrossRefGoogle Scholar
  3. 3.
    Blankertz, B., Dornhege, G., Krauledat, M., Müller, K.-R., Kunzmann, V., Losch, F., Curio, G.: The Berlin brain-computer interface: EEG-based communication without subject training. IEEE Transactions on Neural Systems and Rehabilitation Engineering 14(2), 147–152 (2006)CrossRefGoogle Scholar
  4. 4.
    Brumberg, J.S., Nieto-Castanon, A., Kennedy, P.R., Guenther, F.H.: Brain-Computer Interfaces for Speech Communication. Speech Communication, Special Issue on Silent Speech Interfaces (April 2010) (in press)Google Scholar
  5. 5.
    Carstens: Carstens Medizinelektronik (2008), ?(accessed November 6, 2008)
  6. 6.
    Chan, A.D.C.: Multi-expert automatic speech recognition system using myoelectric signals, Ph.D. Dissertation, Department of Electrical and Computer Engineering, University of New Brunswick, Canada (2003)Google Scholar
  7. 7.
    DaSalla, C., Kambara, H., Sato, M., Koike, Y.: Personal communication on EEG classification of vowel speech imagery using common spatial patterns (2008)Google Scholar
  8. 8.
    Denby, B., Stone, M.: Speech synthesis from real time ultrasound images of the tongue. In: Proceedings IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2004), Montréal, Canada, May 17-21, vol. 1, pp. I-685 – I-I688 (2004)Google Scholar
  9. 9.
    Denby, B., Oussar, Y., Dreyfus, G., Stone, M.: Prospects for a Silent Speech Interface Using Ultrasound Imaging. In: IEEE ICASSP, Toulouse, France, pp. I365–I368 (2006)Google Scholar
  10. 10.
    Denby, B., Schultz, T., Honda, K., Hueber, T., Gilbert, J.: Silent Speech Interfaces. Speech Communication, Special Issue on Silent Speech Interfaces (April 2010) (in press)Google Scholar
  11. 11.
    Dornhege, G., del Millan, J.R., Hinterberger, T., McFarland, D., Müller, K.-R. (eds.): Towards brain-computer interfacing. MIT Press, Cambridge (2007)Google Scholar
  12. 12.
    Fagan, M.J., Ell, S.R., Gilbert, J.M., Sarrazin, E., Chapman, P.M.: Development of a (silent) speech recognition system for patients following laryngectomy. Medical Engineering and Physics 30(4), 419–425 (2008)CrossRefGoogle Scholar
  13. 13.
    Guenther, F.H., Ghosh, S.S., Tourville, J.A.: Neural Modeling and Imaging of the Cortical Interactions underlying Syllable Production. Brain and Language 96, 280–301 (2007)CrossRefGoogle Scholar
  14. 14.
    Heracleous, P., Kaino, T., Saruwatari, H., Shikano, K.: Unvoiced speech recognition using tissue-conductive acoustic sensor. EURASIP Journal on Advances in Signal Processing 2007(1), 1–11 (2007)CrossRefzbMATHGoogle Scholar
  15. 15.
    Hochberg, L.R., Simeral, J.D., Kim, S., Stein, J., Friehs, G.M., Black, M.J., Donoghue, J.P.: More than two years of intracortically-based cursor control via a neural interface system. In: Neurosicence Meeting Planner 2008, Program No. 673.15, Washington, DC (2008)Google Scholar
  16. 16.
    Hueber, T., Aversano, G., Chollet, G., Denby, B., Dreyfus, G., Oussar, Y., Roussel, P., Stone, M.: Eigentongue feature extraction for an ultrasound-based silent speech interface. In: IEEE ICASSP, Honolulu, vol. 1, pp. 1245–1248 (2007)Google Scholar
  17. 17.
    Hueber, T., Benaroya, E.-L., Chollet, G., Denby, B., Dreyfus, G., Stone, M.: Development of a silent speech interface driven by ultrasound and optical images of the tongue and lips. Speech Communication, Special Issue on Silent Speech Interfaces (April 2010) (in press)Google Scholar
  18. 18.
    Hummel, J., Figl, M., Birkfellner, W., Bax, M.R., Shahidi, R., Maurer, C.R., Bergmann, H.: Evaluation of a new electromagnetic tracking system using a standardized assessment protocol. Physics in Medicine and Biology 51, N205–N210 (2006)Google Scholar
  19. 19.
    Izzetoglu, K., Bunce, S., Onaral, B., Pourrezaei, K., Chance, B.: Functional Optical Brain Imaging Using Near-Infrared During Cognitive Tasks. International Journal of HCI 17(2), 211–227 (2004)Google Scholar
  20. 20.
    Jorgensen, C., Lee, D.D., Agabon, S.: Sub auditory speech recognition based on EMG signals. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN), pp. 3128–3133 (2003)Google Scholar
  21. 21.
    Jorgensen, C., Binsted, K.: Web browser control using EMG based sub vocal speech recognition. In: Proceedings of the 38th Annual Hawaii International Conference on System Sciences, pp. 294c.1–294c.8. IEEE, Los Alamitos (2005)Google Scholar
  22. 22.
    Jorgensen, C., Dusan, S.: Speech interfaces based upon surface electromyography. Speech Communication, Special Issue on Silent Speech Interfaces (April 2010) (in press)Google Scholar
  23. 23.
    Jou, S., Schultz, T., Walliczek, M., Kraft, F.: Towards continuous speech recognition using surface electromyography. In: INTERSPEECH 2006 and 9th International Conference on Spoken Language Processing, vol. 2, pp. 573–576 (2006)Google Scholar
  24. 24.
    Jou, S., Schultz, T., Waibel, A.: Multi-stream articulatory feature classifiers for surface electromyographic continuous speech recognition. In: Proceedings of International Conference on Acoustics, Speech, and Signal Processing. IEEE, Honolulu (2007)Google Scholar
  25. 25.
    Kennedy, P.R., Bakay, R.A.E., Moore, M.M., Adams, K., Goldwaithe, J.: Direct control of a computer from the human central nervous system. IEEE Transactions on Rehabilitation Engineering 8(2), 198–202 (2000)CrossRefGoogle Scholar
  26. 26.
    Maier-Hein, L., Metze, F., Schultz, T., Waibel, A.: Session independent non-audible speech recognition using surface electromyography. In: IEEE Workshop on Automatic Speech Recognition and Understanding, San Juan, Puerto Rico, pp. 331–336 (2005)Google Scholar
  27. 27.
    Manabe, H., Hiraiwa, A., Sugimura, T.: Unvoiced speech recognition using EMG-mime speech recognition. In: Proceedings of CHI, Human Factors in Computing Systems, Ft. Lauderdale, Florida, pp. 794–795 (2003)Google Scholar
  28. 28.
    Manabe, H., Zhang, Z.: Multi-stream HMM for EMG-based speech recognition. In: Proceedings of 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Francisco, California, September 1-5, vol. 2, pp. 4389–4392 (2004)Google Scholar
  29. 29.
    Nakajima, Y., Kashioka, H., Shikano, K., Campbell, N.: Non-audible murmur recognition input interface using stethoscopic microphone attached to the skin. In: Proceedings of IEEE ICASSP, pp. 708–711 (2003)Google Scholar
  30. 30.
    Nakajima, Y., Kashioka, H., Campbell, N., Shikano, K.: Non-audible murmur (NAM) recognition. IEICE Transactions on Information and Systems E89-D(1), 1–8 (2006)CrossRefGoogle Scholar
  31. 31.
    Ng, L., Burnett, G., Holzrichter, J., Gable, T.: Denoising of human speech using combined acoustic and EM sensor signal processing. In: Proc. Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), Istanbul, Turkey, June 5-9, vol. 1, pp. 229–232 (2000)Google Scholar
  32. 32.
    Porbadnigk, A., Wester, M., Calliess, J., Schultz, T.: EEG-based speech recognition - impact of temporal effects. In: Biosignals 2009, Porto, Portugal, pp. 376–381 (January 2009)Google Scholar
  33. 33.
    Quatieri, T.F., Messing, D., Brady, K., Campbell, W.B., Campbell, J.P., Brandstein, M., Weinstein, C.J., Tardelli, J.D., Gatewood, P.D.: Exploiting nonacoustic sensors for speech enhancement. IEEE Transactions on Audio, Speech, and Language Processing 14(2), 533–544 (2006)CrossRefGoogle Scholar
  34. 34.
    Schönle, P.W., Gräbe, K., Wenig, P., Höhne, J., Schrader, J., Conrad, B.: Electromagnetic articulography: Use of alternating magnetic fields for tracking movements of multiple points inside and outside the vocal tract. Brain and Language 31, 26–35 (1987)CrossRefGoogle Scholar
  35. 35.
    Schultz, T., Wand, M.: Modeling coarticulation in large vocabulary EMG-based speech recognition, Speech Communication. Special Issue on Silent Speech Interfaces (April 2010) (in press)Google Scholar
  36. 36.
    Suppes, P., Lu, Z.-L., Han, B.: Brain wave recognition of words. Proceedings of the National Academy of Scientists of the USA 94, 14965–14969 (1997)CrossRefGoogle Scholar
  37. 37.
    Tardelli, J.D. (ed.): MIT Lincoln Labs report ESC-TR-2004-084, Pilot Corpus for Multisensor Speech Processing (2004)Google Scholar
  38. 38.
    Titze, I.R., Story, B.H., Burnett, G.C., Holzrichter, J.F., Ng, L.C., Lea, W.A.: Comparison between electroglottography and electromagnetic glottography. Journal of the Acoustical Society of America 107(1), 581–588 (2000)CrossRefGoogle Scholar
  39. 39.
    Tran, V.-A., Bailly, G., Loevenbruck, H., Toda, T.: Improvement to a NAM-captured whisper-to-speech system. Speech Communication, Special Issue on Silent Speech Interfaces (April 2010) (in press)Google Scholar
  40. 40.
    Walliczek, M., Kraft, F., Jou, S.-C., Schultz, T., Waibel, A.: Sub-word unit based non-audible speech recognition using surface electromyography. In: Proceedings of Interspeech, Pittsburgh, USA, pp. 1487–1490 (2006)Google Scholar
  41. 41.
    Wand, M., Schultz, T.: Towards speaker-adaptive speech recognition based on surface electromyography. In: Proceedings of Biosignals, Porto, Portugal (2009) (in press)Google Scholar
  42. 42.
    Wester, M., Schultz, T.: Unspoken speech - speech recognition based on electroencephalography, Master’s thesis, Universität Karlsruhe (TH), Karlsruhe, Germany (2006)Google Scholar
  43. 43.
    Wolpaw, J.R., Birbaumer, N., McFarland, D., Pfurtscheller, G., Vaughan, T.: Brain-computer interfaces for communication and control. Clinical Neurophysiology 113(6), 767–791 (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Tanja Schultz
    • 1
  1. 1.Cognitive Systems LabKarlsruhe Institute of TechnologyKarlsruheGermany

Personalised recommendations