A System for Matching Mathematical Formulas Spoken during a Lecture with Those Displayed on the Screen for Use in Remote Transcription

  • Yoshinori Takeuchi
  • Hironori Kawaguchi
  • Noboru Ohnishi
  • Daisuke Wakatsuki
  • Hiroki Minagawa
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7382)

Abstract

A system is described for extracting and matching mathematical formulas presented orally during a lecture with those simultaneously displayed on the lecture room screen. Each mathematical formula spoken by the lecturer and displayed on the screen is extracted and shown to the transcriber. Investigation showed that, in a lecture in which many mathematical formulas were presented, about 80% of them were both spoken and pointed to on the screen, meaning that the system can help a transcriber correctly transcribe up to 80% of the formulas presented. A speech recognition system is used to extract the formulas from the lecturer’s speech, and a system that analyzes the trajectory of the end of the stick pointer is used to extract the formulas from the projected images. This information is combined and used to match the pointed-to formulas with the spoken ones. In testing using actual lectures, this system extracted and matched 71.4% of the mathematical formulas both spoken and displayed and presented them for transcription with a precision of 89.4%.

Keywords

Speech Recognition Automatic Speech Recognition Mathematical Formula Video Signal Speech Recognition System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Kato, N., Kawano, S., Miyoshi, S., Nishioka, T., Murakami, H., Minagawa, H., Wakatsuki, D., Shirasawa, M., Ishihara, Y., Naito, I.: Subjective Evaluation of Displaying Keywords for Speech to Text Service Operators. The Transactions of Human Interface Society 9(2), 195–203 (2007) (in Japanese)Google Scholar
  2. 2.
    Miyoshi, S., Kawano, S., Nishioka, T., Kato, N., Shirasawa, M., Murakami, H., Minagawa, H., Ishihara, Y., Naito, I., Wakatsuki, D., Kuroki, H., Kobayashi, M.: A Basic Study on Supplementary Visual Information for Real-Time Captionists in the Lecture of Information Science. IEICE Transactions on Information and Systems (Japanese Edition) J91-D(9), 2236–2246 (2008)Google Scholar
  3. 3.
    Miyoshi, S., Kuroki, H., Kawano, S., Shirasawa, M., Ishihara, Y., Kobayashi, M.: Support Technique for Real-Time Captionist to Use Speech Recognition Software. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A.I. (eds.) ICCHP 2008. LNCS, vol. 5105, pp. 647–650. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  4. 4.
    Wald, M., Bain, K.: Universal access to communication and learning: role of automatic speech recognition. Universal Access in the Information Society 6(4), 435–447 (2007)CrossRefGoogle Scholar
  5. 5.
    Sakiyama, T., Mukunoki, M., Katsuo, I.: Detection of the Indicated Area with an Indication Stick. In: Int. Conf. on Multimodal Interfaces, pp. 480–487 (2000)Google Scholar
  6. 6.
    Marutani, T., Nishiguchi, S., Kakusho, K., Minoh, M.: Making a lecture content with deictic information about indicated objects in lecture materials. In: AEARU Workshop on Network Education, pp. 70–75 (2005)Google Scholar
  7. 7.
    Takeuchi, Y., Saito, K., Ito, A., Ohnishi, N., Iizuka, S., Nakajima, S.: Extracting Pointing Object with Demonstrative Speech Phrase for Remote Transcription in Lecture. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A.I. (eds.) ICCHP 2008. LNCS, vol. 5105, pp. 624–631. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  8. 8.
    Ito, A., Saito, K., Takeuchi, Y., Ohnishi, N., Iizuka, S., Nakajima, S.: A Study on Demonstrative Words Extraction in Instructor Utterance on Communication Support for Hearing Impaired Persons. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A.I. (eds.) ICCHP 2008. LNCS, vol. 5105, pp. 632–639. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Takeuchi, Y., Ohta, H., Ohnishi, N., Wakatsuki, D., Minagawa, H.: Extraction of Displayed Objects Corresponding to Demonstrative Words for Use in Remote Transcription. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A.I. (eds.) ICCHP 2010. LNCS, vol. 6180, pp. 152–159. Springer, Heidelberg (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Yoshinori Takeuchi
    • 1
  • Hironori Kawaguchi
    • 2
  • Noboru Ohnishi
    • 2
  • Daisuke Wakatsuki
    • 3
  • Hiroki Minagawa
    • 3
  1. 1.Department of Information Systems, School of InformaticsDaido UniversityMinami-kuJapan
  2. 2.Graduate School of Information ScienceNagoya UniversityChikusa-kuJapan
  3. 3.Tsukuba University of TechnologyJapan

Personalised recommendations