Intelligent Multi-Modal Recognition Interface Using Voice-XML and Embedded KSSL Recognizer

  • Jung-Hyun Kim
  • Kwang-Seok Hong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4251)


A desktop PC and wire communications net-based traditional studies on pattern recognition and multimodal interaction have some restrictions (e.g. limitation of motion, conditionality in space and so on) and general problems according to using of the vision technologies for recognition and representation of the haptic-gesture information. In this paper, we propose and implement Multi-Modal Recognition Interface (hereinafter, MMRI) integrating speech using Voice-XML and gesture based on wireless networks, it have purposes that recognizes and represents the Korean Standard Sign Language (hereinafter, KSSL) which is a dialog system and interactive elements in the Korean deaf communities, and the need to dialogue with deaf person in their own language, sign language, is well recognized and is widely accepted as being a positive influence on communication. The advantages of our approach are as follows: 1) it improves efficiency of the MMRI input module according to the technology of wireless communication, 2) it shows higher recognition performance than uni-modal recognition system 3) it recognizes and represents continuous sign language of users with flexibility in real time and offer to user a wider range of personalized and differentiated information using the MMRI more effectively. Experimental results, the MMRI deduces an average recognition rate of 96.23% for significant, dynamic and continuous the KSSL and speech of various users.


Recognition Rate Speech Recognition Sign Language Gesture Recognition Hand Gesture 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Stéphane, H.M., et al.: Multimodal Interaction Requirements. W3C Note (2003),
  2. 2.
    Barnett, J., et al.: Multimodal Interaction Activity-Multimodal Architecture and Interfaces. W3C Working Draft (2005),
  3. 3.
    Jang, H.-Y., Kim, D.-J., Kim, J.-B., Bien, Z.-N.: A Study on Hand-Signal Recognition System in 3-Dimensional Space. Journal of IEEK 2004-41CI-3-11. IEEK (2004)Google Scholar
  4. 4.
    Use of Signs in Hearing Communities,
  5. 5.
    Kim, S.-G.: Standardization of Signed Korean. Journal of KSSE 9. KSSE (1992)Google Scholar
  6. 6.
    Kim, S.-G.: Korean Standard Sign Language Tutor, 1st edn. Osung Publishing Company, Seoul (2000)Google Scholar
  7. 7.
    5DT Data Glove 5 Manual and FASTRAK® Data Sheet,
  8. 8.
    Kim, J.-H., Kim, D.-G., Shin, J.-H., Lee, S.-W., Hong, K.-S.: Hand Gesture Recognition System using Fuzzy Algorithm and RDBMS for Post PC. In: Wang, L., Jin, Y. (eds.) FSKD 2005. LNCS (LNAI), vol. 3614, Springer, Heidelberg (2005)Google Scholar
  9. 9.
    Chen, C.H.: Fuzzy Logic and Neural Network Handbook, 1st edn. McGraw-Hill, New York (1992)Google Scholar
  10. 10.
    McGlashan, S., et al.: Voice Extensible Markup Language (VoiceXML) Version 2.0. W3C Recommendation (1992),

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jung-Hyun Kim
    • 1
  • Kwang-Seok Hong
    • 1
  1. 1.School of Information and Communication EngineeringSungkyunkwan UniversitySuwonKorea

Personalised recommendations