MMSDS: Ubiquitous Computing and WWW-Based Multi-modal Sentential Dialog System

  • Jung-Hyun Kim
  • Kwang-Seok Hong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4096)


In this study, we suggest and implement Multi-Modal Sentential Dialog System (MMSDS) integrating 2 sensory channels with speech and haptic information based on ubiquitous computing and WWW for clear communication. The importance and necessity of MMSDS for HCI as following: 1) it can allow more interactive and natural communication functions between the hearing-impaired and hearing person without special learning and education, 2) according as it recognizes a sentential Korean Standard Sign Language (KSSL) which is represented with speech and haptics and then translates recognition results into a synthetic speech and visual illustration in real-time, it may provide a wider range of personalized and differentiated information more effectively to them, and 3) above all things, a user need not be constrained by the limitations of a particular interaction mode at any given moment because it can guarantee mobility of WPS (Wearable Personal Station for the post PC) with a built-in sentential sign language recognizer. In experiment results, while an average recognition rate of uni-modal recognizer using KSSL only is 93.1% and speech only is 95.5%, advanced MMSDS deduced an average recognition rate of 96.1% for 32 sentential KSSL recognition models.


Ubiquitous Computing Hand Gesture Fuzzy Relation Average Recognition Rate Data Glove 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Slavík, P., Hejda, P., Fuchs, M.: Architecture of Multi-modal Dialogue System. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2000. LNCS (LNAI), vol. 1902, pp. 433–438. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  2. 2.
    Fitrianie, S., et al.: A Multi-modal Eliza Using Natural Language Processing and Emotion Recognition. In: Matoušek, V., Mautner, P. (eds.) TSD 2003. LNCS (LNAI), vol. 2807, pp. 394–399. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  3. 3.
    Jiangqin, W., et al.: A Simple Sign Language Recognition System Based on Data Glove. In: ICSP 1998, IEEE International Conference Proceedings, pp. 1257–1260 (1998)Google Scholar
  4. 4.
    Borgo, S.: A Multi-agent Modal Language for Concurrency with Non-communicating Agents. In: Mařík, V., Müller, J.P., Pěchouček, M. (eds.) CEEMAS 2003. LNCS (LNAI), vol. 2691, Springer, Heidelberg (2003)CrossRefGoogle Scholar
  5. 5.
    Berti, S., Paternò, F.: Development of Multi-modal Interfaces in Multi-device Environments. In: Costabile, M.F., Paternó, F. (eds.) INTERACT 2005. LNCS, vol. 3585, pp. 1067–1070. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  6. 6.
    Barnard, M., et al.: Multi-Modal Audio-Visual Event Recognition for Football Analysis. In: IEEE XI11 Workshop on Neural Networks for Signal Processing, IEEE Workshop Proceedings, pp. 469–478 (2003)Google Scholar
  7. 7.
    Use of Signs in Hearing Communities:
  8. 8.
    Kim, S.-G.: Korean Standard Sign Language Tutor, 1st edn. Osung Publishing Company, Seoul (2000)Google Scholar
  9. 9.
    Kim, J.-H., et al.: Hand Gesture Recognition System using Fuzzy Algorithm and RDBMS for Post PC. In: Wang, L., Jin, Y. (eds.) FSKD 2005. LNCS (LNAI), vol. 3614, pp. 170–175. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. 10.
    5DT Data Glove 5 Manual and FASTRAK® Data Sheet:
  11. 11.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. Wiley, New York (2001)MATHGoogle Scholar
  12. 12.
    Paulus, D., Hornegger, J.: Applied Pattern Recognition, 2nd edn. Vieweg (1998)Google Scholar
  13. 13.
    Schuermann, J.: Pattern Classification: A Unified View of Statistical and Neural Approaches. Wiley & Sons, Chichester (1996)Google Scholar
  14. 14.
    Relational DataBase Management System:
  15. 15.
    Oracle 10g DW Guide:
  16. 16.
    Vasantha kandasamy, W.B.: Smaranda Fuzzy Algebra. American Research Press, Seattle (2003)Google Scholar
  17. 17.
    McGlashan, S., et al.: Voice Extensible Markup Language (VoiceXML) Version 2.0. W3C Recommendation (1992),

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jung-Hyun Kim
    • 1
  • Kwang-Seok Hong
    • 1
  1. 1.School of Information and Communication EngineeringSungkyunkwan UniversitySuwon, KyungKi-doKorea

Personalised recommendations