Abstract
Multimodal interfaces development history is reviewed briefly in the introduction. Some applications of multimodal interfaces to education software for disabled people are presented. One of them, the LipMouse is a novel, vision-based human-computer interface that tracks user’s lip movements and detect lips gestures. A new approach to diagnosing Parkinson’s disease is also shown. The progression of the disease can be measured employing the UPDRS (Unified Parkinson Disease Rating Scale) scale which is used to evaluate motor and behavioral symptoms of the Parkinson’s disease, based on the multimodal interface called Virtual-Touchpad (VTP) used for supporting medical diagnosis. The scent emitting multimodal computer interface provides an important supplement of the polysensoric stimulation process, playing an essential role in education and therapy of children with certain developmental disorders. The Smart Pen providing a tool for supporting therapy of developmental dyslexia is presented and results achieved with its application are discussed. The eye-gaze tracking system named Cyber Eye, developed at the Multimedia Systems Department employed to many kinds of experiments is presented including analysis of visual activity of patients remaining in vegetative state and their awareness evaluation. The paper is concluded with some general remarks concerning the role of multimodal computer interfaces applied to learning, therapy and everyday usage of computerized devices.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Czyżewski, A., Odya, P., Smulko, J., Lentka, G., Kostek, B., Kotarski, M.: Scent emitting multimodal computer interface for learning enhancement. In: 4th International Workshop on Management and Interaction with Multimodal Information Content, Bilbao, Spain, pp. 142–146 (2010)
Dalka, P., Czyżewski, A.: Lip movement and gesture recognition for a multimodal human-computer interface. In: Proc. International Multiconference on Computer Science and Information Technology, 2nd International Symposium on Multimedia – Applications and Processing, Mrągowo, Poland, pp. 365–369 (2009)
Dalka, P., Czyżewski, A.: Human-computer interface based on visual lip movement and gesture recognition. Intern. J. Comp. Sci. and Appl. 3(7), 124–139 (2010)
Frostig, M., Miller, A.M., Horne, D.: The developmental program in visual perception: beginning pictures and patterns. Follett Publishing Company, Chicago (1964)
Kosikowski, Ł., Czyżewski, A.: Computer Based System for Strabismus and Amblyopia Therapy. In: Proc. International Multiconference on Computer Science and Information Technology, 2nd International Symposium on Multimedia – Applications and Processing, Mrągowo, Poland, pp. 493–496 (2009)
Kosikowski, Ł., Czyżewski, A.: Binocular vision impairments therapy supported by contactless eye-gaze tracking system. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2010, Part II. LNCS, vol. 6180, pp. 373–376. Springer, Heidelberg (2010)
Kunka, B., Kostek, B.: Exploiting audio-visual correlation by means of gaze tracking. Int. J. Comp. Sci. and Appl. 7(3), 104–123 (2010)
Kunka, B., Kostek, B.: Objectivization of audio-visual correlation analysis. Arch. Acoust. 37(1), 63–72 (2012)
Kunka, B., Czyżewski, A., Kwiatkowska, A.: Awareness evaluation of patients in vegetative state employing eye-gaze tracking system. Int. J. Artif. Intell. Tools 21(2), 1–11 (2012)
Kupryjanow, A., Czyżewski, A.: Methods of improving speech intelligibility for listeners with hearing resolution deficit. Diagn. Pathol. 7, 129 (2012)
Kupryjanow, A., Kosikowski, Ł., Odya, P., Czyżewski, A.: Auditory-visual attention stimulator. In: Proc 134th AES Convention, Rome, Italy, Preprint no. 8810 (2013)
Monti, M.M., Laureys, S., Owen, A.M.: The vegetative state. BMJ 341, c3765 (2010)
Odya, P., Czyżewski, A., Grabkowska, A., Grabkowski, M.: Smart pen - new multimodal computer control tool for graphomotorical therapy. Intell. Decis. Technol. J. 4(3), 197–209 (2010)
Ponsen, M.M., Stofers, D., Booij, J.: Idiopathic hyposomia as a preclinical sign of Parkinson’s disease. Ann. Neurol. 56, 173–181 (2004)
Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: Proc. of IEEE International Conference on Neural Networks, San Francisco, USA, pp. 586–591 (1993)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Czyżewski, A., Dalka, P., Kosikowski, Ł., Kunka, B., Odya, P. (2014). Multimodal Human-Computer Interfaces Based on Advanced Video and Audio Analysis. In: Hippe, Z., Kulikowski, J., Mroczek, T., Wtorek, J. (eds) Human-Computer Systems Interaction: Backgrounds and Applications 3. Advances in Intelligent Systems and Computing, vol 300. Springer, Cham. https://doi.org/10.1007/978-3-319-08491-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-08491-6_8
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-08490-9
Online ISBN: 978-3-319-08491-6
eBook Packages: EngineeringEngineering (R0)