Abstract
Storytelling by a child, as a training activity, influences significantly a child’s linguistic abilities, logic and thought process, imagination, and creativity. There presently exist many software-based storytelling applications. However, most are deemed incompatible or not suitable to Chinese children. Due to the limited vocabulary of pre-school and lower-level grade school children, speech-based and pen-based input models are considered the most effective way of input. But now there is not an effective multimodal mode to solve the problem for children’s storytelling systems. In this paper, we propose a multimodal fusion framework that utilizes pen and speech techniques to incorporate both context information and linguistic attributes of the Chinese language into the design. Based on the proposed framework, we formulated specific methods of integration, and developed a prototype for our proposed system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Wright, A.: Creating stories with children. Oxford University Press, England (1995)
Benford, S., Bederson, B., et al.: Designing Storytelling Technologies to Encourage Collaboration Between Young Children. In: Proceedings of CHI 2000, pp. 556–563 (2000)
Montemayor, J., Druin, A., et al.: Physical Programming: Designing Tools for Children to Create Physical Interactive Environments. In: CHI 2002, ACM Conference on Human Factors in Computing Systems, CHI Letters, vol. 4(1), pp. 299–306 (2002)
Bers, M., Cassell, J.: Storytelling Systems: Constructing the Innerface of the Interface. In: Cognitive Technologies Procedings 1997, pp. 98–108. IEEE, Los Alamitos (1997)
Cohen, P.R., Johnston, M., et al.: Quickset: Multimodal interaction for distributed applications. In: Proceedings of ACM Multimedia, Seattle, WA, pp. 31–40 (1997)
Oviatt, S.L.: Multimodal Interfaces. In: Jacko, J., Sears, A. (eds.) Handbook of Human-Computer Interface. Lawrence Erlbaum, New Jersey (2002)
Pfleger, N.: Context Based Multimodal Fusion. In: Proceedings of ICMI 2004, State College, Pennsylvania, USA, pp. 265–272 (2004)
Neal, J.G., Thielman, C.Y., Dobes, A., Haller, S.M., Shapiro, S.C.: Natural language with integrated deictic and graphic gestures. In: Maybury, M.T., Wahlster, W. (eds.) Readings In Intelligent User Interfaces, pp. 38–51. Morgan Kaufmann Publishers, San Francisco (1991)
Campana, E., Baldridge, J., Dowding, J., Hockey, B.A., Remington, R.W., Stone, L.S.: Using eye movements to determine referents in a spoken dialogue system. In: Proceedings. of workshop on perceptive user interface, Orland, Florida (2001)
Zhang, Q., Imamiya, A., Go, K., Mao, X.A.: Gaze and Speech Multimodal Interface. In: Proceedings of the 24th International Conference on Distributed Computing Systems Workshops (ICDCSW 2004) (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wang, D., Zhang, J., Dai, G. (2006). A Multimodal Fusion Framework for Children’s Storytelling Systems. In: Pan, Z., Aylett, R., Diener, H., Jin, X., Göbel, S., Li, L. (eds) Technologies for E-Learning and Digital Entertainment. Edutainment 2006. Lecture Notes in Computer Science, vol 3942. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11736639_71
Download citation
DOI: https://doi.org/10.1007/11736639_71
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-33423-1
Online ISBN: 978-3-540-33424-8
eBook Packages: Computer ScienceComputer Science (R0)