Abstract
In this paper we introduce a dynamic system approach to the design of multimodal interactive systems. We use an example where we support human behavior in browsing a document, by adapting the dynamics of navigation and the visual feedback (using a focus-in-context (F+C) method) to support the current inferred task. We also demonstrate non-speech audio feedback, based on a language model. We argue that to design interaction we need models of key aspects of the process, here for example, we need models for the dynamic system, language model and sonification. We show how the user’s intention is coupled to the visualization technique via the dynamic model, and how the focus-in-context method couples details in context to audio samples via the language identification system. We present probabilistic audio feedback as an example of a multimodal approach to sensing different languages in a multilingual text. This general approach is well suited to mobile and wearable applications, and shared displays.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
McCullough, M.: Abstract Craft: Practical Digital Hand. The MIT Press, Cambridge (1998)
Kelley, C.R.: Manual and Automatic Control. John Wiley and Sons, Inc., New York (1968)
Beaudouin-Lafon, M.: Designing Interaction, not Interfaces. In: AVI 2004. Proceedings of the working conference on Advanced visual interfaces, pp. 15–22 (2004)
Doherty, G., Massink, M.: Continuous Interaction and Human Control. In: Alty, J. (ed.) Proceedings of the XVIII European Annual Conference on Human Decision Making and Manual Control, pp. 80–96 (1999)
Faconti, G., Massink, M.: Continuous interaction with computers: Issues and Requirements. In: Stephanidis, C. (ed.) Proceedings of Universal Access in HCI, vol. 3. Lawrence Erlbaum Associates, Mahwah (2001)
Bederson, B.B.: Fisheye Menus. In: UIST 2000. Proceedings of the 13th annual ACM symposium on User interface software and technology, pp. 217–225 (2000)
Furnas, G.: Generalized Fisheye Views. In: Proceedings of CHI 1986, pp. 16–23 (1986)
Lamping, J., Rao, R., Pirolli, P.: A focus+context technique based on hyperbolic geobreak metry for visualizing large hierarchies. In: Proceedings of CHI 1995, pp. 401–408 (1995)
Mackinlay, J.D., Robertson, G.G., Card, C.K.: The Perspective Wall: Detail and Context Smoothly Integrated. In: Proceedings of CHI 1991, pp. 173–179 (1991)
Sarkar, M., Brown, M.H.: Graphical fisheye views of graphs. In: Bauersfeld, P., Bennett, J., Lynch, G. (eds.) Human Factors in Computing Systems, CHI 1992 Conference Proceedings: Striking A Balance, pp. 83–91. ACM Press, New York (1992)
Carpendale, M.S.T.: A Framework for Elastic Presentation Space. Ph.D thesis, Department of Computing Science, Simon Fraser University, Canada (1999)
Preece, J., Rogers, Y., Sharp, H.: Interaction Design: Beyond Human Computer Interaction. John Wiley, Chichester (2002)
Sheridan, T.B., Ferrell, W.R.: Man-Machine Systems: Information, Control, and Decision Models of Human Performance. MIT Press, Cambridge (1974)
Eslambolchilar, P., Murray-Smith, R.: Tilt-based Automatic Zooming and Scaling in mobile devices-a state-space implementation. In: Dunlop, M.D. (ed.) Mobile HCI 2004. LNCS, vol. 3160, pp. 120–131. Springer, Heidelberg (2004)
Gutwin, C.: Improving focus targeting in interactive fisheye views. In: Proceeding of CHI 2002, pp. 267–274 (2002)
Powers, W.T.: Living Control Systems: Selected papers of Powers, W.T. The Control Systems Group Book (1989)
Powers, W.T.: Living Control Systems II: Selected papers of Powers, W.T. The Control Systems Group Book (1992)
Tischler, M.B.: Advances in Aircraft flight Control. Taylor & Francis, Abington (1994)
Bell, T., Cleary, J., Witten, I.: Text Compression. Prentice Hall Advanced Reference Series. Prentice-Hall, Englewood Cliffs (1990)
Williamson, J., Murray-Smith, R.: Dynamics and probabilistic text entry. In: Murray-Smith, R., Shorten, R. (eds.) Switching and Learning 2004. LNCS, vol. 3355, pp. 333–342. Springer, Heidelberg (2005)
Lesher, G., Rinkus, G.: Leveraging word prediction to improve character prediction in a scanning configuration. In: Proceedings of the RESNA 2002, Annual Conference (2002)
Carpendale, S., Montagnese, C.: A framework for unifying presentation space. In: Proceedings of UIST 2001, pp. 82–92 (2001)
Eslambochilar, P., Williamson, J., Murray-Smith, R.: Multimodal feedback for tilt controlled speed dependent automatic zooming. In: UIST 2004. Proceedings of the 17th annual ACM symposium on User interface software and technology. ACM, New York (2004)
Williamson, J., Murray-Smith, R.: Sonification of probabilistic feedback through granular synthesis. IEEE Multimedia 12(2), 45–52 (2005)
Shannon, C.E.: A mathematical theory of communication. Bell System Technical Journal 27, 379–423, 623–656 (1948), http://cm.bell-labs.com/cm/ms/what/shannonday/paper.html
Foley, J., Dam, A.V., Feiner, S., Hughes, J.F.: Computer Graphics, reissued 2nd edn. Addison-Wesley, Reading (1995)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Eslambolchilar, P., Murray-Smith, R. (2006). Model-Based, Multimodal Interaction in Document Browsing. In: Renals, S., Bengio, S., Fiscus, J.G. (eds) Machine Learning for Multimodal Interaction. MLMI 2006. Lecture Notes in Computer Science, vol 4299. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11965152_1
Download citation
DOI: https://doi.org/10.1007/11965152_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-69267-6
Online ISBN: 978-3-540-69268-3
eBook Packages: Computer ScienceComputer Science (R0)