Model-Based, Multimodal Interaction in Document Browsing

  • Parisa Eslambolchilar
  • Roderick Murray-Smith
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4299)


In this paper we introduce a dynamic system approach to the design of multimodal interactive systems. We use an example where we support human behavior in browsing a document, by adapting the dynamics of navigation and the visual feedback (using a focus-in-context (F+C) method) to support the current inferred task. We also demonstrate non-speech audio feedback, based on a language model. We argue that to design interaction we need models of key aspects of the process, here for example, we need models for the dynamic system, language model and sonification. We show how the user’s intention is coupled to the visualization technique via the dynamic model, and how the focus-in-context method couples details in context to audio samples via the language identification system. We present probabilistic audio feedback as an example of a multimodal approach to sensing different languages in a multilingual text. This general approach is well suited to mobile and wearable applications, and shared displays.


Language Model Continuous Interaction Multimodal Interaction Reading Mode Audio Sample 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    McCullough, M.: Abstract Craft: Practical Digital Hand. The MIT Press, Cambridge (1998)Google Scholar
  2. 2.
    Kelley, C.R.: Manual and Automatic Control. John Wiley and Sons, Inc., New York (1968)Google Scholar
  3. 3.
    Beaudouin-Lafon, M.: Designing Interaction, not Interfaces. In: AVI 2004. Proceedings of the working conference on Advanced visual interfaces, pp. 15–22 (2004)Google Scholar
  4. 4.
    Doherty, G., Massink, M.: Continuous Interaction and Human Control. In: Alty, J. (ed.) Proceedings of the XVIII European Annual Conference on Human Decision Making and Manual Control, pp. 80–96 (1999)Google Scholar
  5. 5.
    Faconti, G., Massink, M.: Continuous interaction with computers: Issues and Requirements. In: Stephanidis, C. (ed.) Proceedings of Universal Access in HCI, vol. 3. Lawrence Erlbaum Associates, Mahwah (2001)Google Scholar
  6. 6.
    Bederson, B.B.: Fisheye Menus. In: UIST 2000. Proceedings of the 13th annual ACM symposium on User interface software and technology, pp. 217–225 (2000)Google Scholar
  7. 7.
    Furnas, G.: Generalized Fisheye Views. In: Proceedings of CHI 1986, pp. 16–23 (1986)Google Scholar
  8. 8.
    Lamping, J., Rao, R., Pirolli, P.: A focus+context technique based on hyperbolic geobreak metry for visualizing large hierarchies. In: Proceedings of CHI 1995, pp. 401–408 (1995)Google Scholar
  9. 9.
    Mackinlay, J.D., Robertson, G.G., Card, C.K.: The Perspective Wall: Detail and Context Smoothly Integrated. In: Proceedings of CHI 1991, pp. 173–179 (1991)Google Scholar
  10. 10.
    Sarkar, M., Brown, M.H.: Graphical fisheye views of graphs. In: Bauersfeld, P., Bennett, J., Lynch, G. (eds.) Human Factors in Computing Systems, CHI 1992 Conference Proceedings: Striking A Balance, pp. 83–91. ACM Press, New York (1992)CrossRefGoogle Scholar
  11. 11.
    Carpendale, M.S.T.: A Framework for Elastic Presentation Space. Ph.D thesis, Department of Computing Science, Simon Fraser University, Canada (1999)Google Scholar
  12. 12.
    Preece, J., Rogers, Y., Sharp, H.: Interaction Design: Beyond Human Computer Interaction. John Wiley, Chichester (2002)Google Scholar
  13. 13.
    Sheridan, T.B., Ferrell, W.R.: Man-Machine Systems: Information, Control, and Decision Models of Human Performance. MIT Press, Cambridge (1974)Google Scholar
  14. 14.
    Eslambolchilar, P., Murray-Smith, R.: Tilt-based Automatic Zooming and Scaling in mobile devices-a state-space implementation. In: Dunlop, M.D. (ed.) Mobile HCI 2004. LNCS, vol. 3160, pp. 120–131. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  15. 15.
    Gutwin, C.: Improving focus targeting in interactive fisheye views. In: Proceeding of CHI 2002, pp. 267–274 (2002)Google Scholar
  16. 16.
    Powers, W.T.: Living Control Systems: Selected papers of Powers, W.T. The Control Systems Group Book (1989)Google Scholar
  17. 17.
    Powers, W.T.: Living Control Systems II: Selected papers of Powers, W.T. The Control Systems Group Book (1992)Google Scholar
  18. 18.
    Tischler, M.B.: Advances in Aircraft flight Control. Taylor & Francis, Abington (1994)Google Scholar
  19. 19.
    Bell, T., Cleary, J., Witten, I.: Text Compression. Prentice Hall Advanced Reference Series. Prentice-Hall, Englewood Cliffs (1990)Google Scholar
  20. 20.
    Williamson, J., Murray-Smith, R.: Dynamics and probabilistic text entry. In: Murray-Smith, R., Shorten, R. (eds.) Switching and Learning 2004. LNCS, vol. 3355, pp. 333–342. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  21. 21.
    Lesher, G., Rinkus, G.: Leveraging word prediction to improve character prediction in a scanning configuration. In: Proceedings of the RESNA 2002, Annual Conference (2002)Google Scholar
  22. 22.
    Carpendale, S., Montagnese, C.: A framework for unifying presentation space. In: Proceedings of UIST 2001, pp. 82–92 (2001)Google Scholar
  23. 23.
    Eslambochilar, P., Williamson, J., Murray-Smith, R.: Multimodal feedback for tilt controlled speed dependent automatic zooming. In: UIST 2004. Proceedings of the 17th annual ACM symposium on User interface software and technology. ACM, New York (2004)Google Scholar
  24. 24.
    Williamson, J., Murray-Smith, R.: Sonification of probabilistic feedback through granular synthesis. IEEE Multimedia 12(2), 45–52 (2005)CrossRefGoogle Scholar
  25. 25.
    Shannon, C.E.: A mathematical theory of communication. Bell System Technical Journal 27, 379–423, 623–656 (1948), zbMATHMathSciNetGoogle Scholar
  26. 26.
    Foley, J., Dam, A.V., Feiner, S., Hughes, J.F.: Computer Graphics, reissued 2nd edn. Addison-Wesley, Reading (1995)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Parisa Eslambolchilar
    • 1
  • Roderick Murray-Smith
    • 1
    • 2
  1. 1.Hamilton InstituteNational University of IrelandMaynoothIreland
  2. 2.Department of Computing ScienceGlasgow UniversityGlasgowScotland

Personalised recommendations