Designing Multi-modal Interaction – A Basic Operations Approach

  • I. C. MariAnne KarlssonEmail author
  • Fredrick Ekman
  • Mikael Johansson
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 827)


A basic operations approach to designing multimodal user interfaces has been explored in two experimental studies. By basic operations is here meant the most fundamental operations that make up a(ny) interaction task. Study A investigated if certain basic operations have a preferred modal counterpart and execution and Study B what happens when several basic operations are combined into tasks. The results suggest interdependencies between basic operations and specific modalities but also indicate that users may prefer unimodal interaction or to choose modality as desired. Hence, further studies are needed to assess the feasibility of a basic operations approach to multimodal interaction design.


Interface design Multi-modal interaction System input 


  1. Althoff F, McGlaun G, Lang M, Rigoll G (2004) Evaluating multimodal interaction patterns in various application scenarios. In: Camurri A, Volpe G (eds) Gesture-based communication in human-computer interaction. Lecture notes in computer science, vol 2915. Springer, Heidelberg, pp 421–435Google Scholar
  2. Carrino F et al (2012) In-vehicle natural interaction based on electromyography. In: Adjunct proceedings of the AutomotiveUI 2012. ACM, New York, pp 23–24Google Scholar
  3. Chen X, Tremaine M (2005) Multimodal user input patterns in a non-visual context. In: Proceedings of the 7th international ACM SIGACCESS conference on computers and accessibility, pp 206–207Google Scholar
  4. Hausen D, Richter H, Hemme A, Butz A (2013) Comparing input modalities for peripheral interaction: a case study on peripheral music control. In: Kotzé P, Marsden G, Lindgaard G, Wesson J, Winckler M (eds) Human-computer interaction – INTERACT 2013. Lecture notes in computer science, vol 8119. Springer, Heidelberg, pp 162–179Google Scholar
  5. Janowski K, Kistler F, André E (2013) Gestures or Speech? Comparing modality selection for different interaction tasks in a virtual environment. In: Proceedings of the Tilburg gesture research meetingGoogle Scholar
  6. Krahnstoever N, Kettebekov S, Yeasin M, Sharma R (2002) A real-time framework for natural multimodal interaction with large screen displays. In: Proceedings of the 4th IEEE international conference on multimodal interfaces, Washington D.C. IEEE Computer Society Press, pp 349–354Google Scholar
  7. Muller C, Weinberg G (2011) Multimodal input in the car, today and tomorrow. IEEE Multimed 18(1):98–103CrossRefGoogle Scholar
  8. Naumann AB, Wechsung I, Möller S (2008) Factors influencing modality choice in multimodal applications. In: André E, Dybkjær L, Minker W, Neumann H, Pieraccini R, Weber M (eds) Perception in multimodal dialogue systems. PIT 2008. Lecture notes in computer science, vol 5078. Springer, Heidelberg, pp 37–43Google Scholar
  9. Oviatt S (1999) Ten myths of multimodal interaction. Commun ACM 42(11):74–81CrossRefGoogle Scholar
  10. Oviatt S, Coulston R, Lunsford R (2004) When do we interact multimodally? Cognitive load and multimodal communication patterns. In: Proceedings of the 6th International Conference on Multimodal Interfaces. ACM, New York, pp 129–136Google Scholar
  11. Oviatt S et al (2003) Toward a theory of organized multimodal integration patterns during human-computer interaction. In: Proceedings of the 5th international conference on multimodal interfaces. ACM, New York, pp 44–51Google Scholar
  12. Shi Y, Taib R, Ruiz N, Choi E, Chen F (2007) Multimodal human-machine interface and user cognitive load measurement. IFAC Proc Vol 40(16):200–205CrossRefGoogle Scholar
  13. Tang L, Lundgren J (2016) Exploring multimodality and natural interaction in a driving context. Master Thesis, Chalmers University of Technology, GothenburgGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Design & Human Factors, Department of Industrial and Materials ScienceChalmers University of TechnologyGothenburgSweden

Personalised recommendations