Journal on Multimodal User Interfaces

, Volume 13, Issue 2, pp 89–97 | Cite as

Investigating the effects of modality switches on driver distraction and interaction efficiency in the car

  • Florian RoiderEmail author
  • Sonja Rümelin
  • Bastian Pfleging
  • Tom Gross
Original Paper


In the last decade, the number and variety of secondary tasks in modern vehicles has grown exponentially. To address this variety, drivers can choose between alternative input modalities to complete each task in the most adequate way. However, the process of switching between different modalities might cause increased cognitive effort and finally result in a loss of efficiency. Therefore, the effects of switching between input modalities have to be examined in detail. We present a user study with 18 participants that investigates these effects when switching between touch and speech input on task efficiency and driver distraction in a dual-task setup. Our results show that the sequential combination of adequate modalities for subtasks did not affect task completion times and thus reduced the duration of the entire interaction. We argue to promote modality switches and discuss the implications on application areas beyond the automotive context.


Interaction efficiency Driver distraction Modality switch costs Touch input Speech input Multimodal interaction 



  1. 1.
    Bradford JH, James H (1995) The human factors of speech-based interfaces. ACM SIGCHI Bull 27(2):61–67. CrossRefGoogle Scholar
  2. 2.
    Card SK, Moran TP, Newell A (1980) The keystroke-level model for user performance time with interactive systems. Commun ACM 23(7):396–410. CrossRefGoogle Scholar
  3. 3.
    Gondan M, Lange K, Rösler F, Röder B (2004) The redundant target effect is affected by modality switch costs. Psychon Bull Rev 11(2):307–313CrossRefGoogle Scholar
  4. 4.
    Harbluk JL, Noy YI, Trbovich PL, Eizenman M (2007) An on-road assessment of cognitive distraction: impacts on drivers visual behavior and braking performance. Accid Anal Prev 39(2):372–379. CrossRefGoogle Scholar
  5. 5.
    Koons DB, Sparrell CJ, Thorisson KR (1993) Integrating simultaneous input from speech, gaze, and hand gestures. In: Maybury MT (ed) Intelligent multimedia interfaces. AAAI Press, Menlo Park, pp 257–276Google Scholar
  6. 6.
    Mitrevska M, Moniri MM, Nesselrath R, Schwartz T, Feld M, Korber Y, Deru M, Muller C (2015) SiAM—situation-adaptive multimodal interaction for innovative mobility concepts of the future. In: Proceedings of the international conference on intelligent environments—IE ’15. IEEE, pp 180–183.
  7. 7.
    Monsell S (2003) Task switching. Trends Cogn Sci 7(3):134–140. CrossRefGoogle Scholar
  8. 8.
    Müller C, Weinberg G, Vetro A (2011) Multimodal input in the car, today and tomorrow. IEEE Multimed 18(1):98–103. CrossRefGoogle Scholar
  9. 9.
    Ohn-Bar E, Trivedi MM (2014) Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans Intell Transp Syst 15(6):2368–2377. CrossRefGoogle Scholar
  10. 10.
    Oviatt S (2012) Multimodal interfaces. In: Jacko JA (ed) The human–computer interaction handbook: fundamentals, evolving technologies and emerging applications, 14th edn. CRC Press, Boca Raton, pp 286–304Google Scholar
  11. 11.
    Oviatt S, Coulston R, Lunsford R (2004) When do we interact multimodally? Cognitive load and multimodal communication patterns. In: Proceedings of the 2004 international conference on multimodal interfaces—ICMI ’04, pp 129–136.
  12. 12.
    Petzoldt T, Bellem H, Krems JF (2014) The critical tracking task: a potentially useful method to assess driver distraction? Hum Fact J Hum Fact Ergonom Soc 56(4):789–808. CrossRefGoogle Scholar
  13. 13.
    Pfleging B, Schneegass S, Schmidt A (2012) Multimodal interaction in the car—combining speech and gestures on the steering wheel. In: Proceedings of the 4th international conference on automotive user interfaces and interactive vehicular applications—AutomotiveUI ’12. ACM, New York, pp 0–7.
  14. 14.
    Pickering CA, Burnham KJ, Richardson MJ (2007) A research study of hand gesture recognition technologies and applications for human vehicle interaction. In: Institution of engineering and technology conference on automotive electronics, pp 1–15Google Scholar
  15. 15.
    Pickering CA, Burnham KKJ, Richardson MJM (2007) A review of automotive human machine interface technologies and techniques to reduce driver distraction. In: 2nd IET international conference on system safety. IEEE, pp 223–228.
  16. 16.
    Roider F, Rümelin S, Gross T (2018) Using visual cues to leverage the use of speech input in the vehicle. In: International conference on persuasive technology. Springer, Cham, pp 120–131.
  17. 17.
    Schneegaß S, Pfleging B, Kern D, Schmidt A (2011) Support for modeling interaction with automotive user interfaces. In: Proceedings of the 3rd international conference on automotive user interfaces and interactive vehicular applications—AutomotiveUI ’11, p 71.
  18. 18.
    Strayer DL, Watson JM, Drews FA (2011) Cognitive distraction while multitasking in the automobile. In: Federmeier KD (ed) Psychology of learning and motivation, vol 54. Elsevier, Amsterdam. Google Scholar
  19. 19.
    Wickens CD, Sandry DL, Vidulich M (1983) Compatibility and resource competition between modalities of input, central processing, and output. Hum Fact 25(2):227–248. CrossRefGoogle Scholar
  20. 20.
    Wierwille WW (1993) Demands on driver resources associated with introducing advanced technology into the vehicle. Transp Res Part C Emerg Technol 1(2):133–142. CrossRefGoogle Scholar
  21. 21.
    Zhang Y, Stellmach S, Sellen A (2015) The costs and benefits of combining gaze and hand gestures for remote. In: Abascal J, Barbosa S, Fetter M, Gross T, Palanque P, Winckler M (eds) Human–computer interaction—INTERACT 2015. INTERACT 2015. Lecture notes in computer science. Springer, Cham. Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.BMW Group Research, New Technologies InnovationGarching bei MünchenGermany
  2. 2.Universität BambergBambergGermany
  3. 3.Ludwig-Maximilians-Universität MünchenMunichGermany

Personalised recommendations