Advertisement

Journal on Multimodal User Interfaces

, Volume 13, Issue 2, pp 71–88 | Cite as

Visualizing natural language interaction for conversational in-vehicle information systems to minimize driver distraction

  • Michael BraunEmail author
  • Nora Broy
  • Bastian Pfleging
  • Florian Alt
Original Paper
  • 100 Downloads

Abstract

In this paper we investigate how natural language interfaces can be integrated with cars in a way such that their influence on driving performance is being minimized. In particular, we focus on how speech-based interaction can be supported through a visualization of the conversation. Our work is motivated by the fact that speech interfaces (like Alexa, Siri, Cortana, etc.) are increasingly finding their way into our everyday life. We expect such interfaces to become commonplace in vehicles in the future. Cars are a challenging environment, since speech interaction here is a secondary task that should not negatively affect the primary task, that is driving. At the outset of our work, we identify the design space for such interfaces. We then compare different visualization concepts in a driving simulator study with 64 participants. Our results yield that (1) text summaries support drivers in recalling information and enhances user experience but can also increase distraction, (2) the use of keywords minimizes cognitive load and influence on driving performance, and (3) the use of icons increases the attractiveness of the interface.

Keywords

Human–computer interaction Natural language interfaces Automotive user interfaces 

Notes

References

  1. 1.
    Asif A, Boll S (2010) Where to turn my car?: comparison of a tactile display and a conventional car navigation system under high load condition. In: Proceedings of the 2nd international conference on automotive user interfaces and interactive vehicular applications, Automotiveui ’10. ACM, New York, pp 64–71.  https://doi.org/10.1145/1969773.1969786
  2. 2.
    Ballagas R, Borchers J, Rohs M, Sheridan JG (2006) The smart phone: a ubiquitous input device. IEEE Pervasive Comput 5(1):70–77.  https://doi.org/10.1109/MPRV.2006.18 CrossRefGoogle Scholar
  3. 3.
    Beanland V, Fitzharris M, Young KL, Lenné MG (2013) Driver inattention and driver distraction in serious casualty crashes: Data from the australian national crash in-depth study. Accid Anal Prev 54:99–107.  https://doi.org/10.1016/j.aap.2012.12.043 http://www.sciencedirect.com/science/article/pii/S000145751300047X
  4. 4.
    Bellegarda JR (2014) Spoken language understanding for natural interaction: the siri experience. In: Natural interaction with robots, knowbots and smartphones: putting spoken dialog systems into practice. Springer, New York, pp 3–14.  https://doi.org/10.1007/978-1-4614-8280-2
  5. 5.
    Braun M, Pfleging B, Alt F (2018) A survey to understand emotional situations on the road and what they mean for affective automotive uis. Multimodal Technol. Interact 2(4).  https://doi.org/10.3390/mti2040075 http://www.mdpi.com/2414-4088/2/4/75
  6. 6.
    Braun M, Pfleging B, Broy N, Alt F (2017) A design space for conversational in-vehicle information systems. In: Proceedings of the 19th international conference on human–computer interaction with mobile devices and services, MobileHCI ’17. ACM, New YorkGoogle Scholar
  7. 7.
    Braun M, Völkel ST, Wiegand G, Puls T, Steidl D, WeißD, Alt F (2018) The smile is the new like: controlling music with facial expressions to minimize driver distraction. In: Proceedings of the 17th international conference on mobile and ubiquitous multimedia, MUM ’18. ACM, New York.  https://doi.org/10.1145/3282894.3289729
  8. 8.
    Bubb H (2003) Fahrerassistenz—primär ein beitrag zum komfort oder fuer die sicherheit? In: Der Fahrer im 21. Jahrhundert: Anforderungen, Anwendungen, Aspekte für Mensch-Maschine-Systeme; Tagung Braunschweig, 2. und 3. Juni 2003, VDI-Berichte 1768. VDI-Verlag, Düsseldorf, pp 25–44Google Scholar
  9. 9.
    Caird JK, Willness CR, Steel P, Scialfa C (2008) A meta-analysis of the effects of cell phones on driver performance. Accid Anal Prev 40(4):1282–1293CrossRefGoogle Scholar
  10. 10.
    Commission of the European Communities (2007) Commission recommendation of 22 december 2006 on safe and efficient in-vehicle information and communication systems: Update of the european statement of principles on human machine interface (2007/78/ec). Off J Eur Union 50(L32): 200–241. http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2007.032.01.0200.01.ENG
  11. 11.
    Cooper JM, Strayer DL (2015) Mental workload of voice interactions with 6 real-world driver interfaces. In: Proceedings of the eighth international driving symposium on human factors in driver assessment, training, and vehicle design, DA ’15, pp 2–8. http://drivingassessment.uiowa.edu/sites/default/files/DA2015/papers/002.pdf
  12. 12.
    Department of Transportation, National Highway Traffic Safety Administration (2013) Visual-manual nhtsa driver distraction guidelines for in-vehicle electronic device. Federal Regist 78(81), pp 24, 817–24, 890. https://federalregister.gov/a/2013-09883
  13. 13.
    Ehsani J, Li K, Simons-Morton BG (2015) Teenage drivers portable electronic device use while driving. In: Proceedings of the eighth international driving symposium on human factors in driver assessment, training, and vehicle design, DA ’15, pp 219–225. http://drivingassessment.uiowa.edu/sites/default/files/DA2015/papers/034.pdf
  14. 14.
    Engström J, Johansson E, Östlund J (2005) Effects of visual and cognitive load in real and simulated motorway driving. Transp Res Part F: Traffic Psychol Behav 8(2):97–120.  https://doi.org/10.1016/j.trf.2005.04.012 http://www.sciencedirect.com/science/article/pii/S1369847805000185. The relationship between distraction and driving performance: towards a test regime for in-vehicle information systems In-vehicle information systems
  15. 15.
    Eyben F, Wöllmer M, Schuller B (2010) Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the 18th ACM international conference on Multimedia. ACM, pp 1459–1462Google Scholar
  16. 16.
    Foley JP, Young R, Angell L, Domeyer JE (2013) Towards operationalizing driver distraction. In: Proceedings of the seventh international driving symposium on human factors in driver assessment, training, and vehicle design, pp 57–63. http://drivingassessment.uiowa.edu/sites/default/files/DA2013/Papers/010_Foley_0.pdf
  17. 17.
    Geiger M, Zobl M, Bengler K, Lang M (2001) Intermodal differences in distraction effects while controlling automotive user interfaces. In: Proceedings of HCI international 2001 volume 1: usability evaluation and interface design: cognitive engineering, intelligent agents and virtual reality, HCII ’01. Lawrence Erlbaum, HillsdaleGoogle Scholar
  18. 18.
    Green P (2013) Standard definitions for driving measures and statistics: overview and status of recommended practice j2944. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’13. ACM, New York, pp 184–191.  https://doi.org/10.1145/2516540.2516542
  19. 19.
    Green PA (2008) Driver interface/hmi standards to minimize driver distraction/overload. In: Convergence 2008 conference proceedings. Society of Automotive Engineers, SAE International, Warrendale (2008)Google Scholar
  20. 20.
    Group ADFTW (2006) Statement of principles, criteria and verification procedures on driver interactions with advanced in- vehicle information and communication systems. Technical report, Alliance of Automobile ManufacturersGoogle Scholar
  21. 21.
    Hackenberg L, Bongartz S, Härtle C, Leiber P, Baumgarten T, Sison JA (2013) International evaluation of nlu benefits in the domain of in-vehicle speech dialog systems. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’13. ACM, New York, pp 114–120.  https://doi.org/10.1145/2516540.2516553
  22. 22.
    Haeuslschmid R, Pfleging B, Alt F (2016) A design space to support the development of windshield applications for the car. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pp 5076–5091Google Scholar
  23. 23.
    Harvey C, Stanton NA, Pickering CA, McDonald M, Zheng P (2011) In-vehicle information systems to meet the needs of drivers. Int J Hum Comp Interact 27(6):505–522.  https://doi.org/10.1080/10447318.2011.555296 CrossRefGoogle Scholar
  24. 24.
    Hassenzahl M, Koller F, Burmester M (2008) Der user experience (ux) auf der spur: zum einsatz von www.attrakdiff.de. In: Brau H, Diefenbach S, Hassenzahl M, Koller F, Peissner M, Röse K (eds) Tagungsband UP08. Fraunhofer Verlag, Stuttgart, pp 78–82
  25. 25.
    Heenan A, Herdman CM, Brown MS, Robert N (2014) Effects of conversation on situation awareness and working memory in simulated driving. Hum Factors 56(6):1077–1092.  https://doi.org/10.1177/0018720813519265 PMID: 25277018CrossRefGoogle Scholar
  26. 26.
    Hernandez J, McDuff D, Benavides X, Amores J, Maes P, Picard R (2014) Autoemotive: bringing empathy to the driving experience to manage stress. In: Proceedings of the 2014 companion publication on designing interactive systems, DIS Companion ’14. ACM, New York, pp 53–56.  https://doi.org/10.1145/2598784.2602780
  27. 27.
    ITFG on Driver Distraction (2013) Report on user interface requirements for automotive applications. Technical report, International Telecommunication UnionGoogle Scholar
  28. 28.
    Jakus G, Dicke C, Sodnik J (2015) A user study of auditory, head-up and multi-modal displays in vehicles. Appl Ergon 46, Part A:184–192.  https://doi.org/10.1016/j.apergo.2014.08.008 http://www.sciencedirect.com/science/article/pii/S0003687014001471
  29. 29.
    Kapoor A, Burleson W, Picard RW (2007) Automatic prediction of frustration. Int J Hum Comput Stud 65(8):724–736.  https://doi.org/10.1016/j.ijhcs.2007.02.003 http://www.sciencedirect.com/science/article/pii/S1071581907000377
  30. 30.
    Kennington C, Kousidis S, Baumann T, Buschmeier H, Kopp S, Schlangen D (2014) Better driving and recall when in-car information presentation uses situationally-aware incremental speech output generation. In: Proceedings of the 6th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’14. ACM, New York, pp 7:1–7:7.  https://doi.org/10.1145/2667317.2667332
  31. 31.
    Kern D, Marshall P, Schmidt A (2010) Gazemarks: gaze-based visual placeholders to ease attention switching. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2093–2102. ACMGoogle Scholar
  32. 32.
    Kern D, Schmidt A (2009) Design space for driver-based automotive user interfaces. In: Proceedings of the 1st international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’09. ACM, New York, pp 3–10.  https://doi.org/10.1145/1620509.1620511
  33. 33.
    Kim S, Dey AK (2009) Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’09. ACM, New York, pp 133–142.  https://doi.org/10.1145/1518701.1518724
  34. 34.
    Kountouriotis GK, Wilkie RM, Gardner PH, Merat N (2015) Looking and thinking when driving: the impact of gaze and cognitive load on steering. Transp Res Part F: Traffic Psychol Behav 34:108–121.  https://doi.org/10.1016/j.trf.2015.07.012 http://www.sciencedirect.com/science/article/pii/S1369847815001163
  35. 35.
    Laberge-Nadeau C, Maag U, Bellavance F, Lapierre SD, Desjardins D, Messier S, Saidi A (2003) Wireless telephones and the risk of road crashes. Accid Anal Prev 35(5):649–660CrossRefGoogle Scholar
  36. 36.
    Lamble D, Kauranen T, Laakso M, Summala H (1999) Cognitive load and detection thresholds in car following situations: safety implications for using mobile (cellular) telephones while driving. Accid Anal Prev 31(6):617–623CrossRefGoogle Scholar
  37. 37.
    Large DR, Burnett G, Anyasodo B, Skrypchuk L (2016) Assessing cognitive demand during natural language interactions with a digital driving assistant. In: Proceedings of the 8th international conference on automotive user interfaces and interactive vehicular applications, Automotive’UI 16. ACM, New York, pp 67–74.  https://doi.org/10.1145/3003715.3005408
  38. 38.
    Lo VEW, Green PA (2013) Development and evaluation of automotive speech interfaces: Useful information from the human factors and the related literature. Int J Veh Technol 2013:924,170:1–924,170:13.  https://doi.org/10.1155/2013/924170
  39. 39.
    Luger E, Sellen A (2016) “like having a really bad pa”: the gulf between user expectation and experience of conversational agents. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ’16. ACM, New York, pp 5286–5297.  https://doi.org/10.1145/2858036.2858288
  40. 40.
    McEvoy SP, Stevenson MR, McCartt AT, Woodward M, Haworth C, Palamara P, Cercarelli R (2005) Role of mobile phones in motor vehicle crashes resulting in hospital attendance: a case-crossover study. Bmj 331(7514):428CrossRefGoogle Scholar
  41. 41.
    Müller C, Weinberg G (2011) Multimodal input in the car, today and tomorrow. IEEE MultiMed 18(1):98–103.  https://doi.org/10.1109/MMUL.2011.14 CrossRefGoogle Scholar
  42. 42.
    Nafari M, Weaver C (2013) Augmenting visualization with natural language translation of interaction: a usability study. Comput Graph Forum 32(3pt4):391–400.  https://doi.org/10.1111/cgf.12126 CrossRefGoogle Scholar
  43. 43.
    Neßelrath R, Feld M (2013) Towards a cognitive load ready multimodal dialogue system for in-vehicle human-machine interaction. In: Adjunct proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI Adjunct ’13, pp 49–52. http://www.auto-ui.org/13/docs/aui_adjunct_proceedings_final.pdf#page=49
  44. 44.
    Oh A, Fox H, Van Kleek M, Adler A, Gajos K, Morency LP, Darrell T (2002) Evaluating look-to-talk: a gaze-aware interface in a collaborative environment. In: CHI ’02 extended abstracts on human factors in computing systems, CHI EA ’02. ACM, New York, pp 650–651.  https://doi.org/10.1145/506443.506528
  45. 45.
    Paas F, Tuovinen JE, Tabbers H, Van Gerven PWM (2003) Cognitive load measurement as a means to advance cognitive load theory. Educ Psychol 38(1):63–71.  https://doi.org/10.1207/S15326985EP3801_8 CrossRefGoogle Scholar
  46. 46.
    Pauzié A (2008) A method to assess the driver mental workload: the driving activity load index (dali). Intell Transp Syst IET 2(4):315–322.  https://doi.org/10.1049/iet-its:20080023  http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4706518&tag=1
  47. 47.
    Petzoldt T, Bellem H, Krems JF (2014) The critical tracking task: a potentially useful method to assess driver distraction? Hum Factors 56(4):789–808.  https://doi.org/10.1177/0018720813501864 CrossRefGoogle Scholar
  48. 48.
    Pieraccini R, Dayanidhi K, Bloom J, Dahan JG, Phillips M, Goodman BR, Prasad KV (2004) Multimodal conversational systems for automobiles. Commun. ACM 47(1):47–49.  https://doi.org/10.1145/962081.962104 CrossRefGoogle Scholar
  49. 49.
    Ranney TA, Garrott WR, Goodman MJ (2001) Nhtsa driver distraction research: Past, present, and future. In: Proceedings of the 17th international technical conference on the enhanced safety of vehicles. u.s. department of transportation, National Highway Traffic Safety Administration (2001). http://www-nrd.nhtsa.dot.gov/pdf/esv/esv17/proceed/00177.pdf
  50. 50.
    Redelmeier DA, Tibshirani RJ (1997) Association between cellular-telephone calls and motor vehicle collisions. N Engl J Med 336(7):453–458CrossRefGoogle Scholar
  51. 51.
    Reimer B, Mehler B (2013) The effects of a production level “voice-command” interface on driver behavior: summary findings on reported workload, physiology, visual attention, and driving performance. Technical Report White Paper 2013-18A, MIT AgeLab—Massachusetts Institute of Technology (2013). http://agelab.mit.edu/files/MIT_AgeLab_White_Paper_2013-18A_(Voice_Interfaces).pdf
  52. 52.
    Rümelin S, Siegl P, Butz A (2013) Could you please... investigating cooperation in the car. In: Adjunct proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI Adjunct ’13, pp 61–64. http://www.auto-ui.org/13/docs/aui_adjunct_proceedings_final.pdf#page=63
  53. 53.
    SAE International (2013) Operational definitions of driving performance measures and statistics. Technical Report SAE J2944, SAE International. http://standards.sae.org/wip/j2944/. Proposed Draft
  54. 54.
    Strayer DL, Cooper JM, Turrill J, Coleman JR, Hopman RJ (2015) The smartphone and the driver’s cognitive workload: a comparison of apple, google, and microsoft’s intelligent personal assistants. Technical report, AAA Foundation for Traffic Safety, Washington, DC. https://www.aaafoundation.org/sites/default/files/strayerIIIa_FINALREPORT.pdf
  55. 55.
    Tsimhoni O, Smith D, Green P (2004) Address entry while driving: speech recognition versus a touch-screen keyboard. Hum Factors 46(4):600–610.  https://doi.org/10.1518/hfes.46.4.600.56813 PMID: 15709323CrossRefGoogle Scholar
  56. 56.
    Vollrath M, Maciej J, Niederée U (2008) In-car distraction study—final report. Technical report, Kognitions-und Ingenieurpsychologie, Technische Universität Carolo-Wilhelmina zu Braunschweig, BraunschweigGoogle Scholar
  57. 57.
    Ward N (2006) Non-lexical conversational sounds in American english. Pragmat Cognit 14(1):129–182CrossRefGoogle Scholar
  58. 58.
    Weinberg G, Harsham B, Medenica Z (2011) Investigating huds for the presentation of choice lists in car navigation systems. In: Proceedings of the sixth international driving symposium on human factors in driver assessment, training and vehicle design, DA ’11, pp 195–202Google Scholar
  59. 59.
    White MP, Eiser JR, Harris PR (2004) Risk perceptions of mobile phone use while driving. Risk Anal 24(2):323–334.  https://doi.org/10.1111/j.0272-4332.2004.00434.x CrossRefGoogle Scholar
  60. 60.
    Yan B, Weng F, Feng Z, Ratiu F, Raya M, Meng Y, Varges S, Purver M, Lien A, Scheideck T, Raghunathan B, Lin F, Mishra R, Lathrop B, Zhang Z, Bratt H, Peters S (2007) A conversational in-car dialog system. In: Proceedings of human language technologies: the annual conference of the North American chapter of the association for computational linguistics: demonstrations, NAACL-demonstrations ’07. Association for Computational Linguistics, Stroudsburg, pp 23–24. http://dl.acm.org/citation.cfm?id=1614164.1614176

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.BMW Group Research, New Technologies, InnovationsMunichGermany
  2. 2.LMU MunichMunichGermany
  3. 3.Eindhoven University of TechnologyEindhovenNetherlands
  4. 4.Bundeswehr UniversityMunichGermany

Personalised recommendations