Skip to main content
SpringerLink
Log in

Inception of Perception—Augmented Reality in Virtual Reality: Prototyping Human–Machine Interfaces for Automated Driving

  • Chapter
  • First Online:
User Experience Design in the Era of Automated Driving

Part of the book series: Studies in Computational Intelligence ((SCI,volume 980))

  • The original version of this chapter was revised: The missing corresponding author and co-author names are inserted with affiliations. The correction to this chapter is available at https://doi.org/10.1007/978-3-030-77726-5_21

Abstract

Augmented Reality in vehicles through Head-Up or Windscreen Displays offer numerous opportunities to develop road safety or user experience applications or both. These opportunities are further expanded in the context of automated driving, where human drivers are partially freed of the safety-critical driving task to attend to other activities, so-called non-driving related tasks. This brings new human factor challenges, such as lack of situation awareness when the driver is asked to take control of the driving vehicle. There are methodological challenges when undertaking user experience design or human factors research in this domain because of the current development status of future technologies. Prototypes of windscreen displays and automated vehicles are difficult to access and expensive. Therefore, we have developed an affordable virtual reality automated driving simulator that implements real-world driving videos for interface prototyping and testing. In this chapter, we discuss two concepts (dynamic windshield tinting to communicate certainty and an Augmented Reality video screen movements to communicate intention) to extend the windshield with additional driving relevant information, which were rapidly prototyped and evaluated in our Virtual Reality simulator. Following the user experience design approach, we report on qualitative findings from a user study design iteration (N = 16) about those concepts. We reflect on the identified strengths and weaknesses of our design concepts, which were critically influenced by participants’ experience and trust in vehicle automation, as well as the methodological process. Lastly, we report on valuable lessons learnt from developing, improving and implementing our prototyping, simulation and evaluation tool for Augmented Reality applications in automated driving.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Change history

  • 01 January 2022

    Correction to: Chapter “Inception of Perception—Augmented Reality in Virtual Reality: Prototyping Human–Machine Interfaces for Automated Driving” in: A. Riener et al. (eds.), User Experience Design in the Era of Automated Driving, Studies in Computational Intelligence 980, https://doi.org/10.1007/978-3-030-77726-5_18

Notes

  1. 1.

    https://www.nhtsa.gov/technology-innovation/automated-vehicles-safety, retrieved 30. June 2020.

  2. 2.

    https://varjo.com/products/xr-1/, retrieved 30. June 2020.

  3. 3.

    Second life and online virtual environment.

  4. 4.

    Photogrammetry is a technique of capturing objects and environment in three dimensions with topological and geographical data respectively.

  5. 5.

    https://www.microsoft.com/en-us/hololens, Retrieved 30 June 2020.

  6. 6.

    https://www.magicleap.com/en-us/magic-leap-1, Retrieved 30 June 2020.

  7. 7.

    https://www.oculus.com/rift/, retrieved 30 June 2020.

  8. 8.

    https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool, retrieved 30 June 2020.

  9. 9.

    Oculus controls were not used in this study.

  10. 10.

    The IPD is the distance between the left and the right eye. The lenses can be moved together and apart to adjust that.

  11. 11.

    Make transparent based on a unified (predefined) color value a.k.a “Green Screen” or “Blue Screen”.

  12. 12.

    The videos can be recorded stereoscopic and then played back on the video-players individually for each eye to receive depth.

References

  1. Abdi, L., Abdallah, F.B., Meddeb, A.: In-vehicle augmented reality traffic information system: a new type of communication between driver and vehicle. Procedia Comput. Sci. https://www.researchgate.net/profile/Lotfi_Abdi/publication/291013097_In-Vehicle_Augmented_Reality_Traffic_Information_System_A_New_Type_of_Communication_Between_Driver_and_Vehicle/links/56e6da7908ae9e803e7bee6d.pdf (2015)

  2. Agarwal, R.: Validation of a pedestrian simulator for interaction between pedestrians and autonomous vehicles. In: de Winter Joost, H.R. (ed.) Master thesis, Delft University of Technology. https://repository.tudelft.nl/islandora/object/uuid%3Ae1581f7c-aa7e-4606-991b-bbaf7940c79c (2019)

  3. Alvarez, I., Rumbel, L., Adams, R.: Skyline: a rapid prototyping driving simulator for user experience. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 101–108 (2015). https://doi.org/10.1145/2799250.2799290

  4. Azuma, R.T.: A survey of augmented reality. Presence: Teleoper. Virtual Environ. 6(4), 355–385 (1997). https://doi.org/10.1162/pres.1997.6.4.355

  5. Barić, D., Havârneanu, G.M., Măirean, C.: Attitudes of learner drivers toward safety at level crossings: Do they change after a 360° video-based educational intervention? Transport. Res. Part F Traffic Psychol. Behav. 69, 335–348 (2020). https://doi.org/10.1016/j.trf.2020.01.018

    Article  Google Scholar 

  6. Brooks, J.O., Goodenough, R.R., Crisler, M.C., Klein, N.D., Alley, R.L., Koon, B.L., Logan, W.C., Jr., Ogle, J.H., Tyrrell, R.A., Wills, R.F.: Simulator sickness during driving simulation studies. Accident; Anal. Prevent. 42(3), 788–796 (2010). https://doi.org/10.1016/j.aap.2009.04.013

    Article  Google Scholar 

  7. Chatzidimitris, T., Gavalas, D., Michael, D.: SoundPacman: audio augmented reality in location-based games. In: 2016 18th Mediterranean Electrotechnical Conference (MELECON), pp. 1–6 (2016). https://doi.org/10.1109/MELCON.2016.7495414

  8. Craig, A.B.: Understanding Augmented Reality: Concepts and Applications. Newnes. https://market.android.com/details?id=book-7_O5LaIC0SwC (2013)

  9. Cruz-Neira, C., Sandin, D.J., DeFanti, T.A., Kenyon, R.V., Hart, J.C.: The CAVE: audio visual experience automatic virtual environment. Commun. ACM 35(6), 64–72 (1992). https://doi.org/10.1145/129888.129892

  10. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: arXiv [cs.LG]. http://arxiv.org/abs/1711.03938 (2017)

  11. Endsley, M.R.: Designing for Situation Awareness: An Approach to User-Centered Design, 2nd edn. CRC Press. https://market.android.com/details?id=book-eRPBkapAsggC (2016)

  12. Endsley, M.R., Kiris, E.O.: The out-of-the-loop performance problem and level of control in automation. Hum. Factors 37(2), 381–394 (1995). https://doi.org/10.1518/001872095779064555

    Article  Google Scholar 

  13. Feierle, A., Beller, D., Bengler, K.: Head-up displays in urban partially automated driving: effects of using augmented reality*. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 1877–1882 (2019). https://doi.org/10.1109/ITSC.2019.8917472

  14. Gerber, M.A., Schroeter, R., Vehns, J.: A video-based automated driving simulator for automotive UI prototyping, UX and behaviour research. In: 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI ‚19). Automotive User Interfaces 2019, Utrecht, Netherlands (2019)

    Google Scholar 

  15. Gerber, M.A., Schroeter, R., Vehns, J.: A video-based automated driving simulator for automotive UI prototyping, UX and behaviour research. In: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 14–23 (2019). https://doi.org/10.1145/3342197.3344533

  16. Gerber, M.A., Schroeter, R., Xiaomeng, L., Elhenawy, M.: Self-interruptions of non-driving related tasks in automated vehicles: mobile versus head-up display. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–9 (2020). https://doi.org/10.1145/3313831.3376751

  17. Goedicke, D., Li, J., Evers, V., Ju, W.: VR-OOM: Virtual Reality On-rOad Driving siMulation. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 165:1–165:11 (2018). https://doi.org/10.1145/3173574.3173739

  18. Guo, C., Sentouh, C., Popieul, J.-C., Haué, J.-B., Langlois, S., Loeillet, J.-J., Soualmi, B., Nguyen That, T.: Cooperation between driver and automated driving system: Implementation and evaluation. Transpor. Res. Part F Traffic Psychol. Behav. 61, 314–325 (2019). https://doi.org/10.1016/j.trf.2017.04.006

    Article  Google Scholar 

  19. Häuslschmid, R., Osterwald, S., Lang, M., Butz, A.: Augmenting the driver’s view with peripheral information on a windshield display. In: Proceedings of the 20th International Conference on Intelligent User Interfaces, pp. 311–321 (2015). https://doi.org/10.1145/2678025.2701393

  20. Häuslschmid, R., von Bülow, M., Pfleging, B., Butz, A.: Supporting trust in autonomous driving. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces, pp. 319–329 (2017). https://doi.org/10.1145/3025171.3025198

  21. Heinrich, F.: Vorhersage der Fahrerbelastung während der Fahrt. https://elib.uni-stuttgart.de/handle/11682/2967 (2012)

  22. Helldin, T., Falkman, G., Riveiro, M., Davidsson, S.: Presenting system uncertainty in automotive UIs for supporting trust calibration in autonomous driving. In: Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 210–217 (2013). https://doi.org/10.1145/2516540.2516554

  23. Horswill, M.S., McKenna, F.P.: Drivers’ hazard perception ability: Situation awareness on the road. In: A Cognitive Approach to Situation. https://www.researchgate.net/profile/Frank_Mckenna2/publication/271767727_Drivers’_hazard_perception_ability_Situation_awareness_on_the_road/links/57b2261d08ae95f9d8f4cb38/Drivers-hazard-perception-ability-Situation-awareness-on-the-road (2004)

  24. Jung, T., tom Dieck, M.C., Lee, H., Chung, N.: Effects of virtual reality and augmented reality on visitor experiences in museum. In: Information and Communication Technologies in Tourism 2016, pp. 621–635 (2016). https://doi.org/10.1007/978-3-319-28231-2_45

  25. Kawulich, B.B., D’Alba, A.: Teaching qualitative research methods with Second Life, a 3-dimensional online virtual environment. Virtual Reality 23(4), 375–384 (2019). https://doi.org/10.1007/s10055-018-0353-4

    Article  Google Scholar 

  26. Kelley, J.F.: An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Infor. Syst. Secur. 2(1), 26–41 (1984). https://doi.org/10.1145/357417.357420

    Article  Google Scholar 

  27. Kern, D., Müller, M., Schneegaß, S., Wolejko-Wolejszo, L., Schmidt, A.: CARS—Configurable automotive research simulator. In: Lucke, U., Kindsmüller, M.C., Fischer, S., Herczeg, M., Seehusen, S. (eds.) Workshop Proceedings der Tagungen Mensch & Computer 2008, DeLFI 2008 und Cognitive Design 2008, pp. 256–260. Logos Verlag (2008)

    Google Scholar 

  28. Kim, H., Gabbard, J.L.: Assessing distraction potential of augmented reality head-up displays for vehicle drivers. Hum. Factors 18720819844845 (2019). https://doi.org/10.1177/0018720819844845

  29. Kim, H., Miranda Anon, A., Misu, T., Li, N., Tawari, A., Fujimura, K.: Look at me: augmented reality pedestrian warning system using an in-vehicle volumetric head up display. In: Proceedings of the 21st International Conference on Intelligent User Interfaces, pp. 294–298 (2016). https://doi.org/10.1145/2856767.2856815

  30. Kim, S., Dey, A.K.: Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 133–142 (2009). https://doi.org/10.1145/1518701.1518724

  31. Klüver, M., Herrigel, C., Heinrich, C., Schöner, H.-P., Hecht, H.: The behavioral validity of dual-task driving performance in fixed and moving base driving simulators. Transport. Res. Part F Traffic Psychol. Behav. 37, 78–96 (2016). https://doi.org/10.1016/j.trf.2015.12.005

    Article  Google Scholar 

  32. Körber, M., Baseler, E., Bengler, K.: Introduction matters: Manipulating trust in automation and reliance in automated driving. Appl. Ergon. 66, 18–31 (2018). https://doi.org/10.1016/j.apergo.2017.07.006

    Article  Google Scholar 

  33. Kunze, A., Summerskill, S.J., Marshall, R., Filtness, A.J.: Automation transparency: implications of uncertainty communication for human-automation interaction and interfaces. Ergonomics 62(3), 345–360 (2019). https://doi.org/10.1080/00140139.2018.1547842

    Article  Google Scholar 

  34. Kunze, A., Summerskill, S.J., Marshall, R., Filtness, A.J.: Augmented reality displays for communicating uncertainty information in automated driving. In: Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 164–175 (2018). https://doi.org/10.1145/3239060.3239074

  35. Langlois, S., Soualmi, B.: Augmented reality versus classical HUD to take over from automated driving: An aid to smooth reactions and to anticipate maneuvers. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 1571–1578 (2016). https://doi.org/10.1109/ITSC.2016.7795767

  36. Lee, J.D., Moray, N.: Trust, self-confidence, and operators’ adaptation to automation. Int. J. Hum. -Comput. Stud. 40(1), 153–184 (1994). https://doi.org/10.1006/ijhc.1994.1007

    Article  Google Scholar 

  37. Li, X., Schroeter, R., Rakotonirainy, A., Kuo, J., Lenné, M.G.: Effects of different non-driving-related-task display modes on drivers’ eye-movement patterns during take-over in an automated vehicle. Transport. Res. Part F Traffic Psychol. Behav. 70, 135–148. https://www.sciencedirect.com/science/article/pii/S136984781930659X (2020)

  38. Lorenz, L., Kerschbaum, P., Schumann, J.: Designing take over scenarios for automated driving: how does augmented reality support the driver to get back into the loop? Proceedings of the Human Factors and Ergonomics Society ... Annual Meeting Human Factors and Ergonomics Society. Meeting, vol. 58, issue 1, pp. 1681–1685 (2014). https://doi.org/10.1177/1541931214581351

  39. Medenica, Z., Kun, A.L., Paek, T., Palinko, O.: Augmented reality versus street views: a driving simulator study comparing two emerging navigation aids. In: Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, pp. 265–274 (2011). https://doi.org/10.1145/2037373.2037414

  40. Milgram, P., Takemura, H., Utsumi, A.: Augmented reality: aclass of displays on the reality-virtuality continuum. Telemanipulator and. http://wiki.commres.org/pds/Project_7eNrf2010/_5.pdf(1994)

  41. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: a class of displays on the reality-virtuality continuum. Telemanipul Telepresence Technol. 2351, 282–292 (1995). https://doi.org/10.1117/12.197321

    Article  Google Scholar 

  42. Ng-Thow-Hing, V., Bark, K., Beckwith, L., Tran, C., Bhandari, R., Sridhar, S.: User-centered perspectives for automotive augmented reality. In: IEEE International Symposium on Mixed and Augmented Reality, pp. 13–22. http://www.academia.edu/download/35225563/ismar13-Aug12-libre.pdf (2013)

  43. Pärsch, N., Harnischmacher, C., Baumann, M., Engeln, A., Krauß, L.: Designing augmented reality navigation visualizations for the vehicle: a question of real world object coverage? In: HCI in Mobility, Transport, and Automotive Systems, pp. 161–175 (2019). https://doi.org/10.1007/978-3-030-22666-4_12

  44. Sadeghian Borojeni, S., Boll, S.C.J., Heuten, W., Bülthoff, H.H., Chuang, L.: Feel the movement: real motion influences responses to take-over requests in highly automated vehicles. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 246:1–246:13 (2018). https://doi.org/10.1145/3173574.3173820

  45. SAE International: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (No. J3016 JUN2018). SAE International (2018)

    Google Scholar 

  46. Schömig, N., Wiedemann, K., Naujoks, F., Neukum, A., Leuchtenberg, B., Vöhringer-Kuhnt, T.: An augmented reality display for conditionally automated driving. In: Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 137–141 (2018). https://doi.org/10.1145/3239092.3265956

  47. Schroeter, R., Gerber, M.A.: A low-cost vr-based automated driving simulator for rapid automotive UI prototyping. In: Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 248–251 (2018). https://doi.org/10.1145/3239092.3267418

  48. Schroeter, R., Steinberger, F.: Pokémon DRIVE: towards increased situational awareness in semi-automated driving. In: Proceedings of the 28th Australian Conference on Computer-Human Interaction, pp. 25–29 (2016). https://doi.org/10.1145/3010915.3010973

  49. Sommerauer, P., Müller, O.: Augmented reality in informal learning environments: a field experiment in a mathematics exhibition. Comput. Educ. 79, 59–68 (2014). https://doi.org/10.1016/j.compedu.2014.07.013

    Article  Google Scholar 

  50. Soro, A., Rakotonirainy, A., Schroeter, R., Wollstädter, S.: Using augmented video to test in-car user experiences of context analog HUDs. In: Adjunct Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 1–6 (2014). https://doi.org/10.1145/2667239.2667302

  51. Spies, R., Ablaßmeier, M., Bubb, H., Hamberger, W.: Augmented interaction and visualization in the automotive domain. In: Human-Computer Interaction. Ambient, Ubiquitous and Intelligent Interaction, pp. 211–220 (2009). https://doi.org/10.1007/978-3-642-02580-8_23

  52. Steinberger, F., Schroeter, R., Lindner, V., Fitz-Walter, Z., Hall, J., Johnson, D.M.: Zombies on the road: a holistic design approach to balancing gamification and safe driving. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 320–327. https://eprints.qut.edu.au/84799/ (2015)

  53. Tönnis, M., Klinker, G., Plavšic, M.: Survey and classification of head-up display presentation principles. In: Proceedings of the International Ergonomics Association (IEA). http://campar.in.tum.de/pub/toennis2009ieaDimensions/toennis2009ieaDimensions.pdf (2009)

  54. Tran, C., Bark, K., Ng-Thow-Hing, V.: A left-turn driving aid using projected oncoming vehicle paths with augmented reality. In: Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 300–307 (2013). https://doi.org/10.1145/2516540.2516581

  55. Tu, H., Li, Z., Sun, Y., Dai, X.: Driving behavior in case of haze: a high fidelity driving simulator study. CICTP 2014, 2408–2421 (2014). https://doi.org/10.1061/9780784413623.230

    Article  Google Scholar 

  56. von Sawitzky, T., Wintersberger, P., Riener, A., Gabbard, J.L.: Increasing trust in fully automated driving: route indication on an augmented reality head-up display. In: Proceedings of the 8th ACM International Symposium on Pervasive Displays, p. 6 (2019). https://doi.org/10.1145/3321335.3324947

  57. Wintersberger, P., Frison, A.-K., Riener, A., von Sawitzky, T.: Fostering user acceptance and trust in fully automated vehicles: evaluating the potential of augmented reality. PRESENCE: Virtual Augment. Reality 27(1), 46–62 (2019). https://doi.org/10.1162/PRES_a_00320

  58. Wintersberger, P., van der Heiden, R., Borojeni, S. S., Gerber, M.A., Green, P.: First workshop on attentive and pervasive UI in automated vehicles. In: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings, pp. 63–70 (2019). https://doi.org/10.1145/3349263.3350764

  59. Wintersberger, P., von Sawitzky, T., Frison, A.-K., Riener, A.: Traffic augmentation as a means to increase trust in automated driving systems. In: Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter, 1–7 (2017). https://doi.org/10.1145/3125571.3125600

Download references

Author information

Authors and Affiliations

  1. Centre for Accident Research & Road Safety, Queensland University of Technology, Brisbane, CRICOS 00213J, Australia

    Michael A. Gerber, Mohammad Faramarzian & Ronald Schroeter

Authors
  1. Michael A. Gerber
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Faramarzian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ronald Schroeter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael A. Gerber .

Editor information

Editors and Affiliations

  1. CARISSMA Institute of Automated Driving (C-IAD), Technische Hochschule Ingolstadt, Ingolstadt, Germany

    Andreas Riener

  2. Grado Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, VA, USA

    Myounghoon Jeon

  3. Systems and Software Research, Autonomous Driving Research Lab, Intel Labs, Portland, OR, USA

    Ignacio Alvarez

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gerber, M.A., Faramarzian, M., Schroeter, R. (2022). Inception of Perception—Augmented Reality in Virtual Reality: Prototyping Human–Machine Interfaces for Automated Driving. In: Riener, A., Jeon, M., Alvarez, I. (eds) User Experience Design in the Era of Automated Driving. Studies in Computational Intelligence, vol 980. Springer, Cham. https://doi.org/10.1007/978-3-030-77726-5_18

Download citation

Publish with us

Policies and ethics

Search

Navigation