Multimedia Tools and Applications

, Volume 75, Issue 16, pp 9587–9607 | Cite as

Augmenting human senses to improve the user experience in cars: applying augmented reality and haptics approaches to reduce cognitive distances

  • SeungJun KimEmail author
  • Anind K. Dey


Augmenting people’s senses with computational support can improve the ability to perceive information and perform tasks. However, the impact of such augmentation may fluctuate according to user context, thereby impacting the quality of a user experience. In this paper, we present two systems that assess the in-situ effects of augmenting senses using Augmented Reality and Haptic technologies. We demonstrate that sensory augmentation systems can improve performance when users are multitasking; however, a hybrid assessment, including eye tracking and psycho-physiological measurement, reveals that the benefits and costs of such systems can differ depending on the demographics of a population with different cognitive capabilities. For elder adults, sensory augmentation improved perception for responding to local incidents in the physical space, but a richer intervention using sensory augmentation (visual, auditory, and haptic) strains cognitive load. For younger adults, additional modes for providing sensory information increased attentiveness for performing tasks, but can lead to overloading of already used sensory channels. Thus, sensory augmentation was more advantageous for improving global awareness for situated physical space, rather than responding to local incidents.


Human-computer interaction Automotive user interfaces Multisensory interaction Sensory augmentation systems 



This project is funded in part by General Motors, Carnegie Mellon University’s Technologies for Safe and Efficient Transportation, The National USDOT University Transportation Center for Safety (T-SET UTC) which is sponsored by the US Department of Transportation.


  1. 1.
    Anderson JR (2002) Spanning seven orders of magnitude: a challenge for cognitive modeling. Cogn Sci 26(1):85–112CrossRefGoogle Scholar
  2. 2.
    Canham M, Hegarty M (2010) Effects of knowledge and display design on comprehension of complex graphics. Learn Instr 20(2):155–166CrossRefGoogle Scholar
  3. 3.
    Carton A, Dunne LE (2013) Tactile distance feedback for firefighters: design and preliminary evaluation of a sensory augmentation glove. In Proceedings of the 4th Augmented Human International Conference (AH’13). ACM, New York, pp 58–64Google Scholar
  4. 4.
    Cegarra J, Chevalier A (2008) The use of tholos software for combining measures of mental workload: toward theoretical and methodological improvements. Behav Res Methods 40(4):988–1000CrossRefGoogle Scholar
  5. 5.
    Chandler P, Sweller J (1991) Cognitive load theory and the format of instruction. Cogn Instr 8(4):293–332CrossRefGoogle Scholar
  6. 6.
    Chandler P, Sweller J (1992) The split-attention effect as a factor in the design of instruction. Br J Educ Psychol 62(2):233–246CrossRefGoogle Scholar
  7. 7.
    Chang W, Hwang W, Ji YG (2011) Haptic seat interfaces for driver information and warning systems. Int J Hum Comput Interact 27:1119–1132CrossRefGoogle Scholar
  8. 8.
    Clive-smith M (2013) Eidos: Audio/Visual Sensory Augmentation. Media Evolution, The Conference Wearable TechnologyGoogle Scholar
  9. 9.
    Collins CC (1985) On mobility aids for the blind. In: Warren DH, Strelow ER (eds) Electronic spatial sensing for the blind. Martinus Nijhoff, Boston, pp 35–64CrossRefGoogle Scholar
  10. 10.
    Conati C, Maclaren H (2009) Empirically building and evaluating a probabilistic model of user affect. User Model User-Adap Inter 19:267–303CrossRefGoogle Scholar
  11. 11.
    Cook AM (1982) Sensory and communication aids. In: Cook AM, Webster JG (eds) Therapeutic Medical Devices: Application and Design. Prentice-Hall, Englewood Cliffs, pp 152–201Google Scholar
  12. 12.
    Coyne JT, Baldwin C, Cole A, Sibley C, Roberts DM (2009) Applying Real Time Physiological Measures of Cognitive Load to Improve Training. HCI 16, of Lecture Notes in Computer Science, vol 5638. Springer, 469–478Google Scholar
  13. 13.
    D’Mello S, Olney A, Williams C, Hays P (2012) Gaze tutor: a gaze-reactive intelligent tutoring system. Int J Hum Comput Stud 70(5):370–389, Elsevier Google Scholar
  14. 14.
    De Waard D (1996) The measurement of drivers’ mental workload. PhD Thesis, University of GroningenGoogle Scholar
  15. 15.
    Fisk AD, Derrick WL, Schneider W (1986–87). A methodological assessment and evaluation of dual-task paradigms. Curr Psychol 5(4): 315–327Google Scholar
  16. 16.
    Goldberg JH, Stimson MJ, Lewenstein M, Scott N, Wichansky AM (2002) Eye tracking in web search tasks: design implications, In Proceedings of the 2002 symposium on Eye tracking research & applications (ETRA’02). ACM, New York, pp 51–58Google Scholar
  17. 17.
    Haapalainen E, Kim S, Forlizzi J, Dey A (2010) Psycho-physiological Measures for assessing Cognitive Load. In: Proc. Ubicomp. 301–310Google Scholar
  18. 18.
    Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human Mental Workload. Amsterdam, North-Holland, pp 139–183CrossRefGoogle Scholar
  19. 19.
    Hwang S, Ryu J (2010) The Haptic steering Wheel: Vibro-tactile based navigation for the driving environment. In: IEEE Intl’ Conf. PERCOM Workshops. 660–665.Google Scholar
  20. 20.
    Iqbal ST, Zheng XS, Bailey BP (2004) Task evoked pupillary response to mental workload in human-computer interaction. In: Proceedings of the ACM conference on human factors in computing systems. 1477–1480Google Scholar
  21. 21.
    Javal E (1879) Essai sur la Physiologie de la Lecture Annales D’OculistiqueGoogle Scholar
  22. 22.
    Kaczmarek KA (1995) Sensory augmentation and substitution. In: Bronzino JD (ed) CRC handbook of biomedical engineering. CRC Press, Boca Raton, pp 2100–2109Google Scholar
  23. 23.
    Kahneman D (1973) Attention and Effort. Prentice-Hall, Englewood CliffsGoogle Scholar
  24. 24.
    Kalyuga S (2007) Expertise reversal effect and its implications for learner-tailored instruction. Educ Psychol Rev 19:509–539CrossRefGoogle Scholar
  25. 25.
    Kalyuga S, Rikers R, Pass F (2012) Educational implications of expertise reversal effect in learning and performance of complex cognitive and sensorimotor skills. Educ Psychol Rev 24:313–337CrossRefGoogle Scholar
  26. 26.
    Kern D, Marshall P, Hornecker E, Rogers Y, Schmidt A (2009) Enhancing Navigation Information with Tactile Output Embedded into the Steering Wheel. In: Tokuda H, Beigl M, Friday A, Brush AJB, Tobe Y (eds) Pervasive 2009. LNCS, vol 5538. Springer, Heidelberg, pp 42–58Google Scholar
  27. 27.
    Kim S, Dey AK (2009) Simulated Augmented Reality Windshield Display as a Cognitive Mapping Aid for Elder Driver Navigation. In: Proc. CHI. 133–142Google Scholar
  28. 28.
    Kim S, Aleven V, Dey AK (2014) Understanding expert-novice differences in geometry problem-solving tasks: a sensor-based approach, In CHI’14 Extended Abstracts on Human Factors in Computing Systems (CHI EA’14). ACM, New York, pp 1867–1872Google Scholar
  29. 29.
    Kim S, Dey AK, Lee J, Forlizzi J (2011) Usability of car dashboard displays for elder drivers. In: Proc. CHI. 493–502Google Scholar
  30. 30.
    Kim S, Hong J, Li KA, Forlizzi J, Dey AK (2012) Route Guidance Modality for Elder Driver Navigation. Pervasive 12:179–196Google Scholar
  31. 31.
    Klauer SG, Guo F, Simons-Morton BG, Ouimet MC, Lee SE, Dingus TA (2014) Distracted driving and risk of road crashes among novice and experienced drivers. N Engl J Med 370(1):54–59CrossRefGoogle Scholar
  32. 32.
    Kort B, Reilly R, Picard RW (2001) An Affective Model of Interplay Between Emotions and Learning: Reengineering educational Pedagogy-Building a Learning Companion. In: Proceedings of International Conference on Advanced Learning Technologies (ICALT 2001), August 2001, Madison, WIGoogle Scholar
  33. 33.
    Kramer AF (1991) Physiological metrics of mental workload: a review of recent progress. In: Damos DL (ed) Multiple-task performance. Taylor & Francis, London, pp 279–328Google Scholar
  34. 34.
    Larson R, Csikszentmihalyi M (1983) The experience sampling method. New Dir Methodol Soc BehavSci 15:41–56Google Scholar
  35. 35.
    Mauri M, Magagnin V, Cipresso P, Mainardi L, Brown EN, Cerutti S, Villamira M, Barbieri R (2010) Psychophysiological signals associated with affective states. Conf Proc IEEE Eng Med Biol Soc 2010:3563–3566Google Scholar
  36. 36.
    Mercier J, Leger PM, Girard C, Dion JS (2012) Bridging the gap between cognitive neuroscience and education: psychophysiological and behavioral data collection in authentic contexts. Neuroeducation 1(1):5–28Google Scholar
  37. 37.
    Mital A, Govindaraju M (1999) Is it possible to have a single measure for all work? Int J Ind Eng Theory 6:190–195Google Scholar
  38. 38.
    Narzt W, Pomberger G, Ferscha A, Kolb D, Muller R, Wieghardt J, Hortner H, Lindinger C (2006) Augmented reality navigation systems. Univ Access Inf Soc 4(3):177–187CrossRefGoogle Scholar
  39. 39.
    Paas FGWC, van Merrienboer JJG (1994) Variability of worked examples and transfer of geometrical problem solving skills: a cognitive load approach. J Educ Psychol 86:122–133CrossRefGoogle Scholar
  40. 40.
    Paas F, Renkel A, Sweller J (2004) Cognitive load theory: instructional implications of the interaction between information structures and cognitive architecture. Instr Sci 32:1–8CrossRefGoogle Scholar
  41. 41.
    Poole A, Ball LJ, Phillips P (2004) In search of salience: A response time and eye movement analysis of bookmark recognition, In People and Computers XVIII (Proceedings of HCI 2004). Springer, London, pp 363–378Google Scholar
  42. 42.
    Posner MI, Boies SJ (1971) Components of attention. Psychol Rev 78(5):391–408CrossRefGoogle Scholar
  43. 43.
    Salvucci DD (1999) Mapping eye movements to cognitive processes (Tech. Rep. No. CMU-CS-99-131). Doctoral Dissertation, Department of Computer Science, Carnegie Mellon University.Google Scholar
  44. 44.
    Salvucci DD (2001) Predicting the effects of in-car interface use on driver performance: an integrated model approach. Int J Hum Comput Stud 55(1):85–107CrossRefzbMATHGoogle Scholar
  45. 45.
    Schiessl M, Duda S, Tholke A, Fischer R (2003) Eye tracking and its application in usability and media research. MMI interaktiv, Vol. 6, available at:
  46. 46.
    Setz C, Arnrich B, Schumm J, La Marca R, Tröster G, Ehlert U (2010) Discriminating stress from cognitive load using a wearable EDA device. IEEE Trans Inf Technol Biomed 14(2):410–417CrossRefGoogle Scholar
  47. 47.
    Simon HA (1971) Designing organizations for an information rich world. In Computers, Communications, and the Public Interest. 37–72Google Scholar
  48. 48.
    Sottilare R, Graesser A, Hu X, Holden H (2013) Design Recommendations for Intelligent Tutoring Systems: Learner Modeling, vol 1. Army Research Laboratory, OrlandoGoogle Scholar
  49. 49.
    Wickens CD (1981) Processing Resources in Attention, Dual Task Performance, and Workload Assessment Technical Report EPL-81-3/ONR-81-3. University of Illinois, Engineering-Psychology Research Laboratory, ChampaignGoogle Scholar
  50. 50.
    Wickens CD (1984) Processing resources in attention. In: Parasuraman R, Davies R (eds) Varieties of Attention. Academic, New York, pp 63–101Google Scholar
  51. 51.
    Wickens CD (1991) Processing resources and attention. In: Damos DL (ed) Multiple Task Performance. Taylor and Francis, Ltd., Bristol, pp 3–34Google Scholar
  52. 52.
    Yarbus AL (1967) Eye Movements and Vision. Plenum, New York, Originally published in Russian 1962 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Human-Computer Interaction Institute (HCII)Carnegie Mellon University (CMU)PittsburghUSA

Personalised recommendations