Human Performance and Cognitive Workload in Multi-sensory Virtual Environments

  • Mortaja AlQassabEmail author
  • David Wang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9738)


In this paper, we present the results of an experiment evaluating human performance and cognitive workload by examining the response time and accuracy in a multi-sensory virtual environment. This study examines Haptic, Audio and Visual Environments (HAVEs). The results of this study indicate that there exists a combination of senses that increase performance and decrease workload.


Human performance Cognitive workload Multi-model virtual environment Collaborative virtual environments Human-Machine Interaction 


  1. 1.
    Kjölberg, J., Sallnäs, E.: Supporting object handling and hand over tasks in haptic collaborative virtual environments. In: EuroHaptics 2002 Conference Proceedings, pp. 71–76 (2002)Google Scholar
  2. 2.
    Huang, Y.Y., Moll, J., Sallnäs, E.L., Sundblad, Y.: Auditory feedback in haptic collaborative interfaces. Int. J. Hum Comput Stud. 70, 257–270 (2012)CrossRefGoogle Scholar
  3. 3.
    Teather, R.J., Natapov, D., Law, F.: Evaluating haptic feedback in virtual environments using ISO 9241-9. In: Virtual Reality Conference, VR 2010, pp. 9–10. IEEE (2010)Google Scholar
  4. 4.
    Srinivasan, M.A., Beauregard, G.L., Brock, D.L.: The impact of visual information on the haptic perception of stiffness in virtual environments. In: Proceedings of the ASME Dynamics Systems and Control Division (1996)Google Scholar
  5. 5.
    Bark, K., Khanna, P., Irwin, R., Kapur, P., Jax, S.A., Buxbaum, L.J., Kuchenbecker, K.J.: Lessons in using vibrotactile feedback to guide fast arm motions. In: 2011 IEEE World Haptics Conference, pp. 355–360 (2011)Google Scholar
  6. 6.
    Young, J.J., Stolfi, C., Tan, H.Z., Chevrier, J., Dick, B., Bertoline, G.: Learning force concepts using visual trajectory and haptic force information at the elementary school level. In: 2011 IEEE World Haptics Conference, pp. 391–396 (2011)Google Scholar
  7. 7.
    Mulder, M., Abbink, D.A., Boer, E.R.: The effect of haptic guidance on curve negotiation behavior of young, experienced drivers. In: 2008 IEEE International Conference Systems, Man and Cybernetics, pp. 804–809 (2008)Google Scholar
  8. 8.
    Ullah, S., Liu, X., Otmane, S., Richard, P., Mallem, M.: What you feel is what i do: a study of dynamic haptic interaction in distributed collaborative virtual environment. In: Jacko, J.A. (ed.) Human-Computer Interaction, Part II, HCII 2011. LNCS, vol. 6762, pp. 140–147. Springer, Heidelberg (2011)Google Scholar
  9. 9.
    Boessenkool, H., Abbink, D.A., Heemskerk, C.J.M., van der Helm, F.C.T.: Haptic shared control improves tele-operated task performance towards performance in direct control. In: 2011 IEEE World Haptics Conference, pp. 433–438 (2011)Google Scholar
  10. 10.
    Wang, J., Chellali, A., Cao, C.G.L.: A study of communication modalities in a virtual collaborative task. In: Proceedings - 2013 IEEE International Conference Systems, Man and Cybernetics, SMC 2013, pp. 542–546 (2013)Google Scholar
  11. 11.
    Hecht, D., Reiner, M.: Sensory dominance in combinations of audio, visual and haptic stimuli. Exp. Brain Res. 193, 307–314 (2009)CrossRefGoogle Scholar
  12. 12.
    McLaughlin, M., Sukhatme, G., Peng, W.P.W., Zhu, W.Z.W., Parks, J.: Performance and co-presence in heterogeneous haptic collaboration. In: 11th Symposium Haptic Interfaces Interfaces for Virtual Environment and Teleoperator Systems, HAPTICS 2003. Proceedings (2003)Google Scholar
  13. 13.
    Zahariev, M.A., MacKenzie, C.L.: Auditory, graphical and haptic contact cues for a reach, grasp, and place task in an augmented environment. In: Proceedings of 5th International Conference Multimodal Interfaces, ICMI 2003, p. 273 (2003)Google Scholar
  14. 14.
    Pinho, M.S., Bowman, D.A., Freitas, C.M.D.S.: Cooperative object manipulation in immersive virtual environments. In: Proceedings of ACM Symposium on Virtual Reality Software and Technology, VRST 2002, p. 171 (2002)Google Scholar
  15. 15.
    Moll, J., Pysander, E.-L.S., Eklundh, K.S., Hellstrom, S.-O.: The effects of audio and haptic feedback on collaborative scanning and placing. Interact. Comput. 26, 177–195 (2013)CrossRefGoogle Scholar
  16. 16.
    Sallnäs, E.: Collaboration in multi-modal virtual worlds: comparing touch, text, voice and video. J. Endod. 28, 172–187 (2002)Google Scholar
  17. 17.
    Chellali, A., Milleville, I.: Haptic communication to enhance collaboration in virtual Environments. In: Proceedings of European Conference on Cognitive Ergonomics, ECCE 2010, pp. 83–90 (2010)Google Scholar
  18. 18.
    Girard, A., Bellik, Y., Auvray, M., Ammi, M.: Visuo-haptic tool for collaborative adjustment of selections. In: Oakley, I., Brewster, S. (eds.) HAID 2013. LNCS, vol. 7989, pp. 40–49. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  19. 19.
    Sallnäs, E., Rassmus-Grohn, K., Sjostrom, C.: Supporting presence in collaborative environments by haptic force feedback. ACM Trans. Comput. Interact. 7, 461–476 (2000)CrossRefGoogle Scholar
  20. 20.
    Nam, C.S., Shu, J., Chung, D.: The roles of sensory modalities in collaborative virtual environments (CVEs). Comput. Hum. Behav. 24, 1404–1417 (2008)CrossRefGoogle Scholar
  21. 21.
    Ullah, S., Richard, P., Otmane, S., Naud, M., Mallem, M.: Haptic guides in cooperative virtual environments: design and human performance evaluation. In: Haptics Symposium 2010 IEEE (2013)Google Scholar
  22. 22.
    Bailenson, J.N., Yee, N.: Virtual interpersonal touch: Haptic interaction and copresence in collaborative virtual environments. Multimed. Tools Appl. 37, 5–14 (2008)CrossRefGoogle Scholar
  23. 23.
    Oguz, S.O., Kucukyilmaz, A., Sezgin, T.M., Basdogan, C.: Haptic negotiation and role exchange for collaboration in virtual environments. In: 2010 IEEE Haptics Symposium, HAPTICS 2010, pp. 371–378 (2010)Google Scholar
  24. 24.
    Smith, J., MacLean, K.: Communicating emotion through a haptic link: Design space and methodology. Int. J. Hum Comput Stud. 65, 376–387 (2007)CrossRefGoogle Scholar
  25. 25.
    Mason, A.H., MacKenzie, C.L.: The effects of visual information about self-movement on graspforces when receiving objects in an augmented environment. In: Proceedings of 10th Symposium Haptic Interfaces Virtual Environment Teleoperator System, HAPTICS 2002 (2002)Google Scholar
  26. 26.
    Sallnäs, E., Zhai, S.: Collaboration meets Fitts’ law : passing virtual objects with and without Haptic force feedback. In: INTERACT, pp. 97–104 (2003)Google Scholar
  27. 27.
    Oakley, I., Brewster, S., Gray, P.: Can you feel the force? an investigation of haptic collaboration in shared editors. In: Proceedings of EuroHaptics (2001)Google Scholar
  28. 28.
    Levitin, D.J., Maclean, K., Mathews, M., Chu, L.: The perception of cross-modal simultaneity. Int. J. Comput. Anticip. Syst. 5, 323–329 (2000)CrossRefGoogle Scholar
  29. 29.
    MacLean, K.E.: Haptic interaction design for everyday interfaces. Rev. Hum. Factors Ergon. 4, 149–194 (2008)CrossRefGoogle Scholar
  30. 30.
    Chan, A., Maclean, K., McGrenere, J.: Learning and identifying haptic icons under workload. In: First Joint Eurohaptics Conference Symposium Haptic Interfaces Virtual Environment Teleoperator Systems, World Haptics Conference (2005)Google Scholar
  31. 31.
    Cockburn, A., Mckenzie, B., Zealand, N.: Evaluating the effectiveness of spatial memory in 2D and 3D physical and virtual environments. In: CHI 2002 (2002)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Electrical and Computer Engineering DepartmentUniversity of WaterlooWaterlooCanada

Personalised recommendations