The Visual Computer

, Volume 31, Issue 9, pp 1207–1216 | Cite as

Effects of sound on visual realism perception and task performance

  • Brent Cowan
  • David Rojas
  • Bill Kapralos
  • Fuad Moussa
  • Adam Dubrowski
Original Article


Before the application of virtual simulations and serious games for surgical education and training becomes more widespread, there are a number of open questions and issues that must be addressed including the relationship between realism, multi-modal cue interaction, immersion, and knowledge transfer and retention. Using the serious game surgical cognitive education and training framework developed specifically for cognitive surgical skills training, here we examine the effect of sound on visual realism perception and task completion time while performing a task within a virtual environment. Our preliminary experimental results indicate that the appropriate use of sound can lead to performance improvements when performing a task within a virtual environment without a corresponding decrease in the perception of visual realism.


Serious games Virtual simulation  Visual realism Audiovisual cue interaction 



This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Social Sciences and Humanities Research Council of Canada (SSHRC), Interactive & Multi-Modal Experience Research Syndicate (IMMERSe) initiative, and the Canadian Network of Centres of Excellence (NCE), Graphics, Animation, and New Media (GRAND) initiative.


  1. 1.
    Halsted, W.S.: The Training of the Surgeon, vol. xv, pp. 267–275. Bull Johns Hopkins Hosp, USA (1904)Google Scholar
  2. 2.
    Reznick, R.K.: Teaching and testing technical skills. Am. J. Surg. 165(3), 358–361 (1993)CrossRefGoogle Scholar
  3. 3.
    Gorman, P.J., Meier, A.H., Rawn, C., Krummel, T.M.: The future of medical education is no longer blood and guts, it is bits and bytes. Am. J. Surg. 180, 353–356 (2000)CrossRefGoogle Scholar
  4. 4.
    Kohls-Gatzoulis, J.A., Regehr, G., Hutchison, C.: Teaching cognitive skills improves learning in surgical skills courses: a blinded, prospective, randomized study. Can. J. Surg. 47(4), 277–283 (2004)Google Scholar
  5. 5.
    Corti, K.: Game-Based Learning: A Serious Business Application. PIXE-Learning, Coventry (2006)Google Scholar
  6. 6.
    Shute, V., Ventura, M., Bauer, M., Zapata-Rivera, D.: Melding the power of serious games and embedded assessment to monitor and foster learning. In: Ritterfeld, U., Cody, M.J., Vorderer, P. (eds.) Serious Games. Mechanisms and Effects, pp. 295–321 (2009)Google Scholar
  7. 7.
    Tashiro, J., Dunlap, D.: The impact of realism on learning engagement in educational games. In: Proceedings of the ACM Future Play 2007 Conference, Toronto, pp. 113–120 (2007)Google Scholar
  8. 8.
    Farmer, E., Rooij, J., Riemersma, J., Joma, P., Morall, J.: Handbook of Simulator Based Training. Ashgate Publishing, Surrey (1999)Google Scholar
  9. 9.
    Cook, D.A., Hamstra, S.J., Brydges, R., Zendejas, B., Szostek, J.H., Wang, A.T., Erwin, P.J., Hatala, R.: Comparative effectiveness of instructional design features in simulation-based education: systematic review and meta-analysis. Med. Teach. 35(1), e867–e898 (2013)CrossRefGoogle Scholar
  10. 10.
    Muchinsky, P.M.: Psychology Applied to Work, 9th edn. Hypergraphic Press, Summerfield (1999)Google Scholar
  11. 11.
    Godden, D.R., Baddeley, A.D.: Context dependent memory in two natural environments: on land and underwater. Brit. J. Psychol. 66(3), 325–331 (1975)CrossRefGoogle Scholar
  12. 12.
    Norman, G., Dore, K., Grierson, L.: The minimal relationship between simulation fidelity and transfer of learning. Med. Educ. 46(7), 636–647 (2012)CrossRefGoogle Scholar
  13. 13.
    Hulusic, V., Aranha, M., Chalmers, A.: The influence of cross-modal interaction on perceived rendering quality thresholds. In: Proceedings of Computer Graphics, Visualization and Vision 2008, Plzen - Bory, Czech Republic, pp. 41–48 (2008)Google Scholar
  14. 14.
    Blascovich, J., Bailenson, J.: Infinite Reality. Harper Collins, New York (2011)Google Scholar
  15. 15.
    Howard, I.P., Templeton, W.B.: Human Spatial Orientation. John Wiley and Sons, Ney York (1966)Google Scholar
  16. 16.
    McGurk, H., Macdonald, J.: Hearing lips and seeing voices. Nature 264, 746–748 (1976)CrossRefGoogle Scholar
  17. 17.
    Rojas, D., Kapralos, B., Crsitancho, S., Collins, K., Conati, C., Dubrowski, A.: The effect of background sound on visual fidelity perception. In: Proceedings of the ACM Audio Mostly Conference 2011, Coimbra (2011)Google Scholar
  18. 18.
    Rojas, D., Kapralos, B., Cristancho, S., Collins, K., Hogue, A., Conati, C., Dubrowski, A.: Developing effective serious games: the effect of background sound on visual fidelity perception with varying texture resolution. Stud. Health Technol. Inform. 173, 386–392 (2012)Google Scholar
  19. 19.
    Rojas, D., Kapralos, B., Hogue, A., Collins, K., Nacke, L., Crsitancho, S., Conati, C., Dubrowski, A.: The effect of ambient auditory conditions on visual fidelity perception in stereoscopic 3D. IEEE Trans. Syst. Man. Cy. B 43(6), 1572–1583 (2013)Google Scholar
  20. 20.
    Rojas, D., Kapralos, B., Collins, K., Dubrowski, A.: The effect of contextual sound cues on visual fidelity perception. Stud. Health Technol. Inform. 196, 346–352 (2014)Google Scholar
  21. 21.
    Cowan, B., Sabri, H., Kapralos, B., Porte, M., Backstein, D., Cristancho, S., Dubrowski, A.: A serious game for total knee arthroplasty procedure education and training. J. Cyberther. Rehabil. 3(3), 285–298 (2010)Google Scholar
  22. 22.
    Kapralos, B., Moussa, F., Dubrowski, A.: An overview of virtual simulations and serious games for surgical education and training. In Brooks, A., Braham, S., Jain, L. (eds.) Serious Games, Alternative Realities, and Play Therapy. Springer Series Studies in Computational Intelligence Ch.14, pp. 289–306 (2014)Google Scholar
  23. 23.
    Mastoropoulou, G., Debattista, K., Chalmers, A., Troscianco, T.: The influence of sound effects on the perceived smoothness of rendered animations. In: Proceedings of the 2nd Symposium on Applied Perception in Graphics and Visualization, La Coruna, pp. 9–15 (2005)Google Scholar
  24. 24.
    Larsson, P., Västjäll, D., Kleiner, M.: On the quality of experience: a multi-modal approach to perceptual ego-motion and sensed presence in virtual environments. In: Proceedings of the First International Speech Communications Association Tutorial and Research Workshop on Auditory Quality of Systems, Akademie Mont-Cenis (2003)Google Scholar
  25. 25.
    Bertelson, P., Radeau, M.: Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept. Psychophys. 29(6), 578–584 (1981)CrossRefGoogle Scholar
  26. 26.
    Hulusic, V., Debattista, K., Aggarwal, V., Chalmers, A.: Maintaining frame rate perception in interactive environments by exploiting audio-visual cross-modal interaction. Visual Comput. 27(1), 57–66 (2011)CrossRefGoogle Scholar
  27. 27.
    Hulusic, V., Czanner, G., Debattista, K., Sikudova, E., Dubla, P., Chalmers.: Investigation of the beat rate effect on frame rate for animated content. In: Proceedings of the 25th Spring Conference on Computer Graphics, Budmerice (2009)Google Scholar
  28. 28.
    Bonneel, N., Suied, C., Viaud-Delmon, I., Drettakis, G.: Bimodal perception of audio-visual material properties for virtual environments. ACM Trans. Appl. Percept. 7(1), 1–16 (2010)CrossRefGoogle Scholar
  29. 29.
    Woods, A.T., Poliakoff, E., Lloyd, D.M., Kuenzela, J., Hodsona, J.R., Gondaa, H., Batchelora, J., Dijksterhuis, G.B., Thomas, A.: Effect of background noise on food perception. Food Qual. Prefer. 22(1), 42–47 (2011)CrossRefGoogle Scholar
  30. 30.
    Chang, R.S., Thompson, N.S.: The attention-getting capacity of whines and child-directed speech. Evol. Psychol. 8(2), 260–274 (2010)CrossRefGoogle Scholar
  31. 31.
    Conrad, C., Konuk, Y., Werner, P., Cao, C.G., Warshaw, A., Rattner, D., Jones, D.B., Gee, D.: The effect of define auditory conditions versus mental loading on the laparoscopic motor skill performance of experts. Surg. Endosc. 24(6), 1347–1352 (2010)CrossRefGoogle Scholar
  32. 32.
    Praamsma, M., Carnahan, H., Backstein, D., Veillette, C.J., Gonzalez, D., Dubrowski, A.: Drilling sounds are used by surgeons and intermediate residents, but not novice orthopedic trainees, to guide drilling motions. Can. J. Surg. 51(6), 442–446 (2008)Google Scholar
  33. 33.
    Hulusic, V., Harvey, C., Debattista, K., Tsingos, N., Walker, S., Howard, D., Chalmers, A.: Acoustic rendering and auditory-visual cross-modal perception and interaction. Comput. Graph. Forum 31(1), 102–131 (2012)CrossRefGoogle Scholar
  34. 34.
    Shams, L., Kim, R.: Crossmodal influences on visual perception. Phys. Life Rev. 7(3), 295–298 (2010)CrossRefGoogle Scholar
  35. 35.
    Morfey, C.: The Dictionary of Acoustics. Academic Press, San Diego (2001)Google Scholar
  36. 36.
    Suied, C., Bonneel, N., Viaud-Delmon, I.: Integration of auditory and visual information in the recognition of realistic objects. Exp. Brain Res. 194(1), 91–102 (2009)CrossRefGoogle Scholar
  37. 37.
    Treisman, A.M., Riley, J.G.: Is selective attention selective perception or selective response? A further test. J. Exp. Psychol. 79(1), 27–34 (1969)CrossRefGoogle Scholar
  38. 38.
    Paas, F., Renkl, A., Sweller, J.: Cognitive load theory and instructional design: recent developments. Educ. Psychol. 38(1), 1–4 (2003)CrossRefGoogle Scholar
  39. 39.
    Faure, F., Duriez, C., Delingette, H., Allard, J., Gilles, B., Marchesseau, Talbot, H., Courtecuisse, H., Bousquet, G., Peterlik, I., Cotin, S.: SOFA: a multi-model framework for interactive physical simulation. In: Payan, Y. (ed.) Soft Tissue Biomechanical Modeling for Computer Assisted Surgery, vol. 11, pp. 283–321. Springer, Berlin (2012)CrossRefGoogle Scholar
  40. 40.
    France, L., Lenoir, J., Angelidis, A., Meseure, P., Cani, M.P., Faure, F., Chaillou, C.: A layered model of a virtual human intestine for surgery simulation. Med Image Anal. 9(2), 123–132 (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Brent Cowan
    • 1
  • David Rojas
    • 1
    • 2
  • Bill Kapralos
    • 1
  • Fuad Moussa
    • 3
  • Adam Dubrowski
    • 4
  1. 1.Faculty of Business and Information Technology, Health Education Technology Research Unit (HETRU)University of Ontario Institute of TechnologyOshawaCanada
  2. 2.Institute of Medical ScienceUniversity of TorontoTorontoCanada
  3. 3.Division of Cardiac and Vascular SurgerySchulich Heart Centre, Sunnybrook Health Sciences CentreTorontoCanada
  4. 4.Divisions of Emergency Medicine and PediatricsFaculty of Medicine, Memorial UniversitySt. John’sCanada

Personalised recommendations