Fidelity and Multimodal Interactions

  • Bill KapralosEmail author
  • Fuad Moussa
  • Karen Collins
  • Adam Dubrowski
Part of the Advances in Game-Based Learning book series (AGBL)


Often, designers and developers of serious games (and virtual simulations in general) strive for high fidelity (realism). However, real-time high-fidelity rendering of complex environments across sensory modalities such as vision, audition, and haptic (sense of touch) is still beyond our computational reach. Previous work has demonstrated that multimodal effects can be considerable, to the extent that a large amount of detail of one sense may be ignored or enhanced by the presence of other sensory inputs. Taking advantage of such multimodal effects, perceptual-based rendering—whereby the rendering parameters are adjusted based on the perceptual system—can be employed to limit computational processing. Motivated by the general lack of emphasis given to the understanding of audio rendering in virtual environments and games, we have started investigating multimodal (audiovisual) interactions within such virtual environments. Our work has shown that sound can directly affect visual fidelity perception and task performance within a virtual environment. These effects can be very individualized, whereby the influence of sound is dependent on various individual factors including musical listening preferences, suggesting the importance of individualizing the virtual environment to each user. This chapter begins with an overview of virtual environments and serious gaming’s open problems, with an emphasis on fidelity, and multimodal interactions, and the implications that these may have on performance and computational requirements. A detailed summary of our own prior work will be provided along with insight and suggestions that may guide designers and developers of serious games and virtual learning environments in general. Although the chapter is contextualized in the use of serious games in health professions education, the information provided is generalizable across a variety of domains.


Serious gaming Virtual simulation Fidelity Realism Audiovisual cue interaction, multimodal interaction 



This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Social Sciences and Humanities Research Council of Canada (SSHRC), Interactive & Multi-Modal Experience Research Syndicate (IMMERSe) initiative, and the Canadian Network of Centres of Excellence (NCE), Graphics, Animation, and New Media (GRAND) initiative.


  1. Alexander, A. L., Brunyé, T., Sidman, J., & Weil, S. A. (2005). From gaming to training: A review of studies on fidelity, immersion, presence, and buy-in and their effects on transfer in PC-based simulations and games. DARWARS Training Impact Group, November. Retrieved from CiteSeer.Google Scholar
  2. Anyanwu, E. G. (2015). Background music in the dissection laboratory. Impact on stress associated with the dissection experience. Advances in Physiology Education, 39(2), 96–10.CrossRefGoogle Scholar
  3. Avila, C., Furnham, A., & McClelland, A. (2012). The influence of distracting familiar vocal music on cognitive performance of introverts and extraverts. Psychology of Music, 40(1), 84–93.CrossRefGoogle Scholar
  4. Becker, K., & Parker, J. (2012). The guide to computer simulations and games. Indianapolis, IN: Wiley.Google Scholar
  5. Blascovich, J., & Bailenson, J. (2011). Infinite reality. New York, NY: Harper Collins.Google Scholar
  6. Bonneel, N., Suied, C., Viaud-Delmon, I., & Drettakis, G. (2010). Bimodal perception of audio-visual material properties for virtual environments. ACM Transactions on Applied Perception, 7(1), 1–16.CrossRefGoogle Scholar
  7. Bracken, C., & Skalski, P. (2006, August). Presence and video games: The impact of image quality and skill level. Paper presented at the 9th Annual International Workshop on Presence. Retrieved from
  8. Cannon-Bowers, J. (2006, March). The state of gaming and simulation. Paper presented at the Training 2006 Conference and Expo.Google Scholar
  9. Chalmers, A., & Debattista, K. (2009, March). Levels of realism for serious games. Paper presented at the 2009 Conference in Games and Virtual Worlds for Serious Applications. doi: 10.1109/VS-GAMES.2009.43.
  10. Chandler, T., Anthony, M., & Klinger, D. (2009, June). Applying high cognitive vs, high physical fidelity within serious games. Paper presented at the Interservice/Industry Training, Simulation, and Education Conference.Google Scholar
  11. Cheng, K., & Cairns, P. A. (2005, April). Behaviour, realism and immersion in games. Paper presented at the CHI ’05 Extended Abstracts on Human Factors in Computing Systems. doi: 10.1145/1056808.1056894.
  12. Cook, D. A., Hamstra, S. J., Brydges, R., Zendejas, B., Szostek, J. H., Wang, A. T., et al. (2013). Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis. Medical Teacher, 35(1), e867–e898.CrossRefGoogle Scholar
  13. Cooke, N. J., & Shope, S. M. (2004). Designing a synthetic task environment. In S. G. Schiflett, L. R. Elliott, E. Salas, & M. D. Coovert (Eds.), Scaled worlds: Development, validation, and application (pp. 263–278). Surry, England: Ashgate.Google Scholar
  14. Cowan, B., Rojas, D., Kapralos, B., Collins, K., & Dubrowski, A. (2013, June). Spatial sound and its effect on visual quality perception and task performance within a virtual environment. Paper presented at the 21st International Congress on Acoustics. doi:
  15. Cowan, B., Rojas, D., Kapralos, B., Moussa, F., & Dubrowski, A. (2015). Effects of sound on visual realism perception and task performance. Visual Computer, 31(9), 1207–1216.CrossRefGoogle Scholar
  16. Cowan, B., Sabri, H., Kapralos, B., Porte, M., Backstein, D., Cristancho, S., et al. (2010). A serious game for total knee arthroplasty procedure education and training. Journal of Cybertherapy and Rehabilitation, 3(3), 285–298.Google Scholar
  17. de Ribaupierre, S., Kapralos, B., Haji, F., Stroulia, E., Dubrowski, A., & Eagleson, R. (2014). Healthcare training enhancement through virtual reality and serious games. In M. Ma, C. Lakhmi, L. Jain, & P. Anderson (Eds.), Virtual, augmented reality and serious games for healthcare (pp. 9–27). Berlin, Germany: Springer.CrossRefGoogle Scholar
  18. Driskell, J. E., Johnston, J. H., & Salas, E. (2001). Does stress training generalize to novel settings? Human Factors, 43(1), 99–110.CrossRefGoogle Scholar
  19. Furnham, A., & Strbac, L. (2002). Music is as distracting as noise: The differential distraction of background music and noise on the cognitive test performance of introverts and extraverts. Ergonomics, 45(3), 203–217.CrossRefGoogle Scholar
  20. Godden, D. R., & Baddeley, A. D. (1975). Context dependent memory in two natural environments: On land and underwater. British Journal of Psychology, 66(3), 325–331.CrossRefGoogle Scholar
  21. Gopher, D. (2006). Emphasis change as a training protocol for high demand tasks. In A. Kramer, D. Wiegman, & A. Kirlik (Eds.), Attention: From theory to practice (pp. 209–224). New York, NY: Oxford Psychology Press.CrossRefGoogle Scholar
  22. Gosen, J., & Washbush, J. (2004). A review of scholarship on assessing experiential learning effectiveness. Simulation & Gaming, 35(2), 270–293.CrossRefGoogle Scholar
  23. Hays, R. T. (2005). The effectiveness of instructional games: A literature review and discussion (Technical Report 2005-004). Orlando, FL: Naval Air Warfare Center, Training Systems Division.Google Scholar
  24. Hays, R. T., & Singer, M. (1989). Simulation fidelity in training system design. New York, NY: Springer.CrossRefGoogle Scholar
  25. Hulusic, V., Debattista, K., Aggarwal, V., & Chalmers, A. (2011). Maintaining frame rate perception in interactive environments by exploiting audio-visual cross-modal interaction. Visual Computer, 27(1), 57–66.CrossRefGoogle Scholar
  26. Hulusic, V., Harvey, C., Debattista, K., Tsingos, N., Walker, S., Howard, D., et al. (2012). Acoustic rendering and auditory-visual cross-modal perception and interaction. Computer Graphics Forum, 31(1), 102–131.CrossRefGoogle Scholar
  27. Isaranuwatchai, W., Brydges, R., Carnahan, H., Backstein, D., & Dubrowski, A. (2014). Comparing the cost-effectiveness of simulation modalities: A case study of peripheral intravenous catheterization training. Advances in Health Sciences Education, 19(2), 219–232.CrossRefGoogle Scholar
  28. Kapralos, B., Moussa, F., & Dubrowski, A. (2014). An overview of virtual simulations and serious games for surgical education and training. In M. Ma, C. Lakhmi, L. Jain, & P. Anderson (Eds.), Virtual, augmented reality and serious games for healthcare (pp. 289–306). Berlin, Germany: Springer.Google Scholar
  29. Kapralos, B., Shewaga, R., & Ng, G. (2014). Serious games: Customizing the audio-visual interface. In R. Shumaker, & S. Lackey (Eds.), Virtual, augmented and mixed reality. Virtual, augmented and mixed reality. Applications of virtual and augmented reality. Lecture notes in computer science (Vol. 8526, pp. 190–199).Google Scholar
  30. Ker, J., & Bradley, P. (2010). Simulation in medical education. In T. Swanwick (Ed.), Understanding medical education: Evidence, theory and practice (pp. 164–190). West Sussex, UK: Wiley-Blackwell.CrossRefGoogle Scholar
  31. Larsson, P., Västjäll, D., & Kleiner, M. (2003, April). On the quality of experience: A multi-modal approach to perceptual ego-motion and sensed presence in virtual environments. Paper presented at the First International Speech Communications Association Tutorial and Research Workshop on Auditory Quality of Systems. Retrieved from Scholar
  32. Lee, C., Rincon, G. A., Meyer, G., Höllerer, T., & Bowman, D. A. (2013). The effects of visual realism on search tasks in mixed reality simulation. IEEE Transactions on Visual Computing and Graphics, 19(4), 547–556.CrossRefGoogle Scholar
  33. Mastoropoulou, G., Debattista, K., Chalmers, A., & Troscianco, T. (2005, August). The influence of sound effects on the perceived smoothness of rendered animations. Paper presented at the 2nd Symposium on Applied Perception in Graphics and Visualization. doi: 10.1145/1080402.1080404.
  34. Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychology, 38(1), 43–52.CrossRefGoogle Scholar
  35. McGaghie, W. C., Issenberg, S. B., Petrusa, E. R., & Scalese, R. J. (2010). A critical review of simulation-based medical education research: 2003–2009. Medical Educator, 44(1), 50–63.CrossRefGoogle Scholar
  36. McMahan, R. P. (2011). Exploring the effects of higher-fidelity display and interaction for virtual reality games. Doctoral dissertation. Retrieved from Digital Libraries and Archives (etd-12162011-140224).Google Scholar
  37. Morris, C. S., Hancock, P. A., & Shirkey, E. C. (2004). Motivational effects of adding context relevant stress in PC-based game training. Military Psychology, 16(1), 135–147.CrossRefGoogle Scholar
  38. Muchinsky, P. M. (1989). Psychology applied to work. Summerfield, NC: Hypergraphic Press.Google Scholar
  39. Netepezuk, D. W. (2013, July). Immersion and realism in video games—The confused moniker of video game engrossment. Paper presented at the 18th International Conference on Computer Games. doi: 10.1109/CGames.2013.6632613.
  40. Norman, G., Dore, K., & Grierson, L. (2012). The minimal relationship between simulation fidelity and transfer of learning. Medical Educator, 46(7), 636–647.CrossRefGoogle Scholar
  41. Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1–4.CrossRefGoogle Scholar
  42. Rademacher, P., Lengyel, J., Cutrell, E., & Whitted, T. (2001). Measuring the perception of visual realism in images. In S. J. Gortler, & K. Myszkowski (Eds.), Proceedings of the 12th Eurographics Workshop on Rendering Techniques (pp. 235–248): London, UK: Springer.Google Scholar
  43. Rojas, D., Cowan, B., Kapralos, B., Collins, K., & Dubrowski, A. (2015, June). The effect of sound on visual quality perception and task completion time in a cel-shaded serious gaming virtual environment. Paper presented at the 7th IEEE International Workshop on Quality of Multimedia Experience. doi: 10.1109/QoMEX.2015.7148136.
  44. Rojas, D., Kapralos, B., Collins, K., & Dubrowski, A. (2014). The effect of contextual sound cues on visual fidelity perception. Studies in Health Technology and Informatics, 196, 346–352.Google Scholar
  45. Rojas, D., Kapralos, B., Cristancho, S., Collins, K., Hogue, A., Conati, C., et al. (2012). Developing effective serious games: The effect of background sound on visual fidelity perception with varying texture resolution. Studies in Health Technology and Informatics, 173, 386–392.Google Scholar
  46. Rojas, D., Kapralos, B., Crsitancho, S., Collins, K., Conati, C., & Dubrowski, A. (2011, September). The effect of background sound on visual fidelity perception. Paper presented at the ACM Audio Mostly 2011 Conference—6th Conference on Interaction with Sound. doi: 10.1145/2095667.2095675.
  47. Rojas, D., Kapralos, B., Hogue, A., Collins, K., Nacke, L., Crsitancho, C., et al. (2013). The effect of ambient auditory conditions on visual fidelity perception in stereoscopic 3D. IEEE Transactions on Systems, Man, and Cybernetics Part B: Cybernetics, 43(6), 1572–1583.Google Scholar
  48. Seitz, A. R., van Wassenhove, V., & Shams, L. (2007). Simultaneous and independent acquisition of multisensory and unisensory associations. Perception, 36(10), 1445–1453.CrossRefGoogle Scholar
  49. Shams, L., & Kim, R. (2010). Crossmodal influences on visual perception. Physics Life Reviews, 7(3), 295–298.CrossRefGoogle Scholar
  50. Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning. In U. Ritterfeld, M. Cody, & P. Vorderer (Eds.), Serious games: Mechanisms and effects (pp. 295–321). New York, NY: Routedle.Google Scholar
  51. Slater, M., Khanna, P., Mortensen, J., & Yu, I. (2009). Visual realism enhances realistic response in an immersive virtual environment. IEEE Computer Graphics and Applications, 29(3), 76–84.CrossRefGoogle Scholar
  52. Slater, M., & Wilbur, M. (1997). A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence Teleoperators and Virtual Environments, 6(6), 603–616.CrossRefGoogle Scholar
  53. Squire, K., & Jenkins, H. (2003). Harnessing the power of games in education. Insight, 3, 5–33.Google Scholar
  54. Storms, S. L., & Zyda, M. J. (2000). Interactions in perceived quality of auditory-visual displays. Presence Teleoperators and Virtual Environments, 9(6), 557–580.CrossRefGoogle Scholar
  55. Sweller, J., Ayres, P. L., Kalyuga, S., & Chandler, P. A. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31.CrossRefGoogle Scholar
  56. Tashiro, J., & Dunlap, D. (2007, November). The impact of realism on learning engagement in educational games. Paper presented at the ACM 2007 Conference on Future Play. doi: 10.1145/1328202.1328223.
  57. Treisman, A. M., & Riley, J. G. (1969). Is selective attention selective perception or selective response? A further test. Journal of Experimental Psychology, 79(1), 27–34.CrossRefGoogle Scholar
  58. Veinott, E. S., Perleman, B., Polander, E., Leonard, J., Berry, G., Catrambone, R., et al. (2014, October). Is more information better? Examining the effects of visual and cognitive fidelity on learning in a serious video game. Paper presented at the 2014 IEEE Games Entertainment and Media Conference. doi: 10.1109/GEM.2014.7048105.

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • Bill Kapralos
    • 1
    Email author
  • Fuad Moussa
    • 2
  • Karen Collins
    • 3
  • Adam Dubrowski
    • 4
  1. 1.Faculty of Business and Information TechnologyUniversity of Ontario Institute of TechnologyOshawaCanada
  2. 2.Division of Cardiac and Vascular Surgery, Schulich Heart Centre, Sunnybrook Health Sciences CentreTorontoCanada
  3. 3.The Games Institute, University of WaterlooWaterlooCanada
  4. 4.Divisions of Emergency Medicine and Pediatrics, Faculty of MedicineMemorial UniversitySt. John’sCanada

Personalised recommendations