Fidelity and Multimodal Interactions
Often, designers and developers of serious games (and virtual simulations in general) strive for high fidelity (realism). However, real-time high-fidelity rendering of complex environments across sensory modalities such as vision, audition, and haptic (sense of touch) is still beyond our computational reach. Previous work has demonstrated that multimodal effects can be considerable, to the extent that a large amount of detail of one sense may be ignored or enhanced by the presence of other sensory inputs. Taking advantage of such multimodal effects, perceptual-based rendering—whereby the rendering parameters are adjusted based on the perceptual system—can be employed to limit computational processing. Motivated by the general lack of emphasis given to the understanding of audio rendering in virtual environments and games, we have started investigating multimodal (audiovisual) interactions within such virtual environments. Our work has shown that sound can directly affect visual fidelity perception and task performance within a virtual environment. These effects can be very individualized, whereby the influence of sound is dependent on various individual factors including musical listening preferences, suggesting the importance of individualizing the virtual environment to each user. This chapter begins with an overview of virtual environments and serious gaming’s open problems, with an emphasis on fidelity, and multimodal interactions, and the implications that these may have on performance and computational requirements. A detailed summary of our own prior work will be provided along with insight and suggestions that may guide designers and developers of serious games and virtual learning environments in general. Although the chapter is contextualized in the use of serious games in health professions education, the information provided is generalizable across a variety of domains.
KeywordsSerious gaming Virtual simulation Fidelity Realism Audiovisual cue interaction, multimodal interaction
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Social Sciences and Humanities Research Council of Canada (SSHRC), Interactive & Multi-Modal Experience Research Syndicate (IMMERSe) initiative, and the Canadian Network of Centres of Excellence (NCE), Graphics, Animation, and New Media (GRAND) initiative.
- Alexander, A. L., Brunyé, T., Sidman, J., & Weil, S. A. (2005). From gaming to training: A review of studies on fidelity, immersion, presence, and buy-in and their effects on transfer in PC-based simulations and games. DARWARS Training Impact Group, November. Retrieved from CiteSeer.Google Scholar
- Becker, K., & Parker, J. (2012). The guide to computer simulations and games. Indianapolis, IN: Wiley.Google Scholar
- Blascovich, J., & Bailenson, J. (2011). Infinite reality. New York, NY: Harper Collins.Google Scholar
- Bracken, C., & Skalski, P. (2006, August). Presence and video games: The impact of image quality and skill level. Paper presented at the 9th Annual International Workshop on Presence. Retrieved from http://ispr.info/presence-conferences/previous-conferences/presence-2006/
- Cannon-Bowers, J. (2006, March). The state of gaming and simulation. Paper presented at the Training 2006 Conference and Expo.Google Scholar
- Chalmers, A., & Debattista, K. (2009, March). Levels of realism for serious games. Paper presented at the 2009 Conference in Games and Virtual Worlds for Serious Applications. doi: 10.1109/VS-GAMES.2009.43.
- Chandler, T., Anthony, M., & Klinger, D. (2009, June). Applying high cognitive vs, high physical fidelity within serious games. Paper presented at the Interservice/Industry Training, Simulation, and Education Conference.Google Scholar
- Cheng, K., & Cairns, P. A. (2005, April). Behaviour, realism and immersion in games. Paper presented at the CHI ’05 Extended Abstracts on Human Factors in Computing Systems. doi: 10.1145/1056808.1056894.
- Cooke, N. J., & Shope, S. M. (2004). Designing a synthetic task environment. In S. G. Schiflett, L. R. Elliott, E. Salas, & M. D. Coovert (Eds.), Scaled worlds: Development, validation, and application (pp. 263–278). Surry, England: Ashgate.Google Scholar
- Cowan, B., Rojas, D., Kapralos, B., Collins, K., & Dubrowski, A. (2013, June). Spatial sound and its effect on visual quality perception and task performance within a virtual environment. Paper presented at the 21st International Congress on Acoustics. doi:http://dx.doi.org/10.1121/1.4798377.
- Cowan, B., Sabri, H., Kapralos, B., Porte, M., Backstein, D., Cristancho, S., et al. (2010). A serious game for total knee arthroplasty procedure education and training. Journal of Cybertherapy and Rehabilitation, 3(3), 285–298.Google Scholar
- de Ribaupierre, S., Kapralos, B., Haji, F., Stroulia, E., Dubrowski, A., & Eagleson, R. (2014). Healthcare training enhancement through virtual reality and serious games. In M. Ma, C. Lakhmi, L. Jain, & P. Anderson (Eds.), Virtual, augmented reality and serious games for healthcare (pp. 9–27). Berlin, Germany: Springer.CrossRefGoogle Scholar
- Hays, R. T. (2005). The effectiveness of instructional games: A literature review and discussion (Technical Report 2005-004). Orlando, FL: Naval Air Warfare Center, Training Systems Division.Google Scholar
- Kapralos, B., Moussa, F., & Dubrowski, A. (2014). An overview of virtual simulations and serious games for surgical education and training. In M. Ma, C. Lakhmi, L. Jain, & P. Anderson (Eds.), Virtual, augmented reality and serious games for healthcare (pp. 289–306). Berlin, Germany: Springer.Google Scholar
- Kapralos, B., Shewaga, R., & Ng, G. (2014). Serious games: Customizing the audio-visual interface. In R. Shumaker, & S. Lackey (Eds.), Virtual, augmented and mixed reality. Virtual, augmented and mixed reality. Applications of virtual and augmented reality. Lecture notes in computer science (Vol. 8526, pp. 190–199).Google Scholar
- Larsson, P., Västjäll, D., & Kleiner, M. (2003, April). On the quality of experience: A multi-modal approach to perceptual ego-motion and sensed presence in virtual environments. Paper presented at the First International Speech Communications Association Tutorial and Research Workshop on Auditory Quality of Systems. Retrieved from http://www.isca_speech.org/archive_open/aqs2003Google Scholar
- Mastoropoulou, G., Debattista, K., Chalmers, A., & Troscianco, T. (2005, August). The influence of sound effects on the perceived smoothness of rendered animations. Paper presented at the 2nd Symposium on Applied Perception in Graphics and Visualization. doi: 10.1145/1080402.1080404.
- McMahan, R. P. (2011). Exploring the effects of higher-fidelity display and interaction for virtual reality games. Doctoral dissertation. Retrieved from Digital Libraries and Archives (etd-12162011-140224).Google Scholar
- Muchinsky, P. M. (1989). Psychology applied to work. Summerfield, NC: Hypergraphic Press.Google Scholar
- Netepezuk, D. W. (2013, July). Immersion and realism in video games—The confused moniker of video game engrossment. Paper presented at the 18th International Conference on Computer Games. doi: 10.1109/CGames.2013.6632613.
- Rademacher, P., Lengyel, J., Cutrell, E., & Whitted, T. (2001). Measuring the perception of visual realism in images. In S. J. Gortler, & K. Myszkowski (Eds.), Proceedings of the 12th Eurographics Workshop on Rendering Techniques (pp. 235–248): London, UK: Springer.Google Scholar
- Rojas, D., Cowan, B., Kapralos, B., Collins, K., & Dubrowski, A. (2015, June). The effect of sound on visual quality perception and task completion time in a cel-shaded serious gaming virtual environment. Paper presented at the 7th IEEE International Workshop on Quality of Multimedia Experience. doi: 10.1109/QoMEX.2015.7148136.
- Rojas, D., Kapralos, B., Collins, K., & Dubrowski, A. (2014). The effect of contextual sound cues on visual fidelity perception. Studies in Health Technology and Informatics, 196, 346–352.Google Scholar
- Rojas, D., Kapralos, B., Cristancho, S., Collins, K., Hogue, A., Conati, C., et al. (2012). Developing effective serious games: The effect of background sound on visual fidelity perception with varying texture resolution. Studies in Health Technology and Informatics, 173, 386–392.Google Scholar
- Rojas, D., Kapralos, B., Crsitancho, S., Collins, K., Conati, C., & Dubrowski, A. (2011, September). The effect of background sound on visual fidelity perception. Paper presented at the ACM Audio Mostly 2011 Conference—6th Conference on Interaction with Sound. doi: 10.1145/2095667.2095675.
- Rojas, D., Kapralos, B., Hogue, A., Collins, K., Nacke, L., Crsitancho, C., et al. (2013). The effect of ambient auditory conditions on visual fidelity perception in stereoscopic 3D. IEEE Transactions on Systems, Man, and Cybernetics Part B: Cybernetics, 43(6), 1572–1583.Google Scholar
- Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning. In U. Ritterfeld, M. Cody, & P. Vorderer (Eds.), Serious games: Mechanisms and effects (pp. 295–321). New York, NY: Routedle.Google Scholar
- Squire, K., & Jenkins, H. (2003). Harnessing the power of games in education. Insight, 3, 5–33.Google Scholar
- Tashiro, J., & Dunlap, D. (2007, November). The impact of realism on learning engagement in educational games. Paper presented at the ACM 2007 Conference on Future Play. doi: 10.1145/1328202.1328223.
- Veinott, E. S., Perleman, B., Polander, E., Leonard, J., Berry, G., Catrambone, R., et al. (2014, October). Is more information better? Examining the effects of visual and cognitive fidelity on learning in a serious video game. Paper presented at the 2014 IEEE Games Entertainment and Media Conference. doi: 10.1109/GEM.2014.7048105.