Interaction and Communication in an Immersive Learning Game: The Challenges of Modelling Real-Time Collaboration in a Virtual Operating Room



In this chapter, we describe the methodology we have engineered during the design process of the collaborative and immersive learning game 3D Virtual Operating Room. The game targets an audience of practitioners involved in the operating room and the training consists in virtually re-enacting typical perioperative activities so as to learn or improve skills related to patient safety. The challenges faced in this project include multiplayer collaboration in a shared, interactive and dynamically evolving virtual environment, and modelling educational scenarios on the basis of actual observations inside the operating room. The model we detail is grounded on a semantic definition of the environment which allowed for three innovative features. A game-mediated communication system where information pertaining to the game is exchanged in real time by the players. AI-controlled characters replacing missing players as fully equal partners. And, the ability for the game to provide feedback in real time or during a debriefing on the team’s performance against predefined pedagogical objectives.


Immersive learning game Design methodology Multiplayer collaboration In-game communication Business process modelling and notation 



3DVOR is supported by the 12th innovation cluster French funding scheme “Fonds Unique Interministériel” (FUI). 3DVOR is a collaborative research project between KTM Advance, Novamotion, the University of Toulouse, Toulouse Hospital and the University JF Champollion. The steering committee of 3DVOR is composed of Pr. Pierre Lagarrigue, M.D. Ph.D. Vincent Lubrano, M.D. Ph.D. Vincent Minville and Catherine Pons-Lelardeux. The scenarios used in the game were designed by Thomas Rodsphon and Vincent Lubrano from Toulouse University Hospital. The experiment described in this chapter has been conducted under the supervision of Christiane Paban, Hoang-Minh Truong and Amélie Lafitte. The authors are also grateful to the designers and programmers who have contributed to the game: Cyrielle Guimbal, Jules de Guglielmi and Romain Régis.


  1. Allweyer, T.: BPMN 2.0: Introduction to the Standard for Business Process Modeling. BoD–Books on Demand, Norderstedt (2010)Google Scholar
  2. Bellotti, F., Berta, R., De Gloria, A., Primavera, L.: Supporting authors in the development of task-based learning in serious virtual worlds. Br. J. Educ. Technol. 41 (1), 86–107 (2010)Google Scholar
  3. Carayon, P., Schultz, K., Hundt, A.S.: Righting wrong site surgery. Jt. Comm. J. Qual. Patient Saf. 30 (7), 405–410 (2004)Google Scholar
  4. Cassell, J.: Embodied Conversational Agents. MIT Press, Cambridge (2000)Google Scholar
  5. Cook, D.A., Triola, M.M.: Virtual patients: a critical literature review and proposed next steps. Med. Educ. 43 (4), 303–311 (2009)CrossRefGoogle Scholar
  6. Coulter, R., Saland, L., Caudell, T., Goldsmith, T.E., Alverson, D.: The effect of degree of immersion upon learning performance in virtual reality simulations for medical education. In: Medicine Meets Virtual Reality 15: In Vivo, in Vitro, in Silico: Designing the Next in Medicine, vol. 125, pp. 155. IOS Press, Amsterdam/Washington, DC (2007)Google Scholar
  7. Currie, L.: Fall and injury prevention. In: Hughes, R.G. (ed.) Patient safety and quality: an evidence-based handbook for nurses. Agency for Healthcare Research and Quality (US), Rockville, MD (2008). Chapter 10Google Scholar
  8. De Freitas, S.: Learning in Immersive Worlds. Joint Information Systems Committee, London (2006)Google Scholar
  9. De Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., Poulovassilis, A.: Learning as immersive experiences: using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. Br. J. Educ. Technol. 41 (1), 69–85 (2010)Google Scholar
  10. Dede, C.: Immersive interfaces for engagement and learning. Science 323 (5910), 66–69 (2009)CrossRefGoogle Scholar
  11. Djaouti, D., Alvarez, J., Jessel, J.P., Rampnoux, O.: Origins of serious games. In: Serious Games and Edutainment Applications, pp. 25–43. Springer, London/New York (2011)Google Scholar
  12. Duval, Y., Panzoli, D., Reymonet, A., Plantec, J., Thomas, J., Jessel, J.: Serious games scenario modeling for non-experts. In: CSEDU 2015 – Proceedings of the 7th International Conference on Computer Supported Education, Lisbon, vol. 1, pp. 474–479, 23–25 May 2015Google Scholar
  13. Fischler, I.S., Kaschub, C.E., Lizdas, D.E., Lampotang, S.: Understanding of anesthesia machine function is enhanced with a transparent reality simulation. Simul. Healthc. 3 (1), 26–32 (2008)CrossRefGoogle Scholar
  14. Fried, M.P., Satava, R., Weghorst, S., Gallagher, A., Sasaki, C., Ross, D., Sinanan, M., Cuellar, H., Uribe, J.I., Zeltsan, M., et al.: The use of surgical simulators to reduce errors. Adv. Patient Saf. 4 (2005)Google Scholar
  15. Gaba, D.M.: The future vision of simulation in health care. Qual. Saf. Health Care 13 (suppl 1), i2–i10 (2004)CrossRefGoogle Scholar
  16. Gallagher, A.G., Ritter, E.M., Champion, H., Higgins, G., Fried, M.P., Moses, G., Smith, C.D., Satava, R.M.: Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann. Surg. 241 (2), 364–372 (2005)CrossRefGoogle Scholar
  17. Gibson, J.J.: The Ecological Approach to Visual Perception. Houghton, Mifflin and Company, Boston (1979)Google Scholar
  18. Göbel, S., Mehm, F., Radke, S., Steinmetz, R.: 80days: adaptive digital storytelling for digital educational games. In: Proceedings of the 2nd International Workshop on Story-Telling and Educational Games (STEG’09), Aachen, vol. 498 (2009)Google Scholar
  19. Graafland, M., Schraagen, J., Schijven, M.P.: Systematic review of serious games for medical education and surgical skills training. Br. J. Surg. 99 (10), 1322–1330 (2012)CrossRefGoogle Scholar
  20. Hennigan, B.: Making the case for NLP in dialogue systems for serious games. In: 8th International Conference on Natural Language Processing (JapTAL), 1st Workshop on Games and NLP, Kanazawa (2012)Google Scholar
  21. Hughes, C.E., Moshell, J.M.: Shared virtual worlds for education: the ExploreNet experiment. Multimed. Syst. 5 (2), 145–154 (1997)CrossRefGoogle Scholar
  22. Johnson, M.T., Clary, M.: A second life virtual clinic for medical student training. In: Second Life Education Community Conference (SLEDcc’08), Tampa (2008)Google Scholar
  23. Kallmann, M., Thalmann, D.: Modeling behaviors of interactive objects for real-time virtual environments. J. Vis. Lang. Comput. 13 (2), 177–195 (2002)CrossRefGoogle Scholar
  24. Kohn, L., Corrigan, J., Donaldson, M.: To Err Is Human: Building a Safer Health System. National Academy Press, Washington, DC (2000)Google Scholar
  25. Kwaan, M.R., Studdert, D.M., Zinner, M.J., Gawande, A.A.: Incidence, patterns, and prevention of wrong-site surgery. Arch. Surg. 141 (4), 353–358 (2006)CrossRefGoogle Scholar
  26. Lagarrigue, P., Lubrano, V., Minville, V., Pons Lelardeux, C.: The 3dvor project., Online (2013)
  27. Lave, J., Wenger, E.: Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, Cambridge/New York (1991)CrossRefGoogle Scholar
  28. Lee, A., Berge, Z.L.: Second life in healthcare education: virtual environment’s potential to improve patient safety. Knowl. Manag. E-Learning: Int. J. (KM&EL) 3 (1), 17–23 (2011)Google Scholar
  29. Lee, C., Liu, A., Del Castillo, S., Bowyer, M., Alverson, D., Muniz, G., Caudell, T.: Towards an immersive virtual environment for medical team training. Stud. Health Technol. Inform. 125, 274–279 (2006)Google Scholar
  30. Lingard, L., Espin, S., Whyte, S., Regehr, G., Baker, G., Reznick, R., Bohnen, J., Orser, B., Doran, D., Grober, E.: Communication failures in the operating room: an observational classification of recurrent types and effects. Qual. Saf. Health Care 13 (5), 330–334 (2004)CrossRefGoogle Scholar
  31. Mathieu, P., Panzoli, D., Picault, S.: Virtual customers in an agent world. In: 10th International Conference on Practical Application of Agent and Multi-Agent Systems (PAAMS’12), Salamanca, vol. 155, pp. 147–152. University of Salamanca, Salamanca, 28–30 Mar 2012Google Scholar
  32. Mateas, M., Stern, A.: Natural language understanding in façade: surfacetext processing. In: Proceedings of the Conference on Technologies for Interactive Digital Storytelling and Entertainment (TIDSE), Darmstadt (2004)Google Scholar
  33. Michael, D.R., Chen, S.L.: Serious Games: Games that Educate, Train, and Inform. Muska & Lipman/Premier-Trade (2005)Google Scholar
  34. Mori, D., Berta, R., De Gloria, A., Fiore, V., Magnani, L.: An easy to author dialogue management system for serious games. J. Comput. Cult. Herit. (JOCCH) 6 (2), 10 (2013)Google Scholar
  35. Morningstar, C., Farmer, R.F.: The lessons of Lucasfilm’s Habitat. In: Benedikt, M. (ed.) The First International Conference on Cyberspace, Austin, May 1990Google Scholar
  36. Murphy, J., Cremonini, F., Kane, G., Dunn, W.: Is simulation based medicine training the future of clinical medicine? Eur. Rev. Med. Pharmacol. Sci. 11 (1), 1 (2007)Google Scholar
  37. Orkin, J., Roy, D.: The restaurant game: learning social behavior and language from thousands of players online. J. Game Dev. 3 (1), 39–60 (2007)Google Scholar
  38. Parvati, D., Heinrichs, W.L., Patricia, Y.: CliniSpaceTM: a multiperson 3d online immersive training environment accessible through a browser. Med. Meets Virtual Real. 18: NextMed 163, 173 (2011)Google Scholar
  39. Pons Lelardeux, C., Panzoli, D., Lubrano, V., Minville, V., Jean-Pierre, J., Lagarrigue, P.: Communication system and team situation awareness in a multiplayer real-time learning environment: application to a virtual operating room. Vis. Comput. (2016, to appear)Google Scholar
  40. Porteous, J., Cavazza, M., Charles, F.: Applying planning to interactive storytelling: narrative control using state constraints. ACM Trans. Intell. Syst. Technol. 1 (2), 10:1–10:21 (2010)Google Scholar
  41. Prensky, M., Prensky, M.: Digital Game-Based Learning, vol. 1. Paragon House, St. Paul (2007)Google Scholar
  42. Rattner, G.N.: The king’s midwife. A history and mystery of madame du Coudray. University of California Press, Berkeley (1998)Google Scholar
  43. Reason, J.T., Reason, J.T.: Managing the risks of organizational accidents, vol. 6. Ashgate Aldershot, Brookfield (1997)Google Scholar
  44. Riedl, M.O., Li, B., Ai, H., Ram, A.: Robust and authorable multiplayer storytelling experiences. In: Proceedings of the 7th Annual Conference on Artificial Intelligence and Interactive Digital Entertainment, Palo Alto (2011)Google Scholar
  45. Rosen, K.R.: The history of medical simulation. J. Crit. Care 23 (2), 157–166 (2008)CrossRefGoogle Scholar
  46. Rus, V., D’Mello, S., Hu, X., Graesser, A.: Recent advances in conversational intelligent tutoring systems. AI Mag. 34 (3), 42–54 (2013)Google Scholar
  47. Sanchez, S., Balet, O., Luga, H., Duthen, Y.: Autonomous virtual actors. In: 2nd International Conference on Technologies for Interactive Digital Storytelling and Entertainment, Darmstadt, 24/06/2004-26/06/2004. LNCS, pp. 68–78. Springer (juin 2004)Google Scholar
  48. Sanselone, M., Sanchez, S., Sanza, C., Panzoli, D., Duthen, Y.: Control of non-playing characters in a medical learning game with Monte Carlo Tree Search (regular paper). In: IEEE Conference on Computational Intelligence and Games, Dortmund, 26/08/2014-29/08/2014, pp. 208–215. IEEE Computer Society (ao. 2014)
  49. Satava, R.M.: Medical virtual reality. the current status of the future. Stud. Health Technol. Inform. 29, 100 (1996)Google Scholar
  50. Sharma, M., Ontañón, S., Mehta, M., Ram, A.: Drama management and player modeling for interactive fiction games. Comput. Intell. 26 (2), 183–211 (2010)MathSciNetCrossRefGoogle Scholar
  51. Taekman, J.M., Segall, N., Hobbs, E., Wright, M.: 3diteams–healthcare team training in a virtual environment. Anesthesiology 107 (A2145), A2145 (2007)Google Scholar
  52. Thomas, D., Vlacic, L.: Collaborative decision making amongst human and artificial beings. In: Intelligent Decision Making: An AI-Based Approach, pp. 97–133. Springer, Berlin (2008)Google Scholar
  53. von Websky, M.W., Vitz, M., Raptis, D.A., Rosenthal, R., Clavien, P., Hahnloser, D.: Basic laparoscopic training using the simbionix lap mentor: setting the standards in the novice group. J. Surg. Educ. 69 (4), 459–467 (2012)CrossRefGoogle Scholar
  54. Warburton, S.: Second life in higher education: Assessing the potential for and the barriers to deploying virtual worlds in learning and teaching. Br. J. Educ. Technol. 40 (3), 414–426 (2009)CrossRefGoogle Scholar
  55. Waters, A., Bassendowski, S., Petrucka, P.: Serious games for students in healthcare: engaging a technically inclined generation. Can. J. Nurs. Inf. 3 (4), 16–27 (2008)Google Scholar
  56. Ziv, A., Small, S.D., Wolpe, P.W.: Patient safety and simulation-based medical education. Med. Teach. 22 (5), 489–495 (2000)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Research Institute on Informatics of Toulouse - UMR 5505University of ToulouseToulouseFrance
  2. 2.Serious Game Research LabChampollion UniversityToulouseFrance
  3. 3.University of Toulouse – Jean JaurèsToulouseFrance
  4. 4.University Hospital Centre of ToulouseToulouseFrance

Personalised recommendations