The Visual Computer

, Volume 33, Issue 4, pp 489–515 | Cite as

Communication system and team situation awareness in a multiplayer real-time learning environment: application to a virtual operating room

  • Catherine Pons Lelardeux
  • David Panzoli
  • Vincent Lubrano
  • Vincent Minville
  • Pierre Lagarrigue
  • Jean-Pierre Jessel
Original Article


Digital multi-player learning games are believed to represent an important step forward in risk management training, especially related to human factors, where they are trusted to improve the performance of a team of learners in reducing serious adverse events, near-misses and crashes in complex socio-technical systems. Team situation awareness is one of the critical factors that can lead the team to consider the situation with an erroneous mental representation. Then, inadequate decisions are likely to be made regarding the actual situation. This paper describes an innovative communication system designed to be used in digital learning games. The system aims at enabling the learners to share information and build a common representation of the situation to help them take appropriate actions, anticipate failures, identify, reduce or correct errors. This innovative system is neither based on voice-chat nor branching dialogues, but on the idea that pieces of information can be manipulated as tangible objects in a virtual environment. To that end, it provides a handful of graphic interactions allowing users to collect, memorize, exchange, listen and broadcast information, ask and answer questions, debate and vote. The communication system was experimented on a healthcare training context with students and their teacher. The training scenario is set in a virtual operating room and features latent critical events (wrong-patient or wrong-side surgery). Teams have to manage such a critical situation, detect anomalies hidden in the environment and share them to make the most suitable decision. Analyzing the results demonstrated the efficacy of the communication system as per the ability for the players to actually exchange information, build a common representation of the situation and make collaborative decisions accordingly. The communication system was considered user-friendly by the users and successfully exposed lifelike behaviors such as debate, conflict or irritation. More importantly, every matter or implicit disagreement was raised while playing the game and led to an argued discussion, although eventually the right decision was not always taken by the team. So, improving the gameplay should help theplayers to manage a conflict and to make them agree on the most suitable decision.


Digital collaborative environment Team situation awareness Communication Information Decision making Learning game Virtual environment Socio-technical system Non technical skills 



The steering committee of 3DVOR is composed of Pr. Pierre Lagarrigue, M.D. Ph.D. Vincent Lubrano, M.D. Ph.D. Vincent Minville and Catherine Pons-Lelardeux. The following authors are also grateful to contributors to the project 3D operating room: Thomas Rodsphon, Cyrielle Guimbal, Michel Galaup and Jules de Guglielmi. The experiment described in Sect. 6.3 has been conducted under the supervision of Christiane Paban (teacher) and two students of the anesthetist nurse school of Toulouse: Hoang and Amelie. These works are part of a global national innovative IT program whose partners are KTM Advance company, Novamotion company, Serious Game Research Network and University Hospital of Toulouse (France). This R&D project is supported by French National Funding : Bpifrance Financement.


  1. 1.
    Reason, J.: A Life in Error. Ashgate Publishing Ltd, Farnham (2013)Google Scholar
  2. 2.
    Reason, J.: Human error: models and management. Br. Med. J. 320(7237), 768–770 (2000)CrossRefGoogle Scholar
  3. 3.
    Endsley, M.R., Robertson, M.M.: Situation awareness in aircraft maintenance teams. Int. J. Ind. Ergon. 26(2), 301–325 (2000)CrossRefGoogle Scholar
  4. 4.
    Fracker, M.L.: Attention gradients in situation awareness. In: Situational awareness in aerospace operations, pp. 6/1–6/10. Nato-Agard, Neuilly Sur Seine (1990)Google Scholar
  5. 5.
    Sarter, N.B., Woods, D.D.: Situation awareness: a critical but ill-defined phenomenon. Int. J. Aviat. Psychol. 1(1), 45–47 (1991)CrossRefGoogle Scholar
  6. 6.
    Hartel, C., Smith, K., Prince, C.: Defining aircrew coordination: searching mishaps for meaning. In: Fifth international symposium on aviation psychology. UQ Business School Publications, Ohio State University, Columbus (1989)Google Scholar
  7. 7.
    Halverson, A.L., Casey, J.T., Andersson, J., Anderson, K., Park, C., Rademaker, A., Moorman, D.: Communication failure in the operating room. Surgery 149(3), 305–310 (2011)CrossRefGoogle Scholar
  8. 8.
    Kohn, L.T., Corrigan, J.M., Donaldson, M.S. (eds.): To Err is Human: Building a Safer Health System. National Academies Press, Washington (2000)Google Scholar
  9. 9.
    Lingard, L., Espin, S., Whyte, S., Regehr, G., Baker, G., Reznick, R.: Communication failure in the operating room: observational classification of reccurent types and effects. Qual. Saf. Heathc. 13(5), 330–334 (2004)CrossRefGoogle Scholar
  10. 10.
    Authority, P.P.S: Pennsylvania patient safety authority 2007. Annual report, Pennsylvania (2007)Google Scholar
  11. 11.
    Joint Commission: Improving Americas hospitals: the joint commission annual report on quality and safety. Retreived February, vol. 25 (2008)Google Scholar
  12. 12.
    Vicente, K.: Cognitive Work Analysis. Lawrence Erlbaum Associates Inc., Mahwah (1999)Google Scholar
  13. 13.
    Effken, J.A.: Different lenses, improved outcomes: a new approach to the analysis and design of healthcare information systems. Int. J. Med. Inform. 65(1), 59–74 (2002)CrossRefGoogle Scholar
  14. 14.
    Plasters, C.L., Seagull, F.J., Xiao, Y.: Coordination challenges in operating-room management: an in-depth field study. In: Proceedings of AMIA annual symposium (2003)Google Scholar
  15. 15.
    Carayon, P.: Human factors of complex sociotechnical systems. Appl. Ergon. 37(4), 525–535 (2006)CrossRefGoogle Scholar
  16. 16.
    Plsek, P.E., Greenhalgh, T.: The challenge of complexity in health care. Br. Med. J. 323(7313), 625 (2001)CrossRefGoogle Scholar
  17. 17.
    Keyton, J., Beck, S.J., Asbury, M.B.: Macrocognition a communication perspective. Theor. Issues Ergon. Sci. 11, 272–286 (2010)CrossRefGoogle Scholar
  18. 18.
    Keyton, J., Beck, S.J.: Perspective: examining communication as macrocognition in STS. Hum. Factors J. Hum. Factors Ergon. Soc. 52(2), 335–339 (2010)CrossRefGoogle Scholar
  19. 19.
    Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors 37(1), 32–64 (1995)CrossRefGoogle Scholar
  20. 20.
    Kaber, D.B., Endsley, M.R.: Team situation awareness for process control safety and performance. Process Saf. Progr. 17(1), 43–48 (1998)CrossRefGoogle Scholar
  21. 21.
    Mathieu, J.E., Heffner, T.S., Goodwin, G.F., Salas, E., Cannon-Bowers, J.A.: The influence of shared mental models on team process and performance. J. Appl. Psychol. 85(2), 273 (2000)CrossRefGoogle Scholar
  22. 22.
    Devreux, G., Faure, G., Chevalier, A., Cegarra, J., Lubrano, V., Rodsphon, T.: The role of expertise and team situation awareness in a dynamic system: the case of the operating room. In: 28th international congress of applied psychology, Paris (2014)Google Scholar
  23. 23.
    Kolb, D.A.: Experiential Learning: Experience as the Source of Learning and Development. Pearson Education, Englewood Cliffs (1984)Google Scholar
  24. 24.
    Capin, T.K., Noser, H., Thalmann, D., Pandzic, I.S., Thalmann, N.M.: Virtual human representation and communication in VLNet. IEEE Comput. Graph. Appl. 2, 42–53 (1997)CrossRefGoogle Scholar
  25. 25.
    Egges, A., Papagiannakis, G., Magnenat-Thalmann, N.: Presence and interaction in mixed reality environments. Vis. Comput. 23(5), 317–333 (2007)CrossRefGoogle Scholar
  26. 26.
    Kopp, S., Wachsmuth, I.: Synthesizing multimodal utterances for conversational agents. Comput. Animat. Virtual worlds 15(1), 39–52 (2004)CrossRefGoogle Scholar
  27. 27.
    Ma, J., Cole, R.: Animating visible speech and facial expressions. Vis. Comput. 20(2–3), 86–105 (2004). doi: 10.1007/s00371-003-0234-y.
  28. 28.
    Grice, H.P., Cole, P., Morgan, J.L.: Syntax and semantics. Log. Conversat. 3, 41–58 (1975)Google Scholar
  29. 29.
    Duranti, A.: Linguistic Anthropology, 0th edn. Cambridge University Press, Cambridge (1997)CrossRefGoogle Scholar
  30. 30.
    Sacks, H., Schegloff, E.A., Jefferson, G.: A simplest systematics for the organization of turn-taking for conversation. Language 50(4), 696–735 (1974)CrossRefGoogle Scholar
  31. 31.
    Duncan Jr., S.: Toward a grammar for dyadic conversation. Semiotica 9(1), 29–46 (1973)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Goodwin, C.: Conversational Organization: Interaction Between Speakers and Hearers. Academic Press, New York (1981)Google Scholar
  33. 33.
    Herring, S.: Interactional coherence in CMC. J. Comput. Mediat. Commun. 4(4) (1999)Google Scholar
  34. 34.
    Fraser, N.: Assessment of Interactive Systems. Handbook on Standards and Resources for Spoken Language SystemsAssessment of Interactive Systems Handbook on Standards and Resources for Spoken Language Systems, pp. 564–615. Mouton de Gruyter, Berlin (1997)Google Scholar
  35. 35.
    Tear, M.F.M.: Spoken dialogue technology: enabling the conversationnal user interface. In: ACM Computing Surveys, vol. 34 (2002)Google Scholar
  36. 36.
    Fraser, N.M., Gilbert, G.N.: Simulating speech systems. Comput. Speech Lang. 5(1), 81–99 (1991)CrossRefGoogle Scholar
  37. 37.
    Navarathna, R., Lucey, P., Dean, D., Fookes, C., Sridharan, S.: Lip detection for audio-visual speech recognition in-car environment. In: Proceedings of the 10th international conference on information science, signal processing and their applications, pp. 598–601. IEEE (2010)Google Scholar
  38. 38.
    Lipovic, I.: Speech and Language Technologies. InTech Open Access Publisher (2011)Google Scholar
  39. 39.
    Mirzaei, M.R., Ghorshi, S., Mortazavi, M.: Audio-visual speech recognition techniques in augmented reality environments. Vis. Comput. 30(3), 245–257 (2013). doi: 10.1007/s00371-013-0841-1 CrossRefGoogle Scholar
  40. 40.
    Lelardeux, C.: Healthcare Games and the Metaphoric Approach. In: Serious games for healthcare: applications and implications: applications and implications, pp. 24–49. IGI Global (2012)Google Scholar
  41. 41.
    Michael, D.R., Chen, S.L.: Serious Games: Games that Educate, Train, and Inform. Muska & Lipman/Premier-Trade, Boston (2005)Google Scholar
  42. 42.
    Lelardeux, C.P., Galaup, M., Segonds, F., Lagarrigue, P.: Didactic study of a learning game to teach mechanical engineering. Proc. Eng. 132, 242–250 (2015)CrossRefGoogle Scholar
  43. 43.
    Zichermann, G., Cunningham, C.: Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps. O’Reilly Media, Inc., Sebastopol (2011)Google Scholar
  44. 44.
    Potier, V., Lelardeux, C.P., Lalanne, M., Lagarrigue, P.: Making complexity fun—machining procedures in mechanical engineering. Gamification Eng. Educ. 1, 1–10 (2016)Google Scholar
  45. 45.
    Daesign: Renault Academy (2009)Google Scholar
  46. 46.
    Disney Stars: The virtual sell (2012)Google Scholar
  47. 47.
    Rilling, S., Wechselberger, U.: A framework to meet didactical requirements for serious game design. Vis. Comput. 27(4), 287–297 (2011). doi: 10.1007/s00371-011-0550-6 CrossRefGoogle Scholar
  48. 48.
    Csikszentmihalyi, M., Abuhamdeh, S., Nakamura, J.: Flow. In: Elliot, A.J., Dweck, C.S. (eds.) Handbook of competence and motivation, pp. 598–609. Guilford Press, New York (2005)Google Scholar
  49. 49.
    Sanselone, M., Sanchez, S., Sanza, C., Panzoli, D., Duthen, Y.: Control of non-playing characters in a medical learning game with Monte Carlo Tree Search (regular paper). In: IEEE conference on computational intelligence and games, pp. 208–215. IEEE Computer Society, Dortmund (2014)Google Scholar
  50. 50.
    Thomas, D., Vlacic, L.: Collaborative decision making amongst human and artificial beings. In: Intelligent decision making: an AI-based approach, pp. 97–133. Springer (2008)Google Scholar
  51. 51.
    Lepper, M.R., Malone, T.W.: Intrinsic motivation and instructional effectiveness in computer-based education. Aptit. Learn. Instruct. 3, 255–286 (1987)Google Scholar
  52. 52.
    Malone, T.W.: What makes things fun to learn? Heuristics for designing instructional computer games. In: Proceedings of the 3rd ACM SIGSMALL symposium and the first SIGPC symposium on small systems, pp. 162–169. ACM, New York (1980)Google Scholar
  53. 53.
    Malone, T.W., Lepper, M.R.: Making learning fun: a taxonomy of intrinsic motivations for learning. Aptit. Learn. Instruct. 3(1987), 223–253 (1987)Google Scholar
  54. 54.
    Godden, D.R., Baddeley, A.D.: Context-dependent memory in two natural environments: on land and underwater. Br. J. Psychol. 66(3), 325–331 (1975)CrossRefGoogle Scholar
  55. 55.
    Cowan, B., Rojas, D., Kapralos, B., Moussa, F., Dubrowski, A.: Effects of sound on visual realism perception and task performance. Vis. Comput. 31(9), 1207–1216 (2014). doi: 10.1007/s00371-014-1006-6 CrossRefGoogle Scholar
  56. 56.
    Poling, N.D.: Collaboration, teamwork, and team cohesion in a starcraft 2 digital game-based course. Ph.D. thesis, University of Florida (2013)Google Scholar
  57. 57.
    Lagarrigue, P., Lubrano, V., Minville, V., Pons-Lelardeux, C.: The 3dvor project (2012).
  58. 58.
    Chin, T.J., You, Y., Coutrix, C., Lim, J.H., Chevallet, J.P., Nigay, L.: Mobile phone-based mixed reality. Snap2play Game 25(1), 25–37 (2009)Google Scholar
  59. 59.
    Liarokapis, F., Macan, L., Malone, G., Rebolledo-Mendez, G., Freitas, S.D.: Multimodal augmented reality tangible gaming. Vis. Comput. 25(12), 1109–1120 (2009). doi: 10.1007/s00371-009-0388-3.
  60. 60.
    Pandzic, I.S., Ostermann, J., Millen, D.: User evaluation: synthetic talking faces for interactive services. Vis. Comput. 15(7), 330–340 (1999)CrossRefGoogle Scholar
  61. 61.
    Parvati, D., Heinrichs, W.L.: CliniSpace: A multiperson 3d online immersive training environment accessible through a browser. Medicine Meets Virtual Reality 18: NextMed 163, 173 (2011)Google Scholar
  62. 62.
    Taekman, J.M., Segall, N., Hobbs, E., Wright, M.: 3diteamsHealthcare team training in a virtual environment. Anesthesiology 107(A2145), A2145 (2007)Google Scholar
  63. 63.
    Mateas, M., Stern, A.: Natural language understanding in faade: surface text processing. In: Proceedings of the conference on technologies for interactive digital storytelling and entertainment (TIDSE) (2004)Google Scholar
  64. 64.
    Cassell, J.: Embodied conversational agents. MIT press, Cambridge (2000)Google Scholar
  65. 65.
    Rus, V., D’mello, S., Hu, X., Graesser, A.: Recent advances in conversational intelligent tutoring systems. AI Mag. 34(3), 42–54 (2013)Google Scholar
  66. 66.
    Hennigan, B.: Making the case for NLP in dialogue systems for serious games. In: 8th international conference on natural language processing (JapTAL), 1st workshop on games and NLP (2012)Google Scholar
  67. 67.
    Morningstar, C., Farmer, R.F.: The Lessons of Lucasfilm’s Habitat. In: Benedikt, M. (ed.) The first international conference on cyberspace. University of Texas, Austin (1990)Google Scholar
  68. 68.
    Johnson, M.T., Clary, M.: A second life virtual clinic for medical student training. Second Life Education Community Conference (SLEDcc’08) (2008)Google Scholar
  69. 69.
    Salas, E., Prince, C., David, Baker, Shrestha, L.: Situation awareness in team performance. Implic. Meas. Train. 37(1), 123–136 (1995)Google Scholar
  70. 70.
    Leckie, G.J., Pettigrew, K.E., Sylvain, C.: Modeling the information seeking of professionals: a general model derived from research on engineers, health care professionals, and lawyers. The Library Quarterly, pp. 161–193 (1996)Google Scholar
  71. 71.
    Reddy, M., Jansen, B., Spence, P.: collaborative information behavior: exploring collaboration and coordination during information seeking and retrieval activities. In: Collaborative information behavior: user engagement and communication sharing. IGI Global, Hershey (2010)Google Scholar
  72. 72.
    Reddy, M., Bernard, J.J.: A model for understanding collaborative information behavior in context: a study of two healthcare teams (2006)Google Scholar
  73. 73.
    Panzoli, D., Sanselone, M., Sanchez, S., Sanza, C., Lelardeux, C., Lagarrigue, P., Duthen, Y.: Introducing a design methodology for multi-character collaboration in immersive learning games (regular paper). In: Proceedings of the sixth international conference on virtual worlds and games for serious applications (VS-Games14), p. (electronic medium). IEEExplore digital library, University of Malta (2014)Google Scholar
  74. 74.
    WHO: Surgical safety checklist (2009)Google Scholar
  75. 75.
    Authority, P.P.S.: Pennsylvania Patient Safety Authority 2012. Annual report 2012, Pennsylvania (2012)Google Scholar
  76. 76.
    Seiden, S.C., Barach, P.: Wrong-side/wrong-site, wrong-procedure, and wrong-patient adverse events: are they preventable? Arch. Surg. 141(9), 931–939 (2006)CrossRefGoogle Scholar
  77. 77.
    Busemann, A., Heidecke, C.D.: Safety checklists in the operating room. Dtsch. rztebl. Int. 109(42), 693–694 (2012). doi: 10.3238/arztebl.2012.0693.

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Catherine Pons Lelardeux
    • 1
  • David Panzoli
    • 1
  • Vincent Lubrano
    • 2
  • Vincent Minville
    • 3
  • Pierre Lagarrigue
    • 4
  • Jean-Pierre Jessel
    • 5
  1. 1.IRIT, University of Toulouse, INU ChampollionSerious Game Research NetworkToulouseFrance
  2. 2.ToNIC, University of Toulouse, UPS, University Hospital, InsermSerious Game Research NetworkToulouseFrance
  3. 3.MATN, University of Toulouse, UPS, University HospitalSerious Game Research NetworkToulouseFrance
  4. 4.ICA, University of Toulouse, INU ChampollionSerious Game Research NetworkToulouseFrance
  5. 5.IRIT, University of Toulouse, UPSSerious Game Research NetworkToulouseFrance

Personalised recommendations