Calibration of Online Situation Awareness Assessment Systems Using Virtual Reality

  • Sebastien Mamessier
  • Daniel Dreyer
  • Matthias Oberhauser
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8529)


In an attempt to predict and prevent accident situations in complex socio-technical systems, one needs to be able to model and simulate concepts such as situation awareness (SA) and processes responsible for maintaining it. This is particularly true in the case of online support systems and adaptive displays which cannot rely on SA measurement techniques based on freeze probe techniques. This work investigates the state of the art in computational models of situation awareness and proposes a method to calibrate and evaluate such models using virtual reality human-in-the-loop experiments. This work introduces a new methodology to evaluate and calibrate online SA assessment systems taking advantage of the flexibility and reconfigurable power of virtual reality environments. This technology provides the experimenter with full control on the scenarios, cockpit types and interfaces. It also allows testing of off-nominal situations such as the loss of an instrument and more severe failures. Moreover, eye tracking capabilities provide an accurate way of registering monitoring events and feed SA assessment models with realistic data.


situation awareness mental models virtual reality 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aslandere, T.I.: Interaction Methods in a Generic Virtual Reality Flight Simulator. Master’s thesis, Technische Universitat Munchen (2013)Google Scholar
  2. 2.
    Dehais, F., Tessier, C., Christophe, L., Reuzeau, F.: The perseveration syndrome in the pilots activity: Guidelines and cognitive countermeasures. In: Palanque, P., Vanderdonckt, J., Winckler, M. (eds.) HESSD 2009. LNCS, vol. 5962, pp. 68–80. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  3. 3.
    Dekker, S., Hollnagel, E.: Human factors and folk models. Cognition, Technology & Work 6(2), 79–86 (2004)CrossRefGoogle Scholar
  4. 4.
    Dreyer, D., Bandow, D., Oberhauser, M.: Hud symbology evaluation in a virtual reality flight simulation. Paper submitted to HCI Aero 2014, Silicon Valley, USA (2014)Google Scholar
  5. 5.
    Ehrlich, J., Knerr, B., Lampton, D., McDonald, D.: Team situational awareness training in virtual environments: Potential capabilities and research issues. Tech. rep., U.S. Army Research Institute for the Behavioral and Social Sciences (1997)Google Scholar
  6. 6.
    Endsley, M.R.: Measurement of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society 37(1), 65–84 (1995)CrossRefGoogle Scholar
  7. 7.
    Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society 37(1), 32–64 (1995)CrossRefGoogle Scholar
  8. 8.
    Endsley, M.R.: Designing for situation awareness: An approach to user-centered design. Taylor & Francis, US (2003)Google Scholar
  9. 9.
    Endsley, M.: Situation awareness global assessment technique (sagat). In: Proceedings of the IEEE 1988 National Aerospace and Electronics Conference, NAECON 1988, pp. 789–795. IEEE (1988)Google Scholar
  10. 10.
    Feigh, K.M., Gelman, G., Mamessier, S., Pritchett, A.R.: Simulating first-principles models of situated human performance. IEEE Transactions on System Man and Cybernetics: Part A (submitted June 2012)Google Scholar
  11. 11.
    Hanson, M.L., Sullivan, O., Harper, K.A.: On-line situation assessment for unmanned air vehicles. In: FLAIRS Conference, pp. 44–48 (2001)Google Scholar
  12. 12.
    Hooey, B.L., Gore, B.F., Wickens, C.D., Scott-Nash, S., Socash, C., Salud, E., Foyle, D.C.: Modeling pilot situation awareness. In: Human Modelling in Assisted Transportation, pp. 207–213. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  13. 13.
    Horsch, C., Smets, N., Neerincx, M., Cuijpers, R.: Comparing performance and situation awareness in usar unit tasks in a virtual and real environment. In: Proceedings of the 10th International ISCRAM Conference, Baden-Baden, Germany, pp. 556–560 (2013)Google Scholar
  14. 14.
    Jones, D.G., Endsley, M.R.: Sources of situation awareness errors in aviation. Aviation, Space, and Environmental Medicine (1996)Google Scholar
  15. 15.
    Kalawsky, R.: New methodologies and techniques for evaluating user performance in advanced 3d virtual interfaces. IEEE Colloquium Digest 98/43 (1998)Google Scholar
  16. 16.
    Kalawsky, R.: Vruse - a computerised diagnostic tool for usability evaluation of virtual/synthetic environment systems. Applied Ergonomics 30, 11–25 (1999)CrossRefGoogle Scholar
  17. 17.
    Kalawsky, R., Bee, S., Nee, S.: Human factors evaluation techniques to aid understanding of virtual interfaces. BT Technology Journal 17(1), 128–141 (1999)CrossRefGoogle Scholar
  18. 18.
    Kleinman, D., Baron, S., Levison, W.: An optimal control model of human response part i: Theory and validation. Automatica 6(3), 357–369 (1970)CrossRefGoogle Scholar
  19. 19.
    Laptaned, U.: Situation awareness in virtual environments: A theoretical model and investigation with different interface designs. In: Proceedings of the 9th IASTED International Conference Computers and Advanced Technology in Education, Lima, Peru, pp. 277–283 (2006)Google Scholar
  20. 20.
    Lee, H.C., Seong, P.H.: A computational model for evaluating the effects of attention, memory, and mental models on situation assessment of nuclear power plant operators. Reliability Engineering & System Safety 94(11), 1796–1805 (2009)CrossRefGoogle Scholar
  21. 21.
    Letsu-Dake, E., Ntuen, C.A.: A case study of experimental evaluation of adaptive interfaces. International Journal of Industrial Ergonomics 40(1), 34–40 (2010)CrossRefGoogle Scholar
  22. 22.
    Liesecke, S.: Eye-Tracking in Virtual Reality. Master’s thesis, Universitt der Bundeswehr Munchen (2013)Google Scholar
  23. 23.
    Mamessier, S.: A computational approach to situation awareness and mental models in aviation. Master’s thesis, Georgia Institute of Technology (2013)Google Scholar
  24. 24.
    Mamessier, S., Feigh, K.: Simulating the impact of mental models on human automation interaction in aviation. In: Duffy, V.G. (ed.) DHM/HCII 2013, Part I. LNCS, vol. 8025, pp. 61–69. Springer, Heidelberg (2013)Google Scholar
  25. 25.
    Mamessier, S., Feigh, K.: A computational approach to situation awareness and mental models for continuous dynamics in aviation. IEEE Transactions on Human-Machine Systems, (2014) (Under review)Google Scholar
  26. 26.
    Matthews, M., Beal, S., Pleban, R.: Situation awareness in a virtual environment: Description of a subjective assessment scale. Tech. rep. U.S. Army Research Institute for the Behavioral and Social Sciences (2002)Google Scholar
  27. 27.
    McCarley, J.S., Wickens, C.D., Goh, J., Horrey, W.J.: A computational model of attention/situation awareness. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 46, pp. 1669–1673. SAGE Publications (2002)Google Scholar
  28. 28.
    Ratwani, R.M., McCurry, J.M., Trafton, J.G.: Single operator, multiple robots: An eye movement based theoretic model of operator situation awareness. In: 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 235–242. IEEE (2010)Google Scholar
  29. 29.
    Salmon, P., Stanton, N., Walker, G., Green, D.: Situation awareness measurement: A review of applicability for c4i environments. Applied Ergonomics 37(2), 225–238 (2006)CrossRefGoogle Scholar
  30. 30.
    Sarter, N.: Coping with complexity through adaptive interface design. In: Jacko, J.A. (ed.) Human-Computer Interaction, Part III, HCII 2007. LNCS, vol. 4552, pp. 493–498. Springer, Heidelberg (2007)Google Scholar
  31. 31.
    Sarter, N.B., Woods, D.D.: Situation awareness: A critical but ill-defined phenomenon. The International Journal of Aviation Psychology 1(1), 45–57 (1991)CrossRefGoogle Scholar
  32. 32.
    Stanton, N.A., Chambers, P., Piggott, J.: Situational awareness and safety. Safety Science 39(3), 189–204 (2001)CrossRefGoogle Scholar
  33. 33.
    Tien, G., Atkins, M.S., Zheng, B., Swindells, C.: Measuring situation awareness of surgeons in laparoscopic training. In: Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications, pp. 149–152. ACM (2010)Google Scholar
  34. 34.
    Vicente, K., Pawlak, W.: Cognitive work analysis for the duress ii system. Cognitive Engineering Laboratory, Department of Industrial Engineering, Toronto, Canada CEL, pp. 94–93. University of Toronto (1994)Google Scholar
  35. 35.
    Vicente, K.: Cognitive work analysis: Toward safe, productive, and healthy computer-based work. Lawrence Erlbaum (1999)Google Scholar
  36. 36.
    Zacharias, G.L., Miao, A.X., Illgen, C., Yara, J.M., Siouris, G.M.: Sample: Situation awareness model for pilot-in-the-loop evaluation. Final Report R 95192 (1996)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Sebastien Mamessier
    • 1
    • 2
  • Daniel Dreyer
    • 1
    • 2
  • Matthias Oberhauser
    • 1
    • 2
  1. 1.Airbus Group InnovationsOttobrunnGermany
  2. 2.Creative Concept and Design CenterOttobrunnGermany

Personalised recommendations