Cluster Computing

, Volume 20, Issue 2, pp 1637–1659 | Cite as

Gathering big data for teamwork evaluation with microworlds

  • Claudio Miguel Sapateiro
  • Pedro AntunesEmail author
  • David Johnstone
  • José A. Pino


We identify some of the challenges related with conducting research into teamwork, addressing in particular the data gathering problem, where researchers face multiple tensions derived from different viewpoints regarding what data to gather and how to do it. To address this problem, we propose a microworld approach for conducting research into teamwork. We present the main requirements guiding the microworld development, and discuss a set of components that realise the requirements. Then, we discuss a study that used the developed microworld to evaluate a groupware tool, which was designed to support team activities related to infrastructure maintenance. The paper emphasises the range of data gathered with the microworld, and how it contributed to simultaneously evaluate team behaviour and tool design. The paper reflects on the major contributions brought by the microworld approach, emphasising in particular the capacity to gather diverse data, and to combine behaviour and design evaluations. This research contributes to consolidate the microworld approach in teamwork research. It also contributes to reduce the gap between behavioural-oriented and design-oriented research. The combination of the behaviour-oriented and design-oriented views is of particular importance to design science, since it is founded on iterative cycles of development and evaluation.


Teamwork data Teamwork evaluation Microworlds Design science 


  1. 1.
    Salas, E., Dickinson, T., Converse, S., Tannenbaum, S.: Toward an understanding of team performance and training. In: Swezey, R.W., Salas, E. (eds.) Teams: Their Training and Performance. Ablex Publishing, Westport (1992)Google Scholar
  2. 2.
    Wilson, K., Salas, E., Priest, H., Andrews, D.: Errors in the heat of battle: taking a closer look at shared cognition breakdowns through teamwork. Hum. FactorsC 49(2), 243–256 (2007)CrossRefGoogle Scholar
  3. 3.
    Ellis, C., Gibbs, S., Rein, G.: Groupware: some issues and experiences. Commun. ACM 34(1), 39–58 (1991)CrossRefGoogle Scholar
  4. 4.
    Jacovi, M., Soroka, V., Gilboa-Freedman, G., Ur, S., Shahar, E., Marmasse, N.: The chasms of CSCW: a citation graph analysis of the CSCW conference. In: Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work 2006, pp. 289–298. ACM (2006)Google Scholar
  5. 5.
    Grudin, J., Poltrock, S.: Taxonomy and theory in computer supported cooperative work. In: Kozlowski, S. (ed.) Handbook of Organizational Psychology, pp. 1323–1348. Oxford University Press, Oxford (2012)Google Scholar
  6. 6.
    Antunes, P., Herskovic, V., Ochoa, S., Pino, J.: Structuring dimensions for collaborative systems evaluation. ACM Comput. Surv. 44(2), 1–28 (2012)CrossRefGoogle Scholar
  7. 7.
    Antunes, P., Xiao, L., Pino, J.: Assessing the impact of educational differences in HCI design practice. Int. J. Technol. Des. Educ. 24(3), 317–335 (2014). doi: 10.1007/s10798-013-9254-8 CrossRefGoogle Scholar
  8. 8.
    Herskovic, V., Pino, J., Ochoa, S., Antunes, P.: Evaluation methods for groupware systems. In: Haake, J., Ochoa, S., Cechich, A. (eds.) Groupware: design, implementation, and use. 13th International Workshop, CRIWG 2007, Bariloche, Argentina, September 2007 Proceedings, vol. 4715. LNCS, pp. 328–336. Springer, Heidelberg (2007)Google Scholar
  9. 9.
    Hamadache, K., Lancieri, L.: Strategies and taxonomy, tailoring your CSCW evaluation. In: International Conference on Collaboration and Technology, pp. 206–221. Springer, Heidelberg (2009)Google Scholar
  10. 10.
    Grudin, J.: Why CSCW applications fail: problems in the design and evaluation of organizational interfaces. In: Proceedings of the 1988 ACM Conference on Computer-Supported Cooperative Work, Portland, pp. 85–93. ACM (1988)Google Scholar
  11. 11.
    Hughes, J., King, V., Rodden, T., Andersen, H.: Moving out from the control room: ethnography in system design. In: Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, Chapel Hill, pp. 429–439. ACM Press (1994)Google Scholar
  12. 12.
    Hevner, A., March, S., Park, J., Ram, S.: Design science in information systems research. Manag. Inf. Syst. Q. 28(1), 75–105 (2004)CrossRefGoogle Scholar
  13. 13.
    Gregor, S.: The nature of theory in information systems. MIS Q. 30(3), 611–642 (2006)CrossRefGoogle Scholar
  14. 14.
    Gregor, S., Jones, D.: The anatomy of a design theory. J. Assoc. Inf. Syst. 8(5), 312–335 (2007)Google Scholar
  15. 15.
    Fjermestad, J., Hiltz, S.: An assessment of group support systems experimental research: methodology and results. J. Manag. Inf. Syst. 15(3), 7–149 (1999)CrossRefGoogle Scholar
  16. 16.
    Klein, G.: Naturalistic decision making. Hum. Factors 50(3), 456–460 (2008)CrossRefGoogle Scholar
  17. 17.
    Hollan, J., Hutchins, E., Kirsh, D.: Distributed cognition: toward a new foundation for human–computer interaction research. ACM Trans. Comput. Hum. Interact. 7(2), 174–196 (2000)CrossRefGoogle Scholar
  18. 18.
    Suchman, L.: Plans and Situated Actions: The Problem of Human–Machine Communication. Cambridge University Press, Cambridge (1987)Google Scholar
  19. 19.
    Endsley, M., Garland, D.: Situation Awareness Analysis and Measurement. CRC Press, Boca Raton (2000)Google Scholar
  20. 20.
    Endsley, M., Jones, W.: Situation awareness. In: Lee, J.D., Kirlik, A. (eds.) The Oxford Handbook of Cognitive Engineering, pp. 88–108. Oxford University Press, New York (2013)Google Scholar
  21. 21.
    Antunes, P., Herskovic, V., Ochoa, S., Pino, J.: Reviewing the quality of awareness support in collaborative applications. J. Syst. Softw. 89, 146–169 (2014)CrossRefGoogle Scholar
  22. 22.
    Wickens, C.: Situation awareness: Review of Mica Endsley’s 1995 articles on situation awareness theory and measurement. Hum. Factors 50(3), 397–403 (2008)CrossRefGoogle Scholar
  23. 23.
    Salmon, P., Stanton, N., Walker, G., Jenkins, D., Ladva, D., Rafferty, L., Young, M.: Measuring situation awareness in complex systems: comparison of measures study. Int. J. Ind. Ergon. 39(3), 490–500 (2009)CrossRefGoogle Scholar
  24. 24.
    Collins, A., Joseph, D., Bielaczyc, K.: Design research: theoretical and methodological issues. J. Learn. Sci. 13(1), 15–42 (2004)CrossRefGoogle Scholar
  25. 25.
    Lew, R., Boring, R., Ulrich, T.: A prototyping environment for research on human-machine interfaces in process control use of Microsoft WPF for microworld and distributed control system development. In: 7th International Symposium on Resilient Control Systems. IEEE (2014)Google Scholar
  26. 26.
    Jercic, P., Astor, P., Adam, M., Hilborn, O., Schaaff, K., Lindley, C., Sennersten, C., Eriksson, J.: A serious game using physiological interfaces for emotion regulation training in the context of financial decision-making. In: European Conference on Information Systems, p. 207 (2012)Google Scholar
  27. 27.
    Naweed, A., Hockey, G., Clarke, S.: Designing simulator tools for rail research: the case study of a train driving microworld. Appl. Ergon. 44(3), 445–454 (2013)CrossRefGoogle Scholar
  28. 28.
    Kluge, A.: Performance assessments with microworlds and their difficulty. Appl. Psychol. Meas. 32(2), 156–180 (2008)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Mosier, C.: A critical examination of the concepts of face validity. Educ. Psychol. Meas. 7, 191–205 (1947)CrossRefGoogle Scholar
  30. 30.
    Wastell, D., Peckover, S., White, S., Broadhurst, K., Hall, C., Pithouse, A.: Social work in the laboratory: using microworlds for practice research. Br. J. Soc. Work 41, 744–760 (2011)CrossRefGoogle Scholar
  31. 31.
    Rolo, G., Diaz-Cabrera, D.: Decision-making processes evaluation using two methodologies: field and simulation techniques. Theor. Issues Ergon. Sci. 6(1), 35–48 (2005)CrossRefGoogle Scholar
  32. 32.
    Gray, W.: Simulated task environments: the role of high-fidelity simulations, scaled worlds, synthetic environments, and laboratory tasks in basic and applied cognitive research. Cogn. Sci. Q. 2(2), 205–207 (2002)MathSciNetGoogle Scholar
  33. 33.
    Rigas, G., Carling, E., Brehmer, B.: Reliability and validity of performance measures in microworlds. Intelligence 30(5), 463–480 (2002)CrossRefGoogle Scholar
  34. 34.
    DiFonzo, N., Hantula, D., Bordia, P.: Microworlds for experimental research: Having your (control and collection) cake, and realism too. Behav. Res. Methods Instrum. Comput. 30(2), 278–286 (1998)CrossRefGoogle Scholar
  35. 35.
    Chen, C., Zhang, C.: Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf. Sci. 275, 314–347 (2014)CrossRefGoogle Scholar
  36. 36.
    Popper, K.: Science: conjectures and refutations. In: Introductory Readings in the Philosophy of Science, pp. 38–47. Prometheus Books, Amherst (1998)Google Scholar
  37. 37.
    Venkatesh, V., Brown, S., Bala, H.: Bridging the qualitative–quantitative divide: guidelines for conducting mixed methods research in information systems. MIS Q. 37(1), 21–54 (2013)CrossRefGoogle Scholar
  38. 38.
    Horn, D., Finholt, T., Birnholtz, J., Motwani, D., Jayaraman, S.: Six degrees of Jonathan Grudin: a social network analysis of the evolution and impact of CSCW research. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 582–591. ACM (2004)Google Scholar
  39. 39.
    Pinelle, D., Gutwin, C.: A review of groupware evaluations. In: Proceedings of 9th IEEE WETICE Infrastructure for Collaborative Enterprises (2000)Google Scholar
  40. 40.
    Grant, A., Wall, T.: The neglected science and art of quasi-experimentation: why-to, when-to, and how-to advice for organizational researchers. Org. Res. Methods 12(4), 653–686 (2008)CrossRefGoogle Scholar
  41. 41.
    Winter, R.: Design science research in Europe. Eur. J. Inf. Syst. 17(5), 470–475 (2008)CrossRefGoogle Scholar
  42. 42.
    Baskerville, R., Wood-Harper, A.: A critical perspective on action research as a method for information systems research. In: Willcocks, L.P., Sauer, C., Lacity, M.C. (eds.) Enacting Research Methods in Information Systems, pp. 169–170. Springer, Berlin (2016)Google Scholar
  43. 43.
    Jenkins, A.: Research methodologies and MIS research. In: Mumford, E., et al. (eds.) Research Methods in Information Systems, pp. 103–117. North-Holland Publishing Co. Amsterdam, The Netherlands (1985)Google Scholar
  44. 44.
    Sinkovics, R., Ghauri, P.: Enhancing the trustworthiness of qualitative research in international business. Manag Int Rev 48(6), 689–714 (2008)CrossRefGoogle Scholar
  45. 45.
    Flach, J.: Mind the gap: a skeptical view of macrocognition. In: Schraagen, J., Militello, L., Ormerod, T., Lipshitz, R. (eds.) Naturalistic Decision Making and Macrocognition. Ashgate, Hampshire (2008)Google Scholar
  46. 46.
    Wallace, B., Ross, A. (eds.): Beyond Human Error—Taxonomies and Safety Science. CRC Taylor and Francis Group, New York (2006)Google Scholar
  47. 47.
    Dekker, S.: The Field Guide to Understanding Human Error. Ashgate, Hampshire (2006)Google Scholar
  48. 48.
    Klein, G.: A recognition-primed decision (RPD) model of rapid decision making. In: Klein, G., Orasanu, J., Calderwood, R., Zsambok, C. (eds.) Decision Making in Action: Models and Methods. Ablex, Westport, CT (1993)Google Scholar
  49. 49.
    Salas, E., Fiore, S., Warner, N., Letsky, M.: Emerging multi-disciplinary theoretical perspectives in team cognition: an overview. Theor. Issues Ergon. Sci. 11(4), 245–249 (2010)CrossRefGoogle Scholar
  50. 50.
    Lipshitz, R., Klein, G., Orasanu, J., Salas, E.: Taking stock of naturalistic decision making. J. Behav. Decis. Mak. 14, 331–352 (2001)CrossRefGoogle Scholar
  51. 51.
    Cleven, A., Gubler, P., Hüner, K.: Design alternatives for the evaluation of design science research artifacts. In: Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology, Philadelphia, pp. 1–8. ACM (2009)Google Scholar
  52. 52.
    Patrick, J., James, N.: Process tracing of complex cognitive work tasks. J. Occup. Org. Psychol. 77(2), 259–280 (2004)CrossRefGoogle Scholar
  53. 53.
    Sá, M., Carriço, L., Antunes, P.: Ubiquitous psychotherapy. IEEE. Pervasive Comput. 6(1), 20–27 (2007)CrossRefGoogle Scholar
  54. 54.
    Piirainen, K., Gonzalez, R., Kolfschoten, G.: Quo Vadis, design science?—a survey of literature. In: Global Perspectives on Design Science Research, vol. 6105. Lecture Notes in Computer Science, pp. 93–108. Springer (2010)Google Scholar
  55. 55.
    Briggs, R.: On theory-driven design and deployment of collaboration systems. Int. J. Hum Comput Stud. 64(7), 573–582 (2006)CrossRefGoogle Scholar
  56. 56.
    Card, S., Moran, T., Newell, A.: The Psychology of Human–Computer Interaction. Lawrance Elrbaum, Hillsdale (1983)Google Scholar
  57. 57.
    Reason, J.: Human Error. Cambridge University Press, Cambridge (1990)CrossRefGoogle Scholar
  58. 58.
    Cacciabue, P.: Guide to Applying Human Factors Methods. Springer, London (2004)CrossRefGoogle Scholar
  59. 59.
    Cooke, N., Gorman, J., Myers, C., Duran, J.: Interactive team cognition. Cogn. Sci. 37(2), 255–285 (2013)CrossRefGoogle Scholar
  60. 60.
    Gibson, J.: The Senses Considered as Perceptual Systems. Houhton Mifflin, Boston (1966)Google Scholar
  61. 61.
    Antunes, P., Zurita, G., Baloian, N.: Key indicators for assessing the design of geocollaborative applications. Int. J. Inf. Technol. Decis. Mak. 13(2), 361–385 (2014)CrossRefGoogle Scholar
  62. 62.
    Turvey, M., Shawn, R.: Toward an ecological physics and physical psychology. In: Solso, R., Massaro, S. (eds.) The Science of the Mind: 2001 and Beyond, pp. 144–169. Oxford University Press, New York (1995)Google Scholar
  63. 63.
    Salas, E., Sims, D., Burke, C.: Is there a “Big Five” in teamwork? Small Group Res. 36(5), 555–599 (2005)CrossRefGoogle Scholar
  64. 64.
    Davis, F.: A technology acceptance model for empirically testing new end-user information systems: theory and results. Ph.D. Thesis, Massachusetts Institute of Technology (1986)Google Scholar
  65. 65.
    Read, A., Hullsiek, B., Briggs, R.: The seven layer model of collaboration: an exploratory study of process identification and improvement. In: 45th Hawaii International Conference on System Science, pp. 412–420. IEEE (2012)Google Scholar
  66. 66.
    Barnard, P., May, J., Duke, D., Duce, D.: Systems, interactions, and macrotheory. ACM Trans. Comput. Hum. Interact. 7(2), 222–262 (2000)CrossRefGoogle Scholar
  67. 67.
    Klein, G., Ross, K., Moon, B., Klein, D., Hoffman, R., Hollnagel, E.: Macrocognition. IEEE Intell. Syst. 18(3), 81–85 (2003)CrossRefGoogle Scholar
  68. 68.
    Fiore, S., Smith-Jentsch, K., Salas, E., Warner, N., Letsky, M.: Towards an understanding of macrocognition in teams: developing and defining complex collaborative processes and products. Theor. Issues Ergon. Sci. 11(4), 250–271 (2010)CrossRefGoogle Scholar
  69. 69.
    Papert, S.: Microworlds: transforming education. Artif. Intell. Educ. 1, 79–94 (1987)Google Scholar
  70. 70.
    Brehmer, B., Dorner, D.: Experiments with computer-simulated microworlds: escaping both the narrow straits of the laboratory and the deep blue sea of the field study. Comput. Hum. Behav. 9, 171–184 (1993)CrossRefGoogle Scholar
  71. 71.
    Johansson, B., Trnka, J., Granlund, R.: The Effect of geographical information systems on a collaborative command and control task. In: Proceedings of the 4th International Conference on Information Systems for Crisis Response and Management (ISCRAM). Delft (2007)Google Scholar
  72. 72.
    Schraagen, J., Van den Ven, J.: Improving decision making in crisis response through critical thinking support. J. Cogn. Eng. Decis. Mak. 2, 311–327 (2008)CrossRefGoogle Scholar
  73. 73.
    Arthur, W., Day, E., Villado, A., Boatman, P., Kowollik, V., Bennet, W., Bhupatkar, A.: The effect of distributed practice on immediate posttraining, and long-term performance on a complex command-and-control simulation task. Hum. Perform. 23(5), 428–445 (2010)CrossRefGoogle Scholar
  74. 74.
    Lew, R., Boring, R., Ulrich, T.: A prototyping environment for research on human-machine interfaces in process control use of Microsoft WPF for microworld and distributed control system development. In: International Symposium on Resilient Control Systems, pp. 1–6. IEEE (2014)Google Scholar
  75. 75.
    O’Brien, K., O’Hare, D.: Situational awareness ability and cognitive skills training in a complex real-world task. Ergonomics 50(7), 1064–1091 (2007)CrossRefGoogle Scholar
  76. 76.
    Berggren, P., Johansson, B., Svensson, E., Baroutsi, N., Dahlbäck, N.: Statistical modelling of team training in a microworld study. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 58(1), 894–898 (2014)CrossRefGoogle Scholar
  77. 77.
    de Heer, J.: How Do architects think? A game based microworld for elucidating dynamic decision-making. In: Complex Systems Design & Management, pp. 133–142. Springer (2016)Google Scholar
  78. 78.
    Chapman, T., Nettelbecka, T., Welsha, M., Millsab, V.: Investigating the construct validity associated with microworld research: a comparison of performance under different management structures across expert and non-expert naturalistic decision-making groups. Aust. J. Psychol. 58(1), 40–47 (2006)CrossRefGoogle Scholar
  79. 79.
    Mavrikis, M., Dragon, T., Abdu, R., Harrer, A., De Groot, R., McLaren, B.: Learning to learn together through planning, discussion and reflection on microworld-based challenges. In: European Conference on Technology Enhanced Learning, pp. 483–488. Springer, Heidelberg (2012)Google Scholar
  80. 80.
    Frezzo, D., DiCerbo, K., Behrens, J., Chen, M.: An extensible micro-world for learning in the data networking professions. Inf. Sci. 264, 91–103 (2014)CrossRefGoogle Scholar
  81. 81.
    Djaouti, D., Alvarez, J., Jessel, J.: Classifying serious games: the G/P/S model. In: Felicia, P. (ed.) Handbook of Research on Improving Learning and Motivation Through Educational Games: Multidisciplinary Approaches, pp. 118–136. IGI Global, Hershey (2011)CrossRefGoogle Scholar
  82. 82.
    Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z.: Predicting protein structures with a multiplayer online game. Nature 466(7307), 756–770 (2010)CrossRefGoogle Scholar
  83. 83.
    Sonnleitner, P., Brunner, M., Greiff, S., Funke, J., Keller, U., Martin, R., Hazotte, C., Mayer, H., Latour, T.: The Genetics Lab. Acceptance and psychometric characteristics of a computer-based microworld to assess complex problem solving. Psychol. Test Assess. Model. 54, 54–72 (2012)Google Scholar
  84. 84.
    Gonzalez, C., Vanyukov, P., Martin, M.: The use of microworlds to study dynamic decision making. Comput. Hum. Behav. 21(2), 273–286 (2005)CrossRefGoogle Scholar
  85. 85.
    Brehmer, B.: Micro-worlds and the circular relation between people and their environment. Theor. Issues Ergon. Sci. 6(1), 73–93 (2005)CrossRefGoogle Scholar
  86. 86.
    Endsley, M., Bolté, B., Jones, D.: Designing for Situation Awareness. Taylor & Francis, London (2003)CrossRefGoogle Scholar
  87. 87.
    Wagner, F., Schmuki, R., Wagner, T., Wolstenholme, P.: Modeling Ssoftware with Finite State Machines: A Practical Approach. CRC Press, New York (2006)CrossRefzbMATHGoogle Scholar
  88. 88.
    Lee, M.: A testing framework based on finite automata for object-oriented software specification. J. Inf. Technol. Theory Soc. 1, 59–88 (2004)Google Scholar
  89. 89.
    Buchner, A., Funke, J.: Finite state automata: dynamic task environments in problem solving research. Q. J. Exp. Psychol. 46A(1), 83–118 (1993)CrossRefGoogle Scholar
  90. 90.
    Taylor, M., Endsley, M., Henderson, S.: Situational awareness workshop report. In: Hayward, B., Lowe, A. (eds.) Applied Aviation Psychology: Achivement, Change and Challenge, pp. 447–454. Ashgate Publishing, Aldershot (1996)Google Scholar
  91. 91.
    Citera, M., McNeese, M., Brown, C., Selvaraj, J.: Fitting information systems to collaborating design teams. J. Am. Soc. Inf. Sci. Technol. 46(7), 551 (1995)CrossRefGoogle Scholar
  92. 92.
    Klein, G., Zsambock, C., Thordsen, M.: Team decision training: five myths and a model. Mil. Rev. 73(4), 36–42 (1993)Google Scholar
  93. 93.
    Berndtsson, J., Normark, M.: The coordinative functions of flight strips : air traffic control revisited. In: Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work, pp. 101–110. ACM, New York (1999)Google Scholar
  94. 94.
    Heath, C., Luff, P.: Collaboration and control crisis management and multimedia technology in London Underground Line Control Rooms. Comput. Support. Coop. Work 1(1–2), 69–94 (1992)CrossRefGoogle Scholar
  95. 95.
    Pettersson, M., Randall, D., Helgeson, B.: Ambiguities, awareness and economy: a study of emergency service work. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work, New Orleans, pp. 286–295. ACM (2002)Google Scholar
  96. 96.
    Monares, A., Ochoa, S., Pino, J., Herskovic, V., Rodriguez-Covili, J., Neyem, A.: Mobile computing in urban emergency situations: improving the support to firefighters in the field. Expert Syst. Appl. 38(2), 1255–1267 (2011)CrossRefGoogle Scholar
  97. 97.
    Whittaker, S., Amento, B.: Seeing what you are hearing: co-ordinating responses to trouble reports in network troubleshooting. In: European Conference on Computer Supported Cooperative Work, Helsinki, pp. 219–238. Springer (2003)Google Scholar
  98. 98.
    Fallman, D.: Enabling physical collaboration in industrial settings by designing for embodied interaction. In: Proceedings of the Latin American Conference on Human–Computer Interaction, Rio de Janeiro, pp. 41–51. ACM (2003)Google Scholar
  99. 99.
    Nulden, U.: Investigating police patrol practice for design of IT. In: CHI ’03 Extended Abstracts on Human Factors in Computing Systems, Ft. Lauderdale, pp. 820–821. ACM (2003)Google Scholar
  100. 100.
    Landgren, J.: Supporting fire crew sensemaking enroute to incidents. Int. J. Emerg. Manag. 2(3), 176–188 (2005)CrossRefGoogle Scholar
  101. 101.
    Salmon, P., Stanton, N., Walker, G., Green, D.: Situation awareness measurement: a review of applicability for C4i environments. Appl. Ergon 37(2), 225–238 (2006)CrossRefGoogle Scholar
  102. 102.
    Bai, X., Tsai, W., Paul, R., Feng, K., Yu, L.: Scenario-based modeling and its applications. In: Proceedings of the Seventh International Workshop on Object-Oriented Real-Time Dependable Systems, pp. 253–260. IEEE Computer Society (2002)Google Scholar
  103. 103.
    Barrett, R., Kandogan, E., Maglio, P., Haber, E., Takayama, L., Prabaker, M.: Field studies of computer system administrators: analysis of system management tools and practices. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 388–395. ACM (2004)Google Scholar
  104. 104.
    Sapateiro, C.: Evaluating mobile collaborative applications support of teamwork in critical incidents response management. Ph.D. Thesis, University of Lisbon (2013)Google Scholar
  105. 105.
    Sapateiro, C., Antunes, P.: An emergency response model toward situational awareness improvement. In: International Conference on Information Systems for Crisis Response and Management. Göteborg (2009)Google Scholar
  106. 106.
    McManus, S., Seville, E., Brunsdon, D., Vargo, J.: Resilience management: a framework for assessing and improving the resilience of organisations. Research Report 2007/01. Resilient Organizations (2007)Google Scholar
  107. 107.
    Hollnagel, E., Woods, D.: Joint Cognitive Systems: Foundations of Cognitive Systems Engineering. CRC Press, Boca Raton (2005)CrossRefGoogle Scholar
  108. 108.
    Endsley, M.: Toward a theory of situation awareness in dynamic systems. Hum. Factors 31(7), 32–64 (1995)CrossRefGoogle Scholar
  109. 109.
    Smith, K., Hancock, P.: Situation awareness is adaptive, externelly directed consciousness. Hum. Factors 37, 137–148 (1995)CrossRefGoogle Scholar
  110. 110.
    Bolstad, C., Cuevas, H., Gonzalez, C., Schneider, M.: Modeling shared situation awareness. In: Proceedings of the 14th Conference on Behavior Representation in Modeling and Simulation. Los Angles (2005)Google Scholar
  111. 111.
    Stanton, N., Stewart, R., Harris, D., Houghton, R., Baber, C., McMaster, R., Salmon, P., Hoyle, G., Walker, G., Young, M., Linsell, M., Dymott, R., Green, D.: Distributed situation awareness in dynamic systems: theoretical develoment and application of an ergonomics methodology. Ergonomics 49(12–13), 1288–1311 (2006)CrossRefGoogle Scholar
  112. 112.
    Tamhane, A., Dunlop, D.: Statistics and Data Analysis: From Elementary to Intermediate. Prentice Hall, Upper Saddle River (2000)Google Scholar
  113. 113.
    Hevner, A.: A three cycle view of design science research. Scand. J. Inf. Syst. 19(2), 87–92 (2007)Google Scholar
  114. 114.
    Kurbalija, V., Ivanović, M., Bernstorff, C., Nachtwei, J., Burkhard, H.: Matching observed with empirical reality-what you see is what you get? Fundam.Inf. 129(1–2), 133–147 (2014)MathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Claudio Miguel Sapateiro
    • 1
  • Pedro Antunes
    • 2
    Email author
  • David Johnstone
    • 2
  • José A. Pino
    • 3
  1. 1.Department of Systems and Informatics Polytechnic of SetubalSetubalPortugal
  2. 2.School of Information Management Victoria University of WellingtonWellingtonNew Zealand
  3. 3.Department of Computer ScienceUniversity of ChileSantiagoChile

Personalised recommendations