The Usefulness of Method-Resources for Evaluating a Collaborative Training Simulator

  • Ebba Thora HvannbergEmail author
  • Gyda Halldorsdottir
  • Jan Rudinsky
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9297)


Voice communication is vital for collaboration between first responders and commanders during crisis management. To decrease cost, training can take place in a virtual environment instead of in a real one. It is non-trivial to build and evaluate a virtual environment for training complex command. To understand the method-resources required for evaluating a training simulator for crisis response, this paper presents a case study of applying several resources. Method-resources were analysed for usability problems and Mechanics of Collaboration (MOC). The results show that the Group Observational Technique and the MOC analysis are appropriate for analysing factors of collaboration and communication. The think-aloud technique, observers, experts in the domain and advanced task scenario were important resources. In only a few cases sound and video were necessary to analyse issues.


Virtual reality Collaboration Evaluation Crisis management Verbal communication Method-resources Mechanics of collaboration 


  1. 1.
    Grudin, J.: Why CSCW applications fail: problems in the design and evaluation of organizational interfaces. In: Proceedings of the 1988 ACM Conference on Computer-Supported Cooperative Work, pp. 85–93. ACM, Portland (1988)Google Scholar
  2. 2.
    Divjak, M., Kore, D.: Visual and audio communication between visitors of virtual worlds. In: Advances in Neural Networks and Applications, pp. 41–46 (2001)Google Scholar
  3. 3.
    MacDonald, J.A., Balakrishnan, J., Orosz, M.D., Karplus, W.J.: Intelligibility of speech in a virtual 3-D environment. Hum. Factors: J. Hum. Factors Ergon. Soc. 44, 272–286 (2002)CrossRefGoogle Scholar
  4. 4.
    Antunes, P., Herskovic, V., Ochoa, S.F., Pino, J.A.: Structuring dimensions for collaborative systems evaluation. ACM Comput. Surv. 44, 1–28 (2012)CrossRefGoogle Scholar
  5. 5.
    de Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., Poulovassilis, A.: Developing an evaluation methodology for immersive learning experiences in a virtual world. In: Games and Virtual Worlds for Serious Applications, VS-GAMES 2009. Conference in, pp. 43–50 (Year)Google Scholar
  6. 6.
    Montoya, M.M., Massey, A.P., Lockwood, N.S.: 3D collaborative virtual environments: exploring the link between collaborative behaviors and team performance. Decis. Sci. 42, 451–476 (2011)CrossRefGoogle Scholar
  7. 7.
    Woolrych, A., Hornbæk, K., Frøkjær, E., Cockton, G.: Ingredients and meals rather than recipes: a proposal for research that does not treat usability evaluation methods as indivisible wholes. Int. J. Hum.-Comput. Interact. 27, 940–970 (2011)CrossRefGoogle Scholar
  8. 8.
    Law, E.L.-C., Hvannberg, E.T., Vermeeren, A.P., Cockton, G., Jokela, T.: Made for sharing: HCI stories of transfer, triumph and tragedy. In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, pp. 3235–3238. ACM (Year)Google Scholar
  9. 9.
    Rudinsky, J., Hvannberg, E.T., Helgason, A.A., Petursson, P.B.: Designing soundscapes of virtual environments for crisis management training. In: Proceedings of the Designing Interactive Systems Conference, pp. 689–692. ACM, Newcastle Upon Tyne (2012)Google Scholar
  10. 10.
    Gutwin, C., Greenberg, S.: The mechanics of collaboration: developing low cost usability evaluation methods for shared workspaces. In: Proeedings of the IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, (WET ICE 2000), pp. 98–103 (Year)Google Scholar
  11. 11.
    Pinelle, D., Gutwin, C.: Evaluating teamwork support in tabletop groupware applications using collaboration usability analysis. Pers. Ubiquit. Comput. 12, 237–254 (2008)CrossRefGoogle Scholar
  12. 12.
    Pinelle, D., Gutwin, C., Greenberg, S.: Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration. ACM Trans. Comput.-Hum. Interact. 10, 281–311 (2003)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • Ebba Thora Hvannberg
    • 1
    Email author
  • Gyda Halldorsdottir
    • 1
  • Jan Rudinsky
    • 1
  1. 1.University of IcelandReykjavíkIceland

Personalised recommendations