A Holistic Approach to Design and Evaluation of Mixed Reality Systems

Chapter
First Online:
Part of the Human-Computer Interaction Series book series (HCIS)

Abstract

This chapter addresses issues related to usability and user experience of mixed reality (MR) systems based on a naturalistic iterative design approach to the development of MR applications. Design and evaluation of MR applications are still mostly based on methods used for development of more traditional desktop graphical user interfaces. MR systems are in many aspects very different from desktop computer applications, so these traditional methods are not sufficient for MR applications. There is a need for new approaches to user-centred design and development of MR systems. One such approach is based on the concepts of cognitive systems engineering (CSE). In this chapter we show how this approach can be applied to the development of MR systems. Two case studies are described, where a holistic CSE approach to design, implementation and evaluation has been used. The results show that allowing real end users (field/domain experts) to interact in a close to naturalistic setting provides insights on how to design MR applications that are difficult to attain otherwise. We also show the importance of iterative design, again involving real end users.

keywords

Mixed reality systems Augmented reality User study User evaluation 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgements

This research is funded by the Swedish Defence Materiel Administration (FMV). The MR system was developed in close cooperation with XM Reality AB. We are deeply indebted to all the participants in our studies who volunteered their time and expertise to these projects.

References

  1. 1.
    Azuma, R.: A survey of augmented reality. Presence 6(4), 355–385 (1997)Google Scholar
  2. 2.
    Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. IEEE Computer Graphics and Applications 21(6), 34–47 (2001). http://computer.org/cga/cg2001/g6034abs.htm Google Scholar
  3. 3.
    Billinghurst, M., Kato, H.: Collaborative augmented reality. Communications of the ACM 45(7), 64–70 (2002)CrossRefGoogle Scholar
  4. 4.
    Clark, H.H.: Using Language. Cambridge University Press, Cambridge (1996)CrossRefGoogle Scholar
  5. 5.
    Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13(3) (1989)Google Scholar
  6. 6.
    Dekker, S., Hollnagel, E.: Human factors and folk models. Cognition Technology Work 6(2), 79–86 (2004). DOIhttp://dx.doi.org/10.1007/s10111-003-0136-9
  7. 7.
    Dünser, A., Grasset, R., Billinghurst, M.: A survey of evaluation techniques used in augmented reality studies. Tech. Rep. Technical Report TR2008-02, Human Interface Technology Laboratory New Zealand (2008)Google Scholar
  8. 8.
    Engeström, Y., Miettinen, R., Punamaki, R. (eds.): Perspectives on activity theory (Learning in doing social, cognitive and computational perspectives). Cambridge University Press, Cambridge (1999)Google Scholar
  9. 9.
    Fiala, M.: Artag rev2 fiducial marker system: Vision based tracking for AR. In: Workshop of Industrial Augmented Reality, Wienna Austria (2005)Google Scholar
  10. 10.
    Fuhrmann, A., Löffelmann, H., Schmalstieg, D.: Collaborative augmented reality: Exploring dynamical systems. In: R. Yagel, H. Hagen (eds.) IEEE Visualization 97, pp. 459–462. IEEE (1997)Google Scholar
  11. 11.
    Gabbard, J.L., Swan II, J.E.: Usability engineering for augmented reality: Employing user-based studies to inform design. IEEE Transactions on Visualization and Computer Graphics 14(3), 513–525 (2008)CrossRefGoogle Scholar
  12. 12.
    Glesne, C., Peshkin, A.: Becoming qualitative researchers: An introduction. Longman White Plains, New york (1992)Google Scholar
  13. 13.
    Granlund, R.: Web-based micro-world simulation for emergency management training. Future Generation Computer Systems 17, 561–572 (2001)CrossRefGoogle Scholar
  14. 14.
    Grasset, R., Lamb, P., Billinghurst, M.: Evaluation of mixed-space collaboration. In: ISMAR ’05: Proceedings of the 4th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 90–99. IEEE Computer Society, Washington, DC, USA (2005). DOI http://dx.doi.org/10.1109/ISMAR.2005.30
  15. 15.
  16. 16.
    Hollnagel, E., Woods, D.D.: Cognitive systems engineering: New wine in new bottles. International Journal of Man-Machine Studies 18(6), 583–600 (1983)CrossRefGoogle Scholar
  17. 17.
    Hollnagel, E., Woods, D.D.: Joint cognitive systems: Foundations of cognitive systems engineering. CRC Press, Boca Raton, FL (2005)CrossRefGoogle Scholar
  18. 18.
    Hutchins, E.: Cognition in the wild. MIT Press, London, England (1995). URL http://www.netlibrary.com/summary.asp?id=1687
  19. 19.
    Klein, G., Feltovich, P.J., Bradshaw, J.M., Woods, D.D.: Common ground and coordination in joint activity, vol. Organizational Simulation, Chap. 6, pp. 139–184. John Wiley & Sons, New York (2005). http://dx.doi.org/10.1002/0471739448.ch6
  20. 20.
    Liao, Z., Landry, R.: An empirical study on organizational acceptance of new information systems in a commercial bank. In: HICSS (2000). URL http://computer.org/proceedings/hicss/0493/04932/04932021abs.htm
  21. 21.
    Lincoln, Y.S., Guba, E.G.: Naturalistic Inquiry. Sage, Beverly Hills, CA (1985)Google Scholar
  22. 22.
    Livingston, M.A.: Evaluating human factors in augmented reality systems. IEEE Computer Graphics and Applications 25(6), 6–9 (2005). URL http://doi.ieeecomputersociety.org/10.1109/MCG.2005.130
  23. 23.
    Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems E77-D(12) (1994)Google Scholar
  24. 24.
    Neisser, U.: Cognition and reality. W.H. Freeman, San Francisc, CA (1976)Google Scholar
  25. 25.
    Nielsen, J.: Usability Engineering. Academic Press, Boston, MA (1993)MATHGoogle Scholar
  26. 26.
    Nilsson, S., Johansson, B.: User experience and acceptance of a mixed reality system in a naturalistic setting – A case study. In: ISMAR, pp. 247–248. IEEE (2006). URL http://dx.doi.org/10.1109/ISMAR.2006.297827
  27. 27.
    Nilsson, S., Johansson, B.: Fun and usable: augmented reality instructions in a hospital setting. In: B. Thomas (ed.) Proceedings of the 2007 Australasian Computer-Human Interaction Conference, OZCHI 2007, Adelaide, Australia, November 28–30, 2007, ACM International Conference Proceeding Series, vol. 251, pp. 123–130. ACM (2007). http://doi.acm.org/10.1145/1324892.1324915
  28. 28.
    Nilsson, S., Johansson, B.: Acceptance of augmented reality instructions in a real work setting. In: M. Czerwinski, A.M. Lund, D.S. Tan (eds.) Extended Abstracts Proceedings of the 2008 Conference on Human Factors in Computing Systems, CHI 2008, Florence, Italy, April 5-10, 2008, pp. 2025–2032. ACM (2008). http://doi.acm.org/10.1145/1358628.1358633
  29. 29.
    Patton, M.Q.: Qualitative evaluation and research methods, (2nd edn). Sage Publications, Thousand Oaks, CA (1990)Google Scholar
  30. 30.
    Qvarfordt, P., Jönsson, A., Dahlbäck, N.: The role of spoken feedback in experiencing multimodal interfaces as human-like. In: Proceedings of ICMI‧03, Vancouver, Canada (2003)Google Scholar
  31. 31.
    Reichardt, C., Cook, T.: Beyond qualitative versus quantitative methods. Qualitative and Quantitative Methods in Evaluation Research pp. 7–32 (1979)Google Scholar
  32. 32.
    Schmalstieg, D.: Rapid prototyping of augmented reality applications with the studierstube framework. In: Workshop of Industrial Augmented Reality, Wienna, Austria (2005)Google Scholar
  33. 33.
    Shneiderman, B.: Designing the user interface (3rd edn). Addison Wesley (1998)Google Scholar
  34. 34.
    Suchman, L.A.: Plans and situated actions. Cambridge University Press, Cambridge (1987)Google Scholar
  35. 35.
    Sutherland, I.E.: A head-mounted three-dimensional display. In: AFIPS Conference Proceedings, vol. 33, pp. 757–764 (1968)Google Scholar
  36. 36.
    Swan II, J.E., Gabbard, J.L.: Survey of user-based experimentation in augmented reality. In: 1st International Conference on Virtual Reality, Las Vegas, NV (2005)Google Scholar
  37. 37.
    Träskbäck, M.: Toward a usable mixed reality authoring tool. In: VL/HCC, pp. 160–162. IEEE Computer Society (2004). URL http://doi.ieeecomputersociety.org/10.1109/VLHCC.2004.60
  38. 38.
    Tushman, M.L., OReilly III, C.A.: Ambidextrous organizations: Managing evolutionary and revolutionary change. California Management Review 38(4), 8–30 (1996)Google Scholar
  39. 39.
    Vygotsky, L.S.: Mind in society. Harvard University Press, Cambridge (1978)Google Scholar
  40. 40.
    Woods, D.D., Roth, E.M.: Cognitive engineering: human problem solving with tools. Human Factors 30(4), 415–430 (1988)Google Scholar

Copyright information

© Springer-Verlag London 2010

Authors and Affiliations

  • Susanna Nilsson
    • 1
  • Björn Johansson
    • 2
  • Arne Jönsson
    • 2
  1. 1.Department of Computer and Information ScienceLinköping UniversityLinköpingSweden
  2. 2.Department of Computer and Information ScienceLinköping University, SaabSecurity, Santa Anna IT Research Institute ABLinköpingSweden

Personalised recommendations