Evaluating Augmented Reality Systems

  • Andreas DünserEmail author
  • Mark Billinghurst


This Chapter discusses issues with Augmented Reality (AR) systems evaluations. First the role of evaluation and various challenges for evaluating novel AR interfaces and interaction techniques are reviewed. A special focus is then provided on user-based and non-user-based evaluation techniques currently used for evaluating AR systems. The practical application of these methods is demonstrated through different examples from the scientific literature. Various points raised in this chapter provide arguments for the development of more specific frameworks and models for AR-based interfaces and interaction techniques. These will provide researchers with a better basis for developing and applying more suitable evaluation methods that address the specific requirements of evaluating AR-based systems.


  1. 1.
    I. Sutherland, “A Head-Mounted Three-Dimensional Display,” presented at the Fall Joint Computer Conf., Am. Federation of Information Processing Soc. (AFIPS), Washington, D.C., USA, 1968.Google Scholar
  2. 2.
    A. Dünser, R. Grasset, H. Seichter, and M. Billinghurst, “Applying HCI principles to AR systems design,” Charlotte, NC, USA, 2007.Google Scholar
  3. 3.
    J. E. Swan and J. L. Gabbard, “Survey of User-Based Experimentation in Augmented Reality,” presented at the 1st International Conference on Virtual Reality, HCI International 2005, Las Vegas, USA, 2005.Google Scholar
  4. 4.
    A. Dünser, R. Grasset, and M. Billinghurst, “A survey of evaluation techniques used in augmented reality studies,” presented at the ACM SIGGRAPH ASIA 2008 courses, Singapore, 2008.Google Scholar
  5. 5.
    J. Nielsen, Usability Engineering. San Francisco: Morgan Kaufmann, 1993.zbMATHGoogle Scholar
  6. 6.
    H. Sharp, Y. Rogers, and J. Preece, Interaction Design: Beyond Human-Computer Interaction: Wiley, 2007.Google Scholar
  7. 7.
    P. M. Fitts, “The information capacity of the human motor system in controlling the amplitude of movement,” Journal of Experimental Psychology, vol. 47, pp. 381–391, 1954.CrossRefGoogle Scholar
  8. 8.
    R. S. Dicks, “Mis-usability: on the uses and misuses of usability testing,” presented at the Proceedings of the 20th annual international conference on Computer documentation, Toronto, Ontario, Canada, 2002.Google Scholar
  9. 9.
    J. Nielsen, Designing Web Usability. Indianapolis, IN, USA: New Rivers, 2000.Google Scholar
  10. 10.
    S. Greenberg and B. Buxton, “Usability evaluation considered harmful (some of the time),” CHI ’08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, 2008.Google Scholar
  11. 11.
    D. A. Bowman, J. L. Gabbard, and D. Hix, “A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods,” Presence – Teleoperators and Virtual Environments, vol. 11, pp. 404–424, 2002.Google Scholar
  12. 12.
    C. Bach and D. L. Scapin, “Adaptation of Ergonomic Criteria to Human-Virtual Environments Interactions,” presented at the INTERACT 2003.Google Scholar
  13. 13.
    A. Sutcliffe and B. Gault, “Heuristic evaluation of virtual reality applications,” Interacting with Computers, vol. 16, pp. 831–849, 2004.CrossRefGoogle Scholar
  14. 14.
    K. M. Stanney, M. Mollaghasemi, L. Reeves, R. Breaux, and D. A. Graeber, ”Usability engineering of virtual environments (VEs): identifying multiple criteria that drive effective VE system design,” Int. J. Hum.-Comput. Stud., vol. 58, pp. 447–481, 2003.CrossRefGoogle Scholar
  15. 15.
    A. Sutcliffe and K. Kaur, “Evaluating the usability of virtual reality user interfaces,” Behaviour and Information Technology, vol. 19, 2001.Google Scholar
  16. 16.
    J. Nielsen and R. Molich, “Heuristic evaluation of user interfaces,” presented at the Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, Seattle, Washington, United States, 1990.Google Scholar
  17. 17.
    W. Newman, “A Preliminary Analysis of the Products of HCI Research, Using Pro Forma Abstracts,” presented at the CHI, Boston, MA, 1994.Google Scholar
  18. 18.
    M. Fjeld, “Introduction: Augmented Reality-Usability and Collaborative Aspects,” International Journal of Human-Computer Interaction, vol. 16, p. 387–393, 2003.CrossRefGoogle Scholar
  19. 19.
    R. T. Azuma, “A Survey of Augmented Reality,” Presence – Teleoperators and Virtual Environments, vol. 6, pp. 355–385, 1997.Google Scholar
  20. 20.
    A. Sutcliffe and K. Kaur, “Evaluating the usability of virtual reality user interfaces,” Behaviour and Information Technology, vol. 19, 2000.Google Scholar
  21. 21.
    D. Hix, J. L. Gabbard, J. E. S. II, M. A. Livingston, T. H. Höllerer, S. J. Julier, Y. Baillot, and D. Brown, “A Cost-Effective Usability Evaluation Progression for Novel Interactive Systems,” presented at the Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS’04) - Track 9 - Volume 9, 2004.Google Scholar
  22. 22.
    S. W. Gilroy, M. Cavazza, and M. Benayoun, “Using affective trajectories to describe states of flow in interactive art,” presented at the Proceedings of the International Conference on Advances in Computer Enterntainment Technology, Athens, Greece, 2009.Google Scholar
  23. 23.
    A. Morrison, A. Oulasvirta, P. Peltonen, S. Lemmela, G. Jacucci, G. Reitmayr, J. Näsänen, and A. Juustila, “Like bees around the hive: a comparative study of a mobile augmented reality map,” presented at the Proceedings of the 27th international conference on Human factors in computing systems, Boston, MA, USA, 2009.Google Scholar
  24. 24.
    E. Hughes, E. Smith, C. B. Stapleton, and D. E. Hughes, “Augmenting Museum Experiences with Mixed Reality,” presented at the Knowledge Sharing and Collaborative Engineering, St. Thomas, US Virgin Islands, 2004.Google Scholar
  25. 25.
    L. Barkhuus and J. A. Rode, “From Mice to Men – 24 years of Evaluation in CHI,” presented at the CHI, San Jose, USA, 2007.Google Scholar
  26. 26.
    J. L. Gabbard and J. E. Swan, “Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design,” IEEE Transactions on Visualization and Computer Graphics, vol. 14, pp. 513–525, 2008.CrossRefGoogle Scholar
  27. 27.
    B. Knörlein, M. D. Luca, and M. Harders, “Influence of visual and haptic delays on stiffness perception in augmented reality,” presented at the Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality, 2009.Google Scholar
  28. 28.
    A. Dünser, K. Steinbügl, H. Kaufmann, and J. Glück, “Virtual and augmented reality as spatial ability training tools,” presented at the Proceedings of the 7th ACM SIGCHI New Zealand chapter’s international conference on Computer-human interaction: design centered HCI, Christchurch, New Zealand, 2006.Google Scholar
  29. 29.
    H. Kaufmann, K. Steinbügl, A. Dünser, and J. Glück, “General Training of Spatial Abilities by Geometry Education in Augmented Reality,” Annual Review of CyberTherapy and Telemedicine: A Decade of VR, vol. 3, pp. 65–76, 2005.Google Scholar
  30. 30.
    CEEB College Entrance Examination Board, Special Aptitude Test in Spatial Relations MCT: CEEB, 1939.Google Scholar
  31. 31.
    G. K. Bennett, H. G. Seashore, and A. G. Wesman, Differential Aptitude Tests, Forms S and T. New York: The Psychological Corporation, 1973.Google Scholar
  32. 32.
    M. Peters, B. Laeng, K. Latham, M. Jackson, R. Zaiyouna, and C. Richardson, “A redrawn Vandenberg and Kuse mental rotations test: Different versions and factors that affect performance,” Brain and Cognition, vol. 28, pp. 39–58, 1995.CrossRefGoogle Scholar
  33. 33.
    M. Hegarty and D. Waller, “A dissociation between mental rotation and perspective-taking spatial abilities,” Intelligence, vol. 32, pp. 175–191, 2004.CrossRefGoogle Scholar
  34. 34.
    M. Billinghurst, H. Kato, K. Kiyokawa, D. Belcher, and I. Poupyrev, “Experiments with Face-To-Face Collaborative AR Interfaces,” Virtual Reality, vol. 6, pp. 107–121, 2002.CrossRefGoogle Scholar
  35. 35.
    S. Nilsson and B. Johansson, “Acceptance of augmented reality instructions in a real work setting,” presented at the CHI ’08 extended abstracts on Human factors in computing systems, Florence, Italy, 2008.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.The Human Interface Technology Laboratory, New Zealand (HIT Lab NZ)The University of CanterburyChristchurchNew Zealand

Personalised recommendations