Skip to main content
SpringerLink
Log in

Concepts and Subjective Measures for Evaluating User Experience of Mobile Augmented Reality Services

  • Chapter
  • First Online:
Human Factors in Augmented Reality Environments

Abstract

Augmented reality (AR) can demonstrate significant novelty aspects for a user of mobile systems, resulting from both the interaction paradigm and the novel activities and information content made available. Having recently entered the mobile domain, AR has great potential for creating a rich and multifaceted user experience (UX) in various mobile application areas for regular consumers. However, successful design or evaluation of the subjective UX requires insight into the subjective experiences that can take place with mobile AR and how the different elements of future mobile AR services might affect them. This chapter builds on our earlier findings about early adopters’ expectations of future mobile AR services. Based on this understanding we highlight an extensive set of different characteristics of experience that mobile AR services are expected to create. The identified experience categories serve as inspiration and targets for design. Furthermore, to provide mobile AR service developers with practical evaluation tools, we propose a set of measures to be utilized in subjective user evaluations of future AR services. Finally, we discuss related methodological issues to consider in planning UX evaluations of mobile AR services.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. AttrakDiff. Online: www.attrakdiff.com, accessed: December 2011

  2. Bach C, Scapin D (2004) Obstacles and perspectives for evaluating mixed reality systems usability. IUI-CADUI Workshop on Exploring the Design and Engineering of Mixed Reality Systems (MIXER)

    Google Scholar 

  3. Billinghurst M, Kato H (2002) Collaborative augmented reality. Communications of the ACM 45(7):64–70

    Article  Google Scholar 

  4. Brooke J (1996) SUS: a “quick and dirty” usability scale. In Jordan PW, Thomas B, Weerdmeester BA, McClelland AL (ed) Usability Evaluation in Industry. Taylor and Francis

    Google Scholar 

  5. Buccini M, Padovani S (2007) Typology of experiences. Proceedings of DPPI 2007. ACM, pp 495–504

    Google Scholar 

  6. Carter S, Mankoff J (2004) Challenges for ubicomp evaluation. EECS Technical Reports, CSD-04-1331. University of California, Berkeley

    Google Scholar 

  7. Curtis D, Mizell D, Gruenbaum P, Janin A (1999) Several devils in the details: Making an AR application work in the airplane factory. Proceedings of IWAR’98. AK Peters, Massachusetts, 48p

    Google Scholar 

  8. Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3):319–339

    Article  Google Scholar 

  9. Desmet PMA (2002) Designing emotions. Dissertation, Delft University of Technology

    Google Scholar 

  10. Desmet P, Hekkert P (2007) Framework of product experience. International Journal of Design 1(1):57–66

    Google Scholar 

  11. Desmet PMA, Overbeeke CJ, Tax SJET (2001) Designing products with added emotional value; Development and application of an approach for research through design. The Design Journal 4(1):32–47

    Article  Google Scholar 

  12. Dünser A, Grasset R, Billinghurst M (2008) A survey of evaluation techniques used in augmented reality studies. Proceedings of ACM SIGGRAPH 2008. ACM Press, New York

    Google Scholar 

  13. Ermi L, Mäyrä F (2007) Fundamental components of the gameplay experience: Analysing immersion. In: de Castell S and Jenson J (ed) Changing views: worlds in play - International perspectives on digital games research. Peter Lang, New York, pp 37–53

    Google Scholar 

  14. Fields B, Amaldi P, Wong W, Gill S (2007) Editorial: In-use, in-situ: Extending field research methods. International Journal of Human Computer Interaction 22(1):1–6

    Google Scholar 

  15. Fogg BJ (2003) Persuasive technology: Using computers to change what we think and do. Morgan Kaufmann publishers, San Francisco, 283p

    Google Scholar 

  16. Gabbard JL, Swan JE (2008) Usability engineering for augmented reality: employing user-based studies to inform design. IEEE Transactions on visualization and computer graphics 14(3):513–524

    Article  Google Scholar 

  17. Gabbard JL, Hix D, Swan II JE (1999) User-centered design and evaluation of virtual environments. IEEE Computer graphics and applications 19(6):51–59

    Article  Google Scholar 

  18. Gandy M et al. (2010) Experience with an AR evaluation test bed: presence, performance, and physiological measurement. Proceedings of ISMAR 2010. IEEE, pp 127–136

    Google Scholar 

  19. Google Goggles. Online: http://www.google.com/mobile/goggles/, accessed: December 2011

  20. Hart S, Staveland L (1988) Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Hancock P, Meshkati N (Ed), Human mental workload. North Holland, Amsterdam, pp 139–183

    Google Scholar 

  21. Hassenzahl M (2003) The thing and I: Understanding the relationship between user and product. In: Blythe M, Monk AF, Overbeeke K, Wright P (ed) Funology: From Usability to Enjoyment. Kluwer Academic, pp 31–42

    Google Scholar 

  22. Hassenzahl M (2004) The interplay of beauty, goodness, and usability in interactive products. Hum.-Comput. Interact. 19(4):319–349

    Article  Google Scholar 

  23. Hassenzahl M, Ullrich D (2007) To do or not to do: Differences in user experience and retrospective judgments depending on the presence or absence of instrumental goals. Interacting with Computers 19(4):429–437

    Article  Google Scholar 

  24. Hsu CL, Lu HP, Hsu HH (2007) Adoption of the mobile Internet: an empirical study of multimedia message service (MMS). Omega, 35(6):715–726

    Article  Google Scholar 

  25. Hollerer T, Feiner S (2004) Mobile augmented reality. In: Karimi H and Hammad A (eds) Telegeoinformatics: Location-Based Computing and Services. Taylor & Francis Books Ltd, London.

    Google Scholar 

  26. ISO FDIS 9241–210:2009 (2009) Ergonomics of human system interaction - Part 210: Human-centred design for interactive systems (formerly known as 13407). International organization for standardization, ISO

    Google Scholar 

  27. Jordan P (2002) Designing pleasurable products. CRC Press

    Google Scholar 

  28. Junaio. Online: www.junaio.com, accessed: December 2011

  29. Kelley JF (1984) An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Office Information Systems 2(1):26–41

    Article  MathSciNet  Google Scholar 

  30. Korhonen H, Montola M, Arrasvuori J (2009) Understanding playful user experience through digital games. Proceedings of DPPI’09, pp 274–285

    Google Scholar 

  31. Kruijff E, Swan E, Feiner S (2010) Perceptual issues in augmented reality revisited. Proceedings of ISMAR 2010. IEEE, pp 3–12

    Google Scholar 

  32. Kujala S, Nurkka P (2009) Product symbolism in designing for user experience. Proceedings of DPPI’09. Springer, pp 176–186

    Google Scholar 

  33. Lang PJ (1980) Behavioral treatment and bio-behavioral assessment: computer applications. In: Sidowski JB, Johnson JH, Williams TA (Ed), Technology in mental health care delivery systems. Ablex, Norwood, pp 119–137

    Google Scholar 

  34. Law EL, Schaik P (2010) Modelling user experience – An agenda for research and practice. Interacting with computers, 22(5):313–322

    Article  Google Scholar 

  35. Law EL, Roto V, Hassenzahl M, Vermeeren AP (2009) Understanding, scoping and defining user experience: A survey approach. Proceedings of CHI’09. ACM Press, pp 719–728

    Google Scholar 

  36. Layar. Online: www.layar.com, accessed: December 2011

  37. Looser J, Billinghurst M, Grasset R, Cockburn A (2007) An evaluation of virtual lenses for object selection in augmented reality. Proceedings of GRAPHITE’07. ACM Press, pp 203–210

    Google Scholar 

  38. Mackay WE (1996) Augmenting Reality: A new paradigm for interacting with computers, La Recherche, Mar. 1996

    Google Scholar 

  39. McCarthy J, Wright P, Wallace J, Dearden A (2006) The experience of enchantment in human–computer interaction. Personal and Ubiquitous Computing 10(6):369–378. doi: 10.1007/s00779-005-0055-2

    Article  Google Scholar 

  40. Nielsen J (1993) Usability engineering. Morgan Kaufmann, San Diego, 362 p

    MATH  Google Scholar 

  41. Nilsson S (2010) Augmentation in the wild: user centered development and evaluation of augmented reality applications. Dissertation, Linköping university

    Google Scholar 

  42. Olsson T (2009) Understanding collective content: Purposes, characteristics and collaborative practices. Proceedings of Communities and Technologies 2009. ACM Press, pp 21–30

    Google Scholar 

  43. Olsson T, Salo M (2011) Online user survey on current mobile augmented reality applications. Proceedings of ISMAR 2011. IEEE, pp 75–84

    Google Scholar 

  44. Olsson T, Väänänen-Vainio-Mattila K (2011) Expected user experience with mobile ­augmented reality services. Workshop of Mobile Augmented Reality, MobileHCI 2011. ACM Press, New York

    Google Scholar 

  45. Olsson T, Ihamäki P, Lagerstam E, Ventä-Olkkonen L, Väänänen-Vainio-Mattila K (2009) User expectations for mobile mixed reality services. Proceedings of ECCE’09. ACM Press, 177–184

    Google Scholar 

  46. Olsson T, Kärkkäinen T, Ventä-Olkkonen L, Lagerstam E (2012) User evaluation of mobile augmented reality scenarios. Forthcoming in Journal of ambient intelligence and smart environments, IOS Press

    Google Scholar 

  47. Olsson T, Lagerstam E, Kärkkäinen T, Väänänen-Vainio-Mattila K (2011) Expected user experience of mobile augmented reality services: A user study in the context of shopping centers. Journal of Personal and Ubiquitous Computing, Springer, DOI: 10.1007/s00779-011-0494-x

  48. Roto V, Law EL, Vermeeren AP, Hoonhout J (eds) (2010) User experience white paper: Results from Dagstuhl seminar on demarcating user experience. Available at: http://www.allaboutux.org/files/UX-WhitePaper.pdf

  49. Scholtz J, Consolvo S (2004) Towards a discipline for evaluating ubiquitous computing applications. Report from National Institute of Standards and Technology, IRS-TR-04-004.

    Google Scholar 

  50. Stanney K (1995) Realizing the full potential of virtual reality: Human factors issues that could stand in the way. Proceedings of VRAIS’95. IEEE, pp 28–33

    Google Scholar 

  51. Stevens SS (1946) On the theory of scales of measurement. Science 103(2684): 677–680

    Article  MATH  Google Scholar 

  52. Swan JE, Gabbard JL (2005) Survey of user-based experimentation in augmented reality. Proceedings of 1st International Conference on Virtual Reality

    Google Scholar 

  53. Theofanos M, Scholtz J (2005) A framework for evaluation of ubicomp applications. Workshop on Social Implications of Ubiquitous Applications, CHI’05. ACM Press, New York

    Google Scholar 

  54. Thompson ER (2007) Development and validation of an internationally reliable short-form of the positive and negative affect schedule (PANAS). Journal of Cross-Cultural Psychology 38(2):227–242

    Article  Google Scholar 

  55. Trochim W, Donnelly JP (2006) The research methods knowledge base. Atomic Dog, 3rd ­edition, 361p

    Google Scholar 

  56. Vaittinen T, Kärkkäinen T, Olsson T (2010) A diary study on annotating locations with mixed reality information. Proceedings of MUM 2010, Article no. 21

    Google Scholar 

  57. Vermeeren AP, Law EL, Roto V, Obrist M, Hoonhout J, Väänänen-Vainio-Mattila K (2010) User experience evaluation methods: current state and development needs. Proceedings of NordiCHI’10. ACM Press, New York, pp 521–530

    Google Scholar 

  58. Väänänen-Vainio-Mattila K, Roto V, Hassenzahl M (2008) Towards practical user experience evaluation methods. Proceedings of the International Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM), pp 19–22

    Google Scholar 

  59. Wagner D, Schmalstieg D (2009) Making augmented reality practical on mobile phones, Part 1. IEEE Computer Graphics and Applications 29(3):12–15

    Article  Google Scholar 

  60. Wellner P, Mackay W, Gold R (1993) Back to the real world. Communications of the ACM 36(7):24–26

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Unit of Human-Centered Technology, Tampere University of Technology, Korkeakoulunkatu 6, 33720, Tampere, Finland

    Thomas Olsson

Authors
  1. Thomas Olsson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Olsson .

Editor information

Editors and Affiliations

  1. CSIRO ICT Centre, Pembroke Road, Marsfield, 1710, New South Wales, Australia

    Weidong Huang

  2. CSIRO ICT Centre, Pembroke Road, Marsfield, 1710, New South Wales, Australia

    Leila Alem

  3. Mail Code 5580, Washington, 20375, District of Columbia, USA

    Mark A. Livingston

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

Olsson, T. (2013). Concepts and Subjective Measures for Evaluating User Experience of Mobile Augmented Reality Services. In: Huang, W., Alem, L., Livingston, M. (eds) Human Factors in Augmented Reality Environments. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-4205-9_9

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-4205-9_9

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4614-4204-2

  • Online ISBN: 978-1-4614-4205-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation