Bridging the Gap Between Field- and Lab-Based User Studies for Location-Based Services

  • Ioannis Delikostidis
  • Holger Fritze
  • Thore Fechner
  • Christian KrayEmail author
Part of the Lecture Notes in Geoinformation and Cartography book series (LNGC)


There is a long-running debate about how to best evaluate mobile location-based services with users: in the lab or in the field? In this paper, we investigate how to combine benefits of both methods using an Immersive Video Environment (IVE), providing a convincing audio-visual simulation of real-world settings. We contrast three methods to evaluate mobile navigation systems: one in the field, one in the lab and one “hybrid” solution (IVE). We found that using the IVE allowed us to identify nearly the same number of major usability problems as the field test. We also observed similarities between the field study and the IVE study in terms of participants’ performance, which provides initial evidence that in some settings, an IVE study may yield results comparable to a field study.


Usability evaluation Mobile navigation systems Immersive environments Field-based testing Lab-based testing 


  1. Betiol A, Cybis AW (2005) Usability testing of mobile devices: a comparison of three approaches. In: Proceedings of interact 2005. Springer, Berlin, pp 470–481Google Scholar
  2. Chervest K, Mitchell K, Davies N (2002) The role of adaptive hypermedia in a context-aware tourist guide. Commun ACM-The Adapt Web 45(5):47–51Google Scholar
  3. Delikostidis I, van Elzakker CPJM (2009) Geo-identification and pedestrian navigation with geo-mobile applications: how do users proceed? In: Proceedings of the 5th international conference on location based services and telecartography. Springer, Berlin, pp 185–206Google Scholar
  4. Duh HBL, Tan CBG, Chen VHH (2006) Usability evaluation for mobile device: a comparison of laboratory and field tests. In: Proceedings of mobile HCI ’06. ACM, New York, pp 181–186Google Scholar
  5. Goodman J, Brewster S, Gray P (2004) Using field experiments to evaluate mobile guides. In: Proceedings of HCI in mobile guides, workshop at mobile HCI 2004. Springer, Glaskow, pp 38–48Google Scholar
  6. Kaikkoken A, Kekäläinen A, Cankar M, Kallio T, Kankainen A (2008) Will laboratory test results be valid in mobile contexts? In: Lumsden J (ed) Handbook of research on user interface design and evaluation for mobile technology. Information Science Reference, Hersey, pp 897–909CrossRefGoogle Scholar
  7. Kimber J, Georgievski M, Sharda N (2005) Developing usability testing systems and procedures for mobile tourism services. In: Proceedings of the annual conference on information technology in the hospitality industry, HITA 2005. Los Angeles, USA, pp 79–96Google Scholar
  8. Kjeldskov J, Paay J (2012) A longitudinal review of mobile HCI research methods. In: Proceedings of Mobile HCI ’12. ACM Press, San Francisco, pp 69–78Google Scholar
  9. Kjeldskov J, Skov MB, Als BS, Høegh RT (2004) Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the field. In: Proceedings of mobile HCI ‘04. Springer, Berlin, pp 61–73Google Scholar
  10. Loomis JM, Blascovich JJ, Beall AC (1999) Immersive virtual environment technology as a basic research tool in psychology. Behav Res Methods Instrum Comput 31(4):557–564CrossRefGoogle Scholar
  11. Jensen KL, Larsen LB (2008) The challenges of evaluating the mobile and ubiquitous user experience. In: Proceedings of 2nd international workshop on improved mobile user experience, pp 198–207Google Scholar
  12. Lumsden J (2008) Handbook of research on user interface design and evaluation for mobile technology. Information Science Reference, HersheyCrossRefGoogle Scholar
  13. Lund AM (2008) Measuring usability with the use questionnaire. Accessed 17 May 2014
  14. Nielsen CM, Overgaard M, Pedersen MB, Stage J, Stenild S (2006) It’s worth the hassle!: the added value of evaluating the usability of mobile systems in the field. NordiCHI ’06. ACM, New York, pp 272–280CrossRefGoogle Scholar
  15. Nielsen J (1994) Estimating the number of subjects needed for a think aloud test. Int J Hum Comput Stud 41(3):385–397CrossRefGoogle Scholar
  16. Ostkamp M, Kray C (2014) Supporting design, prototyping, and evaluation of public display systems. EICS’2014. ACM, New York, pp 263–272Google Scholar
  17. Palen L, Salzman M, Youngs E (2000) Going wireless: behavior and practice of new mobile phone users. In: Proceedings of CSCW ’00. ACM, Philadelphia, pp 201–210Google Scholar
  18. Rogers Y (2011) Interaction design gone wild: striving for wild theory. Interactions 18(4):58–62CrossRefGoogle Scholar
  19. Roto V, Oulasvirta A, Haikarainen T, Kuorelahti J, Lehmuskallio H, Nyyssönen T (2004) Examining mobile phone use in the wild with Quasi-experimentation. Helsinki Institute for Information Technology, FinlandGoogle Scholar
  20. Schellenbach M, Lövdén M, Verrel J, Krüger A, Lindenberger U (2009) Adult age differences in familiarization to treadmill walking within virtual environments. Gait Posture 31(3):295–299CrossRefGoogle Scholar
  21. Singh P, Ha NH, Kuang Z, Olivier P, Kray C, Blythe P, James P (2006) Immersive video as a rapid prototyping and evaluation tool for mobile and ambient applications. In: Proceedings of mobile HCI ’06. ACM, New York, p 264Google Scholar
  22. Snowdon C, Kray C (2009) Exploring the use of landmarks for mobile navigation support in natural environments. In: Proceedings of mobile HCI ’09. ACM, New York, pp 1–10Google Scholar
  23. van Elzakker CPJM, Delikostidis I, van Oosterom PPJM (2008) Field-based usability evaluation methodology for mobile geo-applications. The Cartographic J 45(2):139–149CrossRefGoogle Scholar
  24. van Veen HAHC, Distler HK, Braun SJ, Bülthoff HH (1998) Navigating through a virtual city: using virtual reality technology to study human action and perception. Future Gener Comput Syst 14(3–4):231–242CrossRefGoogle Scholar
  25. Winters JM, Story MF, Campbell S, Lemke M, Danturthi D, Barr A, Rempel DM (2001) Mobile usability lab: a tool for studying medical device accessibility for users with diverse abilities. In: Proceedings of the 27th RESNA conferenceGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Ioannis Delikostidis
    • 1
  • Holger Fritze
    • 1
  • Thore Fechner
    • 1
  • Christian Kray
    • 1
    Email author
  1. 1.Institute for Geoinformatics (ifgi)University of MuensterMuensterGermany

Personalised recommendations