Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field

  • Jesper Kjeldskov
  • Mikael B. Skov
  • Benedikte S. Als
  • Rune T. Høegh
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3160)


Evaluating the usability of mobile systems raises new concerns and questions, challenging methods for both lab and field evaluations. A recent literature study showed that most mobile HCI research projects apply lab-based evaluations. Nevertheless, several researchers argue in favour of field evaluations as mobile systems are highly context-dependent. However, field-based usability studies are difficult to conduct, time consuming and the added value is unknown. Contributing to this discussion, this paper compares the results produced by a laboratory- and a field-based evaluation of the same context-aware mobile system on their ability to identify usability problems. Six test subjects used the mobile system in a laboratory while another six used the system in the field. The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.


Mobile Device Usability Evaluation Test Subject Field Evaluation Electronic Patient Record 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abowd, G., Mynatt, E.: Charting past, present and future research in ubiquitous computing. ACM Transactions on Computer-Human Interaction 7(1), 29–58 (2000)CrossRefGoogle Scholar
  2. 2.
    Bardram, J., Kjær, T.A.K., Nielsen, C.: Supporting Local Mobility in Healthcare by Application Roaming among Heterogeneous Devices. In: Chittaro, L. (ed.) Mobile HCI 2003. LNCS, vol. 2795, Springer, Heidelberg (2003)CrossRefGoogle Scholar
  3. 3.
    Barkhuus, L., Dey, A.: Is Context-Aware Computing Taking Control away from the User? Three Levels of Interactivity Examined. In: Dey, A.K., Schmidt, A., McCarthy, J.F. (eds.) UbiComp 2003. LNCS, vol. 2864, pp. 149–156. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  4. 4.
    Bohnenberger, T., Jameson, A., Krüger, A., Butz, A.: Location-Aware Shopping Assistance: Evaluation of a Decision-Theoretic Approach. In: Paternó, F. (ed.) Mobile HCI 2002. LNCS, vol. 2411, p. 155. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  5. 5.
    Brewster, S.: Overcoming the Lack of Screen Space on Mobile Computers. Personal and Ubiquitous Computing 6, 188–205 (2002)CrossRefGoogle Scholar
  6. 6.
    Esbjörnsson, M., Juhlin, O., Östergren, M.: Motorcyclists Using Hocman – Field Trials on Mobile Interaction. In: Chittaro, L. (ed.) Mobile HCI 2003. LNCS, vol. 2795, Springer, Heidelberg (2003)CrossRefGoogle Scholar
  7. 7.
    Graham, R., Carter, C.: Comparison of Speech Input and Manual Control of In-Car Devices while on-the-move. In: Proceedings of the Second Workshop on Human Computer Interaction with Mobile Devices, Mobile HCI 1999, Edinburgh, Scotland (1999)Google Scholar
  8. 8.
    Johnson, P.: Usability and Mobility; Interactions on the move. In: Proceedings of the First Workshop on Human-Computer Interaction with Mobile Devices, Glasgow, Scotland, GIST Technical Report G98-1 (1998)Google Scholar
  9. 9.
    Kjeldskov, J., Stage, J.: New Techniques for Usability Evaluation of Mobile Systems. International Journal of Human-Computer Studies (IJHCS) 60, 599–620 (2004)CrossRefGoogle Scholar
  10. 10.
    Kjeldskov, J., Graham, C.: A Review of Mobile HCI Research Methods. In: Chittaro, L. (ed.) Mobile HCI 2003. LNCS, vol. 2795, Springer, Heidelberg (2003)CrossRefGoogle Scholar
  11. 11.
    Kjeldskov, J., Skov, M.B.: Creating a Realistic Laboratory Setting: A Comparative Study of Three Think-Aloud Usability Evaluations of a Mobile System. In: Proceedings of the 9th IFIP TC13 International Conference on Human Computer Interaction, Interact 2003, Zürich, Switzerland, IOS Press, Amsterdam (2003)Google Scholar
  12. 12.
    Lai, J., Cheng, K., Green, P., Tsimhoni, O.: On the Road and on the Web? Comprehension of synthetic speech while driving. In: Proceedings of CHI 2001, pp. 206–212. ACM, New York (2001)Google Scholar
  13. 13.
    Molich, R.: Usable Web Design (In Danish). Ingeniøren | bøger (2000)Google Scholar
  14. 14.
    Nielsen, C.: Testing in the Field. In: Proceedings of the third Asia Pacific Computer Human Interaction Conference, IEEE Computer Society, Los Alamitos (1998)Google Scholar
  15. 15.
    Nielsen, J.: Usability Engineering. Morgan Kaufmann, San Francisco (1993)zbMATHGoogle Scholar
  16. 16.
    Pascoe, J., Ryan, N., Morse, D.: Using While Moving: HCI Issues in Fieldwork Environments. Transactions on Computer-Human Interaction 7(3), 417–437 (2000)CrossRefGoogle Scholar
  17. 17.
    Pirhonen, A., Brewster, S.A., Holguin, C.: Gestural and audio metaphors as a means of control for mobile devices. In: Proceedings of CHI 2002, ACM, New York (2002)Google Scholar
  18. 18.
    Rantanen, J., Impio, J., Karinsalo, T., Reho, A., Tasanen, M., Vanhala, J.: Smart Clothing Prototype for the Artic Environment. Personal and Ubiquitous Computing 6, 3–16 (2002)CrossRefGoogle Scholar
  19. 19.
    Rubin, J.: Handbook of Usability Testing. Wiley, Chichester (1994)Google Scholar
  20. 20.
    Salvucci, D.D.: Predicting the Effects of In-Car Interfaces on Driver Behaviour using a Cognitive Architecture. In: Proceedings of CHI 2001, New York, pp. 120–127 (2001)Google Scholar
  21. 21.
    Schilit, B.N., Theimer, M.M.: Disseminating Active Map information to Mobile Hosts. IEEE Network 8(5), 22–32 (1994)CrossRefGoogle Scholar
  22. 22.
    Skov, M.B., Høegh, R.T.: Supporting Cooperation and Coordination in a Hospital Ward by a Context-Aware Mobile Electronic Patient Record. Submitted to the Journal of Computer-Supported Cooperative Work, Elsevier (2004)Google Scholar
  23. 23.
    Urban, M., Kunath, M.: Design, Deployment and Evaluation of a Clinical Information System which uses Mobile Computers and Workstations. In: Proceedings of the 2nd conference on Mobile Computing in Medicine, Heidelberg (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Jesper Kjeldskov
    • 1
  • Mikael B. Skov
    • 1
  • Benedikte S. Als
    • 1
  • Rune T. Høegh
    • 1
  1. 1.Department of Computer ScienceAalborg UniversityFredrik Bajers Vej 7EDenmark

Personalised recommendations