Abstract
Evaluating the usability of mobile systems raises new concerns and questions, challenging methods for both lab and field evaluations. A recent literature study showed that most mobile HCI research projects apply lab-based evaluations. Nevertheless, several researchers argue in favour of field evaluations as mobile systems are highly context-dependent. However, field-based usability studies are difficult to conduct, time consuming and the added value is unknown. Contributing to this discussion, this paper compares the results produced by a laboratory- and a field-based evaluation of the same context-aware mobile system on their ability to identify usability problems. Six test subjects used the mobile system in a laboratory while another six used the system in the field. The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Abowd, G., Mynatt, E.: Charting past, present and future research in ubiquitous computing. ACM Transactions on Computer-Human Interaction 7(1), 29–58 (2000)
Bardram, J., Kjær, T.A.K., Nielsen, C.: Supporting Local Mobility in Healthcare by Application Roaming among Heterogeneous Devices. In: Chittaro, L. (ed.) Mobile HCI 2003. LNCS, vol. 2795, Springer, Heidelberg (2003)
Barkhuus, L., Dey, A.: Is Context-Aware Computing Taking Control away from the User? Three Levels of Interactivity Examined. In: Dey, A.K., Schmidt, A., McCarthy, J.F. (eds.) UbiComp 2003. LNCS, vol. 2864, pp. 149–156. Springer, Heidelberg (2003)
Bohnenberger, T., Jameson, A., Krüger, A., Butz, A.: Location-Aware Shopping Assistance: Evaluation of a Decision-Theoretic Approach. In: Paternó, F. (ed.) Mobile HCI 2002. LNCS, vol. 2411, p. 155. Springer, Heidelberg (2002)
Brewster, S.: Overcoming the Lack of Screen Space on Mobile Computers. Personal and Ubiquitous Computing 6, 188–205 (2002)
Esbjörnsson, M., Juhlin, O., Östergren, M.: Motorcyclists Using Hocman – Field Trials on Mobile Interaction. In: Chittaro, L. (ed.) Mobile HCI 2003. LNCS, vol. 2795, Springer, Heidelberg (2003)
Graham, R., Carter, C.: Comparison of Speech Input and Manual Control of In-Car Devices while on-the-move. In: Proceedings of the Second Workshop on Human Computer Interaction with Mobile Devices, Mobile HCI 1999, Edinburgh, Scotland (1999)
Johnson, P.: Usability and Mobility; Interactions on the move. In: Proceedings of the First Workshop on Human-Computer Interaction with Mobile Devices, Glasgow, Scotland, GIST Technical Report G98-1 (1998)
Kjeldskov, J., Stage, J.: New Techniques for Usability Evaluation of Mobile Systems. International Journal of Human-Computer Studies (IJHCS) 60, 599–620 (2004)
Kjeldskov, J., Graham, C.: A Review of Mobile HCI Research Methods. In: Chittaro, L. (ed.) Mobile HCI 2003. LNCS, vol. 2795, Springer, Heidelberg (2003)
Kjeldskov, J., Skov, M.B.: Creating a Realistic Laboratory Setting: A Comparative Study of Three Think-Aloud Usability Evaluations of a Mobile System. In: Proceedings of the 9th IFIP TC13 International Conference on Human Computer Interaction, Interact 2003, Zürich, Switzerland, IOS Press, Amsterdam (2003)
Lai, J., Cheng, K., Green, P., Tsimhoni, O.: On the Road and on the Web? Comprehension of synthetic speech while driving. In: Proceedings of CHI 2001, pp. 206–212. ACM, New York (2001)
Molich, R.: Usable Web Design (In Danish). Ingeniøren | bøger (2000)
Nielsen, C.: Testing in the Field. In: Proceedings of the third Asia Pacific Computer Human Interaction Conference, IEEE Computer Society, Los Alamitos (1998)
Nielsen, J.: Usability Engineering. Morgan Kaufmann, San Francisco (1993)
Pascoe, J., Ryan, N., Morse, D.: Using While Moving: HCI Issues in Fieldwork Environments. Transactions on Computer-Human Interaction 7(3), 417–437 (2000)
Pirhonen, A., Brewster, S.A., Holguin, C.: Gestural and audio metaphors as a means of control for mobile devices. In: Proceedings of CHI 2002, ACM, New York (2002)
Rantanen, J., Impio, J., Karinsalo, T., Reho, A., Tasanen, M., Vanhala, J.: Smart Clothing Prototype for the Artic Environment. Personal and Ubiquitous Computing 6, 3–16 (2002)
Rubin, J.: Handbook of Usability Testing. Wiley, Chichester (1994)
Salvucci, D.D.: Predicting the Effects of In-Car Interfaces on Driver Behaviour using a Cognitive Architecture. In: Proceedings of CHI 2001, New York, pp. 120–127 (2001)
Schilit, B.N., Theimer, M.M.: Disseminating Active Map information to Mobile Hosts. IEEE Network 8(5), 22–32 (1994)
Skov, M.B., Høegh, R.T.: Supporting Cooperation and Coordination in a Hospital Ward by a Context-Aware Mobile Electronic Patient Record. Submitted to the Journal of Computer-Supported Cooperative Work, Elsevier (2004)
Urban, M., Kunath, M.: Design, Deployment and Evaluation of a Clinical Information System which uses Mobile Computers and Workstations. In: Proceedings of the 2nd conference on Mobile Computing in Medicine, Heidelberg (2002)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kjeldskov, J., Skov, M.B., Als, B.S., Høegh, R.T. (2004). Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. In: Brewster, S., Dunlop, M. (eds) Mobile Human-Computer Interaction - MobileHCI 2004. Mobile HCI 2004. Lecture Notes in Computer Science, vol 3160. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-28637-0_6
Download citation
DOI: https://doi.org/10.1007/978-3-540-28637-0_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23086-1
Online ISBN: 978-3-540-28637-0
eBook Packages: Springer Book Archive