Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field
Evaluating the usability of mobile systems raises new concerns and questions, challenging methods for both lab and field evaluations. A recent literature study showed that most mobile HCI research projects apply lab-based evaluations. Nevertheless, several researchers argue in favour of field evaluations as mobile systems are highly context-dependent. However, field-based usability studies are difficult to conduct, time consuming and the added value is unknown. Contributing to this discussion, this paper compares the results produced by a laboratory- and a field-based evaluation of the same context-aware mobile system on their ability to identify usability problems. Six test subjects used the mobile system in a laboratory while another six used the system in the field. The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.
Unable to display preview. Download preview PDF.
- 7.Graham, R., Carter, C.: Comparison of Speech Input and Manual Control of In-Car Devices while on-the-move. In: Proceedings of the Second Workshop on Human Computer Interaction with Mobile Devices, Mobile HCI 1999, Edinburgh, Scotland (1999)Google Scholar
- 8.Johnson, P.: Usability and Mobility; Interactions on the move. In: Proceedings of the First Workshop on Human-Computer Interaction with Mobile Devices, Glasgow, Scotland, GIST Technical Report G98-1 (1998)Google Scholar
- 11.Kjeldskov, J., Skov, M.B.: Creating a Realistic Laboratory Setting: A Comparative Study of Three Think-Aloud Usability Evaluations of a Mobile System. In: Proceedings of the 9th IFIP TC13 International Conference on Human Computer Interaction, Interact 2003, Zürich, Switzerland, IOS Press, Amsterdam (2003)Google Scholar
- 12.Lai, J., Cheng, K., Green, P., Tsimhoni, O.: On the Road and on the Web? Comprehension of synthetic speech while driving. In: Proceedings of CHI 2001, pp. 206–212. ACM, New York (2001)Google Scholar
- 13.Molich, R.: Usable Web Design (In Danish). Ingeniøren | bøger (2000)Google Scholar
- 14.Nielsen, C.: Testing in the Field. In: Proceedings of the third Asia Pacific Computer Human Interaction Conference, IEEE Computer Society, Los Alamitos (1998)Google Scholar
- 17.Pirhonen, A., Brewster, S.A., Holguin, C.: Gestural and audio metaphors as a means of control for mobile devices. In: Proceedings of CHI 2002, ACM, New York (2002)Google Scholar
- 19.Rubin, J.: Handbook of Usability Testing. Wiley, Chichester (1994)Google Scholar
- 20.Salvucci, D.D.: Predicting the Effects of In-Car Interfaces on Driver Behaviour using a Cognitive Architecture. In: Proceedings of CHI 2001, New York, pp. 120–127 (2001)Google Scholar
- 22.Skov, M.B., Høegh, R.T.: Supporting Cooperation and Coordination in a Hospital Ward by a Context-Aware Mobile Electronic Patient Record. Submitted to the Journal of Computer-Supported Cooperative Work, Elsevier (2004)Google Scholar
- 23.Urban, M., Kunath, M.: Design, Deployment and Evaluation of a Clinical Information System which uses Mobile Computers and Workstations. In: Proceedings of the 2nd conference on Mobile Computing in Medicine, Heidelberg (2002)Google Scholar