Why It’s Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp

(Nominated for the Best Paper Award)
  • Yvonne Rogers
  • Kay Connelly
  • Lenore Tedesco
  • William Hazlewood
  • Andrew Kurtz
  • Robert E. Hall
  • Josh Hursey
  • Tammy Toscos
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4717)


How should Ubicomp technologies be evaluated? While lab studies are good at sensing aspects of human behavior and revealing usability problems, they are poor at capturing context of use. In-situ studies are good at demonstrating how people appropriate technologies in their intended setting, but are expensive and difficult to conduct. Here, we show how they can be used more productively in the design process. A mobile learning device was developed to support teams of students carrying out scientific inquiry in the field. An initial in-situ study showed it was not used in the way envisioned. A contextualized analysis led to a comprehensive understanding of the user experience, usability and context of use, leading to a substantial redesign. A second in-situ study showed a big improvement in device usability and collaborative learning. We discuss the findings and conclude how in-situ studies can play an important role in the design and evaluation of Ubicomp applications and user experiences.


In-situ studies design evaluation user experience usability mobile learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bell, G.: No More SMS from Jesus: Ubicomp, Religion and Techno-spiritual Practices. In: Dourish, P., Friday, A. (eds.) UbiComp 2006. LNCS, vol. 4206, pp. 141–158. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  2. 2.
    Bell, M., Chalmers, M., Barkhuus, L., Hall, M., Sherwood, S., Tennent, P., Brown, B., Rowland, D., Benford, S., Hampshire, A., Captra, M.: Interweaving mobile games with everyday life. In: Proc. of CHI, pp. 417–426 (2006)Google Scholar
  3. 3.
    Bellotti, V., Back, M., Edwards, K., Grinter, R., Henderson, A., Lopes, C.: Making sense of sensing systems: five questions for designers and researchers. In: Proc. of CHI, pp. 415–422 (2002)Google Scholar
  4. 4.
    Benford, S., Seager, W., Flintham, M., Anastasi, R., Rowland, D., Humble, J., Stanton, S., Bowers, J., Tandavanitj, N., Adams, M., Farr, J.R., Oldroyd, A., Sutton, J.: The error of our ways: the experience of self-reported position in a location-based game. In: Davies, N., Mynatt, E.D., Siio, I. (eds.) UbiComp 2004. LNCS, vol. 3205, pp. 721–730. Springer, Heidelberg (2004)Google Scholar
  5. 5.
    Benford, S., Schanädelbach, H., Koleva, B., Anastasi, R., Greenhalgh, C., Rodden, T., Green, J., Ghali, A., Pridmore, T., Gaver, B., Boucher, A., Walker, B., Pennington, S., Schmidt, A., Gellersen, H., Steed, A.: Expected, sensed, and desired: A framework for designing sensing-based interaction. Proc. of TOCHI, 12(1), 3–30 (2005)Google Scholar
  6. 6.
    Boucher, A., Gaver, W.: Developing the drift table. Interactions 13(1), 24–27 (2006)CrossRefGoogle Scholar
  7. 7.
    Brignull, H., Izadi, S., Fitzpatrick, G., Rogers, Y., Rodden, T.: The introduction of a shared interactive surface into a communal space. In: Proc. of CSCW, pp. 49–58 (2004)Google Scholar
  8. 8.
    Consolvo, S., Walker, M.: Using the experience sampling method to evaluate Ubicomp applications. IEEE Pervasive Computing Mobile and Ubiquitous Systems 2(2), 24–31 (2003)Google Scholar
  9. 9.
    Consolvo, S., Everitt, K., Smith, I., Landay, J.: Design requirements for technologies that motivate physical activity. In: Proc. of CHI, pp. 457–466 (2006)Google Scholar
  10. 10.
    Crabtree, A., Rodden, T.: Domestic routines and design for the home. Computer Supported Cooperative Work: The Journal of Collaborative Computing 13(2), 191–220 (2004)CrossRefGoogle Scholar
  11. 11.
    Dumas, J.S., Redish, J.C.: A Practical Guide to Usability Testing. Ablex, Norwood, NJ (1994)Google Scholar
  12. 12.
    Gaver, W., Dunne, T., Pacenti, E.: Cultural probes and the value of uncertainty. Interactions 11(5), 53–56 (2004)CrossRefGoogle Scholar
  13. 13.
    Hughes, J.A., Randall, D., Shapiro, D.: Faltering from ethnography to design. In: Proc. of CSCW, pp. 115–122 (1992)Google Scholar
  14. 14.
    Hull, R., Reid, J., Geelhoed, E.: Creating experiences with wearable computing. IEEE Pervasive Computing 1(4), 56–61 (2002)CrossRefGoogle Scholar
  15. 15.
    Hutchins, E., Klausen, T.: Distributed Cognition in an Airline Cockpit. In: Middleton, Engeström, Y. (eds.) Communication and Cognition at Work, pp. 15–34. Cambridge University Press, D. Cambridg (1996)Google Scholar
  16. 16.
    Intille, S., Larson, K., Tapia, E., Beaudin, J., Kaushik, P., Nawyn, J., Rockinson, R.: Using a live-in laboratory for ubiquitous computing research. In: Fishkin, K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) PERVASIVE 2006. LNCS, vol. 3968, pp. 349–365. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  17. 17.
    Kidd, C., Orr, R., Abowd, G., Atkeson, C., Essa, I., MacIntyre, B., Mynatt, E., Starner, T.: The Aware Home: A Living Laboratory for Ubiquitous Computing Research. In: Streitz, N.A., Hartkopf, V. (eds.) CoBuild 1999. LNCS, vol. 1670, pp. 191–198. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  18. 18.
    Kjeldskov, J., Skov, M., Als, B., Høegh, R.: Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the field. In: Brewster, S., Dunlop, M.D. (eds.) Mobile Human-Computer Interaction – MobileHCI 2004. LNCS, vol. 3160, pp. 61–73. Springer, Heidelberg (2004)Google Scholar
  19. 19.
    Larkin, J., Simon, H.: Why a diagram is (sometimes) worth ten thousand words. Cognitive Science 11, 65–99 (1987)CrossRefGoogle Scholar
  20. 20.
    Lin, J., Mamykina, L., Lindtner, S., Delajoux, G., Strub, H.: Fish’n’Steps: Encouraging Activity with an Interactive Computer Game. In: Dourish, P., Friday, A. (eds.) UbiComp 2006. LNCS, vol. 4206, pp. 261–278. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  21. 21.
    Mankoff, J., Dey, A., Hsieh, G., Kientz, J., Lederer, J., Ames, M.: Heuristic evaluation of ambient displays. In: Proc. of CHI, pp. 169–176 (2003)Google Scholar
  22. 22.
    Nielsen, J.: Usability engineering at a discount. In: Salvendy, G., Smith, M.J. (eds.) Human-Computer interaction on Designing and Using Human-Computer Interfaces and Knowledge Based Systems, pp. 394–401 (1989)Google Scholar
  23. 23.
    Olsen, A., Rogers, Y., Sharp, H.: The Snap Method. In: Workshop Proceedings, Designing Methods for New Users, Technologies, and Design Processes, CHI (2007)Google Scholar
  24. 24.
    Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W.R.: Mobile technologies for integrated scientific inquiry. Journal of Learning Sciences (submitted)Google Scholar
  25. 25.
    Rogers, Y., Price, S., Randell, C., Stanton-Fraser, D., Weal, M., Fitzpatrick., G.: Ubi-learning: Integrating outdoor and indoor learning experiences. Comm. of ACM 48(1), 55–59 (2005)CrossRefGoogle Scholar
  26. 26.
    Rowan, G., Mynatt, E.: Digital Family Portrait Field Trial: Support for Aging in Place. In: Proc. of CHI, pp. 521–530 (2005)Google Scholar
  27. 27.
    Salovaara, A., Jacucci, G., Oulasvirta, A., Saari, T., Kanerva, P., Kurvinen, E., Tiitta, S.: Collective creation and sense-making of mobile media. In: Proc. of CHI, pp. 1211–1220 (2006)Google Scholar
  28. 28.
    Scholtz, J., Consolvo, S.: Toward a Framework for Evaluating Ubiquitous Computing Applications. Pervasive Computing 3(2), 82–88 (2004)CrossRefGoogle Scholar
  29. 29.
    Sharp, H., Rogers, Y., Preece, J.: Interaction Design, 2nd edn. Wiley, ChichesterGoogle Scholar
  30. 30.
    Taylor, A.S., Swan, L.: Artful systems in the home. In: Proc. CHI, pp. 641–650 (2005)Google Scholar
  31. 31.
    Toscos, T., Faber, A., An, S., Gandhi, M.: Chick Clique: persuasive technology to motivate teenage girls to exercise. In: Proc. of CHI, pp. 1873–1878 (2006)Google Scholar
  32. 32.
    Winograd, T.: Bringing Design to Software. Addison Wesley, Reading (1996)Google Scholar
  33. 33.
    Yeh, R., Liao, C., Klemmer, S., Guimbretière, F., Lee, B., Kakaradov, B., Stamberger, J., Paepcke, A.: ButterflyNet: a mobile capture and access system for field biology research. In: Proc. of CHI, pp. 571–580 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Yvonne Rogers
    • 1
    • 2
  • Kay Connelly
    • 2
  • Lenore Tedesco
    • 3
  • William Hazlewood
    • 2
  • Andrew Kurtz
    • 2
  • Robert E. Hall
    • 3
  • Josh Hursey
    • 2
  • Tammy Toscos
    • 2
  1. 1.The Open University, Computing Department, Milton Keynes, MK7 6AAUK
  2. 2.Indiana University, School of Informatics, Bloomington, IN 47405USA
  3. 3.Indiana University~Purdue University, Indianapolis, Center for Earth & Environmental Science Indiana 46202USA

Personalised recommendations