User-Centered Design and Evaluation – The Big Picture

  • Victoria Bellotti
  • Shin’ichi Fukuzumi
  • Toshiyuki Asahi
  • Shunsuke Suzuki
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5610)


This paper provides a high-level overview of the field of usability evaluation as context for a panel “Systematization, Modeling and Quantitative Evaluation of Human Interface” in which several authors report on a collaborative effort to apply CogTool, an automated usability evaluation method, to mobile phone interfaces and to assess whether usability predictions made by CogTool correlate with user subjective impressions of usability. If the endeavor, which is still underway at the time of writing, is successful, then CogTool may be applied economically within the product development lifecycle to reduce the risk of usability problems.


Usability evaluation methods metrics systematization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Anderson, J.R.: Language, Memory and Thought. Erlbaum Associates, Hillsdale, NJ (1976)Google Scholar
  2. 2.
    Anderson, J.R.: The Architecture of Cognition. Harvard University Press, Cambridge, MA (1983)Google Scholar
  3. 3.
    Anderson, J.R.: Rules of the Mind. Lawrence Erlbaum Associates, Hillsdale (1993)Google Scholar
  4. 4.
    Anderson, J.R.: ACT: A Simple Theory of Complex Cognition. American Psychologist 51, 355–365 (1996)CrossRefGoogle Scholar
  5. 5.
    Apple: Apple Human Interface Guidelines for Mac OS X (2008),
  6. 6.
    Bak, J.O., Nguyen, K., Risgaard, P., Stage, J.: Obstacles to Usability Evaluation in Practice: A Survey of Software Development Organizations. In: Proceedings of the 5th Nordic Conference on Human-Computer interaction: Building Bridges, NordiCHI 2008, vol. 358, pp. 23–32. ACM, New York (2008)CrossRefGoogle Scholar
  7. 7.
    Bellotti, V.: Implications of Current Design Practice for the Use of HCI Techniques. In: Jones, D.M., Winder, R. (eds.) People and Computers IV, pp. 13–34. Cambridge University Press, Cambridge (1988)Google Scholar
  8. 8.
    Bellotti, V., Dalal, B., Good, N., Bobrow, D.G., Ducheneaut, N.: What a To-do: Studies of Task Management Towards the Design of a Personal Task List Manager. In: ACM Conference on Human Factors in Computing Systems, CHI 2004, pp. 735–742. ACM, New York (2004)CrossRefGoogle Scholar
  9. 9.
    Bellotti, V., Ducheneaut, N., Howard, M.A., Smith, I.E.: Taking Email to Task: The Design and Evaluation of a Task Management Centered Email Tool. In: CSCW 2002 Workshop: Redesigning Email for the 21st Century, New Orleans, LA, ACM, New York (2003)Google Scholar
  10. 10.
    Bellotti, V., Smith, I.: Informing the Design of an Information Management System with Iterative Fieldwork. In: Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques, ACM, New York (2000)Google Scholar
  11. 11.
    Benel, D.C.R., Ottens, D., Horst, R.: Use of an Eye Tracking System in the Usability Laboratory. In: Proceedings of the Human Factors Society 35th Annual Meeting, Santa Monica, Human Factors and Ergonomics Society, pp. 461–465 (1991)Google Scholar
  12. 12.
    Byrne, M.D.: ACT-R/PM and Menu Selection: Applying a Cognitive Architecture to HCI. International Journal of Human-Computer Studies 55, 41–84 (1999)CrossRefzbMATHGoogle Scholar
  13. 13.
    Card, S.K., Newell, A., Moran, T.P.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates Inc., Hillsdale (1983)Google Scholar
  14. 14.
    Constantine, L.L.: Beyond User-Centered Design and User Experience: Designing for User Performance. Cutter IT Journal 17(2), 16–25 (2004)Google Scholar
  15. 15.
    Dix, A., Finlay, J., Abowd, G., Beale, R.: Human Computer Interaction, 2nd edn. Prentice-Hall, Englewood Cliffs (1993)zbMATHGoogle Scholar
  16. 16.
    Earthy, J., Sherwood Jones, B., Bevan, N.: The Improvement of Human-centred Processes – Facing the Challenge and Reaping the Benefit of ISO 13407. International Journal of Human Computer Studies 55(4), 553–585 (2001)CrossRefzbMATHGoogle Scholar
  17. 17.
    Fu, W.-T., Pirolli, P.: SNIF-ACT: A Cognitive Model of User Navigation on the World Wide Web. Human-Computer Interaction 22, 355–412 (2007)Google Scholar
  18. 18.
    Gray, W.D., Salzman, M.C.: Damaged Merchandise? A Review of Experiments that Compare Usability Evaluation Methods. Human Computer Interaction 13(3), 203–261 (1998)CrossRefGoogle Scholar
  19. 19.
    Grudin, J.: Systematic Sources of Suboptimal Interface Design in Large Product Development Organizations. Human-Computer Interaction 6(2), 147–196 (1991)CrossRefGoogle Scholar
  20. 20.
    Gunther, R., Janis, J., Butler, S.: The UCD Decision Matrix: How, When, and Where to Sell User-Centered Design into the Development Cycle (2001), (retrieved January 24, 2009)
  21. 21.
    Hartson, H.R., Andre, T.S., Williges, R.C.: Criteria for Evaluating Usability Evaluation Methods. International Journal of Human-Computer Interaction 15(1), 145–181 (2003)CrossRefGoogle Scholar
  22. 22.
    Hollingsed, T., Novick, D.G.: Usability Inspection Methods After 15 Years of Research and Practice. In: Proceedings of the 25th Annual ACM international Conference on Design of Communication, SIGDOC 2007, pp. 249–255. ACM, New York (2007)Google Scholar
  23. 23.
    ISO 9241-11:1998. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) – Part 11: Guidance on Usability. International Organization for Standardization (1998)Google Scholar
  24. 24.
    ISO 9241-110:2006 Ergonomics of Human-System Interaction – Part 110: Dialogue Principles (2006)Google Scholar
  25. 25.
    Ivory, M.Y., Hearst, M.A.: The State of the Art in Automating Usability Evaluation of User Interfaces. ACM Computing Surveys 33(4), 470–516 (2001)CrossRefGoogle Scholar
  26. 26.
    John, B.E., Prevas, K., Salvucci, D.D., Koedinger, K.: Predictive Human Performance Modeling Made Easy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2004, pp. 455–462. ACM, New York (2004)CrossRefGoogle Scholar
  27. 27.
    Kieras, D., Polson, P.: An Approach to the Formal Analysis of User Complexity. Int. Journ. of Man-Machine Studies 22, 365–394 (1985)CrossRefGoogle Scholar
  28. 28.
    Koyani, S.J., Bailey, R.W., Nall, J.R.: Research-Based Web Design & Usability Guidelines. U.S. Department of Health and Human Services (HHS) and the U.S. General Services Administration (GSA), Washington DC (2004), (retrieved February 21, 2009)
  29. 29.
    Laird, J., Rosenbloom, P., Newell, A.: Universal Subgoaling and Chunking: the Automatic Generation and Learning of Goal Hierarchies. Kluwer Academic Publishers, Dordrecht (1986)CrossRefGoogle Scholar
  30. 30.
    Lee, Y.S., Hong, S.W., Smith-Jackson, T.L., Nussbaum, M.A., Tomioka, K.: Systematic Evaluation Methodology for Cell Phone User Interfaces. Interacting with Computers 18(2), 304–325 (2006)CrossRefGoogle Scholar
  31. 31.
    Lewis, C., Polson, P.G., Wharton, C., Rieman, J.: Testing a Walkthrough Methodology for Theory-Based Design of Walk-up-and-use Interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 1990, pp. 235–242. ACM, New York (1990)Google Scholar
  32. 32.
    Macleod, M., Bowden, R., Bevan, N., Curson, I.: The MUSiC performance measurement method. Behaviour & Information Technology 16(4-5), 279–293 (1997)CrossRefGoogle Scholar
  33. 33.
    Muller, M.J., Geyer, W., Brownholtz, B., Wilcox, E., Millen, D.R.: One-Hundred Days in an Activity-Centric Collaboration Environment Based on Shared Objects. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 2004, pp. 375–382. ACM, New York (2004)CrossRefGoogle Scholar
  34. 34.
    Nielsen, J.: The Usability Engineering Life Cycle. Computer 25(3), 12–22 (1992)CrossRefGoogle Scholar
  35. 35.
    Nielsen, J.: Usability engineering. Academic Press, Boston (1993)zbMATHGoogle Scholar
  36. 36.
    Nielsen, J.: Using Discount Usability Engineering to Penetrate the Intimidation Barrier. In: Bias, R.G., Mayhew, D.J. (eds.) Cost-Justifying Usability, Academic Press, London (1994)Google Scholar
  37. 37.
    Nielsen, J.: Durability of Usability Guidelines. Jakob Nielsen’s Alertbox (January 17, 2005), (retrieved January 27, 2009)
  38. 38.
    Nielsen, J.: Cost of User Testing a Website. Alertbox (May 3, 1998), (retrieved January 24, 2009)
  39. 39.
    Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1990, pp. 249–256. ACM Press, New York (1990)Google Scholar
  40. 40.
    Polson, P.G., Lewis, C., Rieman, J., Wharton, C.: Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces. International Journal of Man-Machine Studies 36, 741–773 (1992)CrossRefGoogle Scholar
  41. 41.
    Poltrock, S.E., Grudin, J.: Organizational Obstacles to Interface Design and Development: Two Participant-Observer Studies. ACM Trans. Comput.-Hum. Interact 1(1), 52–80 (1994)CrossRefGoogle Scholar
  42. 42.
    Ryu, Y.S.: Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods, Doctoral dissertation, State University, Blacksburg, VA, USA (2005)Google Scholar
  43. 43.
    Sauro, J., Kindlund, E.: A Method to Standardize Usability Metrics into a Single Score. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 2005, pp. 401–409. ACM, New York (2005)Google Scholar
  44. 44.
    Scholtz, J.: Usability Evaluation. National Institute of Standards and Technology (2006), (retrieved January 24, 2009)
  45. 45.
    Stickel, C., Scerbakov, A., Kaufmann, T., Ebner, M.: Usability Metrics of Time and Stress - Biological Enhanced Performance Test of a University Wide Learning Management System. In: Holzinger, A. (ed.) Proceedings of the 4th Symposium of the Workgroup Human-Computer interaction and Usability Engineering of the Austrian Computer Society on HCI and Usability For Education and Work. Lecture Notes In Computer Science, vol. 5298, pp. 173–184. Springer, Heidelberg (2008)Google Scholar
  46. 46.
    Teo, L., John, B.E.: Towards Predicting User Interaction with CogTool-Explorer. In: Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting, pp. 950–954 (2008)Google Scholar
  47. 47.
    Teo, L., John, B.E., Pirolli, P.: Towards a Tool for Predicting User Exploration. In: CHI 2007 Extended Abstracts on Human Factors in Computing Systems, CHI 2007, pp. 2687–2692. ACM, New York (2007)Google Scholar
  48. 48.
    Vredenburg, K., Mao, J., Smith, P.W., Carey, T.: A Survey of User-Centered Design Practice. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 471–478. ACM, New York (2002)Google Scholar
  49. 49.
    Wixon, D.: Evaluating Usability Methods: Why The Current Literature Fails the Practitioner. Interactions 10(4), 28–34 (2003)CrossRefGoogle Scholar
  50. 50.
    Zaharias, P.A.: Usability Evaluation Method for E-Learning: Focus on Motivation to Learn. In: CHI 2006 Extended Abstracts on Human Factors in Computing Systems, pp. 1571–1576. ACM, New York (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Victoria Bellotti
    • 1
  • Shin’ichi Fukuzumi
    • 2
  • Toshiyuki Asahi
    • 2
  • Shunsuke Suzuki
    • 2
  1. 1.Palo Alto Research CenterPalo AltoUSA
  2. 2.NEC CorporationNaraJapan

Personalised recommendations