Advertisement

Beyond the five-user assumption: Benefits of increased sample sizes in usability testing

  • Laura Faulkner
Article

Abstract

It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more. Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%. With 10 users, the lowest percentage of problems revealed by any one set was increased to 80%, and with 20 users, to 95%.

Keywords

Usability Test Usability Problem Tick Mark User Deviation Usability Professional 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. Grosvenor, L. (1999).Software usability: Challenging the myths and assumptions in an emerging field. Unpublished master’s thesis, University of Texas, Austin.Google Scholar
  2. Hudson, W. (2001). How many users does it take to change a website?SIGCHI Bulletin May/June 2001. Retrieved April 15, 2003 from http://www.acm.org/sigchi/bulletin/2001.3/mayjun01.pdf.Google Scholar
  3. Landauer, T. K., & Nielsen, J. (1993). A mathematical model of the finding of usability problems.Interchi ’93, ACM Computer-Human Interface Special Interest Group.Google Scholar
  4. Nielsen, J. (1993).Usability engineering. Boston: AP Professional.Google Scholar
  5. Nielsen, J. (2000, March).Why you only need to test with 5 users: Alertbox. Retrieved April 15, 2003 from http://www.useit.com/ alertbox/20000319.html.Google Scholar
  6. Perfetti, C., & Landesman, L. (2002).Eight is not enough. Retrieved April 14, 2003 from http://world.std.com/~uieweb/Articles/eight_is_ not enough.htm.Google Scholar
  7. Reason, J. (1997).Managing the risks of organizational accidents. Brookfield, VT: Ashgate.Google Scholar
  8. Spool, J., &Schroeder, W. (2001). Testing web sites: Five users is nowhere near enough. InCHI 2001 Extended Abstracts (pp. 285–286). New York: ACM Press.CrossRefGoogle Scholar
  9. Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough?Human Factors,34, 457–468.Google Scholar
  10. Wentworth, R. J. (1996, March).Group chairman’s factual report (American Airlines Flight 965). Washington, DC: National Transportation Safety Board, Office of Aviation Safety.Google Scholar
  11. Woolrych, A., &Cockton, G. (2001). Why and when five test users aren’t enough. In J. Vanderdonckt, A. Blandford, & A. Derycke (Eds.),Proceedings of IHM-HCI 2001 Conference: Vol. 2 (pp. 105–108). Toulouse, France: Cépadèus.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2003

Authors and Affiliations

  1. 1.Applied Research LaboratoriesUniversity of Texas at AustinAustin

Personalised recommendations