Beyond the five-user assumption: Benefits of increased sample sizes in usability testing

Abstract

It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more. Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%. With 10 users, the lowest percentage of problems revealed by any one set was increased to 80%, and with 20 users, to 95%.

References

  1. Grosvenor, L. (1999).Software usability: Challenging the myths and assumptions in an emerging field. Unpublished master’s thesis, University of Texas, Austin.

    Google Scholar 

  2. Hudson, W. (2001). How many users does it take to change a website?SIGCHI Bulletin May/June 2001. Retrieved April 15, 2003 from http://www.acm.org/sigchi/bulletin/2001.3/mayjun01.pdf.

  3. Landauer, T. K., & Nielsen, J. (1993). A mathematical model of the finding of usability problems.Interchi ’93, ACM Computer-Human Interface Special Interest Group.

  4. Nielsen, J. (1993).Usability engineering. Boston: AP Professional.

    Google Scholar 

  5. Nielsen, J. (2000, March).Why you only need to test with 5 users: Alertbox. Retrieved April 15, 2003 from http://www.useit.com/ alertbox/20000319.html.

  6. Perfetti, C., & Landesman, L. (2002).Eight is not enough. Retrieved April 14, 2003 from http://world.std.com/~uieweb/Articles/eight_is_ not enough.htm.

  7. Reason, J. (1997).Managing the risks of organizational accidents. Brookfield, VT: Ashgate.

    Google Scholar 

  8. Spool, J., &Schroeder, W. (2001). Testing web sites: Five users is nowhere near enough. InCHI 2001 Extended Abstracts (pp. 285–286). New York: ACM Press.

    Google Scholar 

  9. Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough?Human Factors,34, 457–468.

    Google Scholar 

  10. Wentworth, R. J. (1996, March).Group chairman’s factual report (American Airlines Flight 965). Washington, DC: National Transportation Safety Board, Office of Aviation Safety.

    Google Scholar 

  11. Woolrych, A., &Cockton, G. (2001). Why and when five test users aren’t enough. In J. Vanderdonckt, A. Blandford, & A. Derycke (Eds.),Proceedings of IHM-HCI 2001 Conference: Vol. 2 (pp. 105–108). Toulouse, France: Cépadèus.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Laura Faulkner.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Faulkner, L. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers 35, 379–383 (2003). https://doi.org/10.3758/BF03195514

Download citation

Keywords

  • Usability Test
  • Usability Problem
  • Tick Mark
  • User Deviation
  • Usability Professional