Skip to main content

The Bootstrap Discovery Behaviour (BDB): a new outlook on usability evaluation

Cognitive Processing volume 12, pages 23–31 (2011)Cite this article

Abstract

The value of λ is one of the main issues debated in international usability studies. The debate is centred on the deficiencies of the mathematical return on investment model (ROI model) of Nielsen and Landauer (1993). The ROI model is discussed in order to identify the base of another model that, respecting Nielsen and Landauer’s one, tries to consider a large number of variables for the estimation of the number of evaluators needed for an interface. Using the bootstrap model (Efron 1979), we can take into account: (a) the interface properties, as the properties at zero condition of evaluation and (b) the probability that the population discovery behaviour is represented by all the possible discovery behaviours of a sample. Our alternative model, named Bootstrap Discovery Behaviour (BDB), provides an alternative estimation of the number of experts and users needed for a usability evaluation. Two experimental groups of users and experts are involved in the evaluation of a website (http://www.serviziocivile.it). Applying the BDB model to the problems identified by the two groups, we found that 13 experts and 20 users are needed to identify 80% of usability problems, instead of 6 experts and 7 users required according to the estimation of the discovery likelihood provided by the ROI model. The consequence of the difference between the results of those models is that in following the BDB the costs of usability evaluation increase, although this is justified considering that the results obtained have the best probability of representing the entire population of experts and users.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. Actually, only Nielsen et al. (1993) used λ, instead of p (Lewis 1994; 2006; Virzi 1992; Wright and Monk 1991; Schmettow 2008) in the formula 1, partly because they derived their formula from the “Poisson process” (see Nielsen and Landauer 1993). Many authors (Lewis 1994; 2006; Virzi 1992; Wright and Monk 1991; Schmettow 2008) use the formula (1) written as: P = 1 – (1 – p) n, where “P” is the total number of problems in the interface, “p” the probability of finding the average usability problem when running a single average subject test and “n” is the number of participants.

  2. In the review phase of this work, a reviewer claimed that “The authors should do a Monte Carlo resampling exercise to assess the extent to which randomly selected sets of 6 experts (for the CW data) and 7 users (for the TA data) find or fail to find at least 80% of the problems discovered by the full samples,” since, according to the reviewer’s opinion, “The authors simply state the different sample size estimates and appear to assume that the BDB are correct without further, evaluation or any tests of significance”. In accordance with the reviewer’s suggestions, we have added this section.

References

  • Bellman JR, Park CW (1980) Effects of prior knowledge and experience and phase of the choice process on consumer decision processes: a protocol analysis. J Consum Res 7(3):234–248

    Article  Google Scholar 

  • Bettman JR (1979) An information processing theory of consumer choice. Addison-Wesley, Cambridge

    Google Scholar 

  • Biehal G, Chakravarti D (1982a) Experiences with the Bettman-park verbal-protocol coding scheme. J Consum Res 8(4):442–448

    Article  Google Scholar 

  • Biehal G, Chakravarti D (1982b) Information-presentation format and learning goals as determinants of consumers’ memory retrieval and choice processes. J Consum Res 8(4):431–441

    Article  Google Scholar 

  • Biehal G, Chakravarti D (1986) Consumers’ use of memory and external information in choice: Macro and micro perspectives. J Consum Res 12(4):382–405

    Article  Google Scholar 

  • Biehal G, Chakravarti D (1989) The effects of concurrent verbalization on choice processing. J Mark Res 26(1):84–96

    Article  Google Scholar 

  • Borsci S, Federici S (2009) The partial concurrent thinking aloud: a new usability evaluation technique for blind users. In: Emiliani PL, Burzagli L, Como A, Gabbanini F, Salminen A-L (eds) Assistive technology from adapted equipment to inclusive environments—AAATE 2009, vol 25. Assistive technology research series. IOS Press, Florence, pp 421–425. doi:10.3233/978-1-60750-042-1-421

  • Caulton D (2001) Relaxing the homogeneity assumption in usability testing. Behav Inf Technol 20(1):1–7. doi:10.1080/01449290010020648

    Google Scholar 

  • Efron B (1979) Bootstrap methods: another look at the jackknife. Ann Statist 7(1):1–26. doi:10.1214/aos/1176344552

    Article  Google Scholar 

  • Federici S, Borsci S (2010) Usability evaluation: models, methods, and applications. International encyclopedia of rehabilitation. Center for International rehabilitation research information and exchange (CIRRIE), Buffalo. http://cirrie.buffalo.edu/encyclopedia/article.php?id=277&language=en. Accessed 20 Sept 2010

  • Federici S, Borsci S, Mele ML (2010a) Usability evaluation with screen reader users: a video presentation of the pcta’s experimental setting and rules. Cogn Process 11(3):285–288. doi:10.1007/s10339-010-0365-9

    Article  PubMed  Google Scholar 

  • Federici S, Borsci S, Stamerra G (2010b) Web usability evaluation with screen reader users: implementation of the partial concurrent thinking aloud technique. Cogn Process 11(3):263–272. doi:10.1007/s10339-009-0347-y

    Article  PubMed  Google Scholar 

  • Fox J (2002) An r and s-plus companion to applied regression. SAGE, California

    Google Scholar 

  • Goodstein RL (1963) Boolean algebra. Pergamon Press, Oxford

    Google Scholar 

  • Green A (1995) Verbal protocol analysis. Psychologist 8(3):126–129

    Google Scholar 

  • Hertzum M, Jacobsen NE (2003) The evaluator effect: a chilling fact about usability evaluation methods. Int J Hum Comput Interact 15(4):183–204. doi:10.1207/S15327590IJHC1501_14

    Article  Google Scholar 

  • Kuusela H, Pallab P (2000) A comparison of concurrent and retrospective verbal protocol analysis. Am J Psychol 113(3):387–404

    CAS  Article  PubMed  Google Scholar 

  • Kuusela H, Spence MT, Kanto AJ (1998) Expertise effects on prechoice decision processes and final outcomes: A protocol analysis. Eur J Mark 32(5/6):559

    Article  Google Scholar 

  • Lewis JR (1994) Sample sizes for usability studies: additional considerations. Hum Factors 36(2):368–378

    CAS  PubMed  Google Scholar 

  • Lewis JR (2001) Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. Int J Hum Comput Interact 13(4):445–479

    Article  Google Scholar 

  • Lewis JR (2006) Sample sizes for usability tests: mostly math, not magic. Interactions 13(6):29–33. doi:10.1145/1167948.1167973

    Article  Google Scholar 

  • Lewis C, Rieman J (1993) Task-centered user interface design: a practical introduction. http://users.cs.dal.ca/~jamie/TCUID/tcuid.pdf. Accessed 20 Jun 2010

  • Nielsen J (2000) Why you only need to test with 5 users. www.useit.com/alertbox/20000319.html. Accessed 20 Jun 2010

  • Nielsen J, Landauer TK A mathematical model of the finding of usability problems. In: Proceedings of the INTERACT ‘93 and CHI ‘93 Conference on human factors in computing systems, Amsterdam, 24–29 Apr 1993. ACM, New York, NY, USA, pp 206–213

  • Nielsen J, Mack RL (eds) (1994) Usability inspection methods.Wiley, New York

  • Polson PG, Lewis C, Rieman J, Wharton C (1992) Cognitive walkthroughs: a method for theory-based evaluation of user interfaces. Int J Man Mach Stud 36(5):741–773. doi:10.1016/0020-7373(92)90039-N

    Article  Google Scholar 

  • Rieman J, Franzke M, Redmiles D Usability evaluation with the cognitive walkthrough. In: Conference companion on human factors in computing systems, Denver, Colorado, United States, 1995. ACM, 223735, pp 387–388. doi:10.1145/223355.223735

  • Schmettow M Heterogeneity in the usability evaluation process. In: Proceedings of the 22nd British HCI group annual conference on people and computers: culture, creativity, interaction—Volume 1, Liverpool, United Kingdom, 2008. British Computer Society, 1531527, pp 89–98

  • Spool J, Schroeder W Testing web sites: Five users is nowhere near enough. In: CHI ‘01 extended abstracts on human factors in computing systems, Seattle, Washington, 2001. ACM, 634236, pp 285–286. doi:10.1145/634067.634236

  • Turner CW, Lewis JR, Nielsen J (2006) Determining usability test sample size, vol 2. International encyclopedia of ergonomics and human factors, Second edn. CRC Press, Boca Raton

    Google Scholar 

  • Virzi RA (1990) Streamlining the design process: running fewer subjects. Human factors and ergonomics society annual meeting proceedings 34:291–294

  • Virzi RA (1992) Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors 34(4):457–468

    Google Scholar 

  • Wharton C, Rieman J, Lewis C, Polson PG (1994) The cognitive walkthrough method: a practitioner’s guide. In: Nielsen J, Mack RL (eds) Usability inspection methods. Wiley, New York, pp 105–140

    Google Scholar 

  • Woolrych A, Cockton G Why and when five test users aren’t enough. In: Vanderdonckt J, Blandford A, Derycke A (eds) Proceedings of IHM-HCI 2001 conference, Toulouse, FR, 10–14 Sept 2001. Cépadčus Éditions, pp 105–108

  • Wright PC, Monk AF (1991) A cost-effective evaluation method for use by designers. Int J Man Mach Stud 35(6):891–912. doi:10.1016/s0020-7373(05)80167-1

    Article  Google Scholar 

Download references

Author information

Affiliations

  1. ECoNA—Interuniversity Centre for Research on Cognitive Processing in Natural and Artificial Systems, Sapienza University of Rome, Rome, Italy

    Simone Borsci, Alessandro Londei & Stefano Federici

  2. Department of Human and Education Sciences, University of Perugia, Perugia, Italy

    Stefano Federici

Authors
  1. Simone Borsci
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessandro Londei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefano Federici
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simone Borsci.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (TXT 1 kb)

Appendix: User scenarios

Appendix: User scenarios

  1. 1.

    A friend of yours is enrolled on a 1-year activity in social service. You are interested in finding more information about social service activities and acquiring information in order to apply for a one-year job. Go to the website http://www.serviziocivile.it/, find that information and download the documents for the job application.

  2. 2.

    A friend of yours, who lives in Rome, has some internet connection problems, so he or she telephones you for assistance. In fact, he or she is interested in social service work, but he or she does not know where the office is and when it is open in order to present his curriculum vitae. Go to the website http://www.serviziocivile.it/ in order to find that information for him or her.

  3. 3.

    You are interested in social service activities, so you go to the website http://www.serviziocivile.it/ in order to see whether this website offers a newsletter service, even though you are not enrolled on the social service activities. If the newsletter service requires you log in, sign up to the newsletter.

  4. 4.

    A friend of yours is working on a 1-year social service project in the Republic of the Philippines. You are interested in applying for a job on this project. Go to the website http://www.serviziocivile.it/ in order to find information about the project and whether is possible to obtain a job.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Borsci, S., Londei, A. & Federici, S. The Bootstrap Discovery Behaviour (BDB): a new outlook on usability evaluation. Cogn Process 12, 23–31 (2011). https://doi.org/10.1007/s10339-010-0376-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10339-010-0376-6

Keywords

  • Asymptotic test
  • Bootstrap
  • Effectiveness
  • Return of investment
  • User experience evaluation