Comparing Concurrent and Retrospective Verbal Protocols for Blind and Sighted Users

  • Andreas SavvaEmail author
  • Helen Petrie
  • Christopher Power
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9296)


Verbal protocols are widely used in user studies for evaluating websites. This study investigated the effectiveness and efficiency of concurrent and retrospective verbal protocols (CVP and RVP) for both blind and sighted participants, as well as participant workload and attitudes towards these methods. Eight blind and eight sighted participants undertook both protocols in a website evaluation. RVP was more effective as measured by problems encountered for both groups, although it was no more efficient than CVP. The severity of problems identified by both protocols was equivalent. As measured on the NASA TLX, participants found RVP found more demanding than CVP. Sighted participants found rating problems during CVP more disruptive than blind participants. These results show that RVP is a more useful protocol for practitioners and researchers even though it takes more time and is more demanding for participants. It is equally applicable for both blind and sighted participants.


User evaluation Think aloud protocol Concurrent verbal protocol Retrospective verbal protocol Web accessibility Web usability Blind users 



We thank the National Council for the Blind of Ireland for their assistance in running this study, and all the participants for their time. Andreas Savva thanks the Engineering and Physical Science Research Council of the UK and the Cyprus State Scholarship Foundation for his PhD funding.

Research Data Access. Researchers wishing to access the data used in this study should visit the following URL for more information:


  1. 1.
    Boren, T., Ramey, J.: Thinking aloud: reconciling theory and practice. IEEE Trans. Prof. Commun. 43, 261–278 (2000)CrossRefGoogle Scholar
  2. 2.
    Borsci, S., Federici, S.: The partial concurrent thinking aloud: a new usability evaluation technique for blind users. Assistive technology from adapted equipment to inclusive environments—AAATE, vol. 25, pp. 421–425 (2009)Google Scholar
  3. 3.
    Bowers, V.A., Snyder, H.L.: Concurrent versus retrospective verbal protocol verbal for comparing window usability. In: Human Factors and Ergonomics Society Annual Meeting, pp. 1270–1274 (1990)Google Scholar
  4. 4.
    Chandrashekar, S., Stockman, T., Fels, D., Benedyk, R.: Using think aloud protocol with blind users: a case for inclusive usability evaluation methods. In: 8th international ACM SIGACCESS Conference on Computers and Accessibility, pp. 251–252. ACM, New York (2006)Google Scholar
  5. 5.
    Disability Rights Commission.: The Web: Access and inclusion for disabled people. The Stationery Office, London (2004)Google Scholar
  6. 6.
    Dumas, J.F., Redish, J.C.: A practical Guide to Usability Testing. Greenwood Publishing Group Inc., Westpost, CT, USA (1993)Google Scholar
  7. 7.
    Ericsson, K.A., Simon, H.A.: Protocol Analysis. MIT-press, Cambridge (1984)Google Scholar
  8. 8.
    Harrison, C., Petrie, H.: Severity of usability and accessibility problems in eCommerce and eGovernment websites. In: BryanKinns, N., Blandfor, A., Curzon, P., Nigay, L. (Eds.), People and Computers XX - Engage. pp. 255–262, Godalming: Springer-Verlag London Ltd (2007)Google Scholar
  9. 9.
    Krahmer, E., Ummelen, N.: Thinking about thinking aloud: a comparison of two verbal protocols for usability testing. IEEE Trans. Prof. Commun. 47, 105–117 (2004)CrossRefzbMATHGoogle Scholar
  10. 10.
    Kuusela, H., Paul, P.: A comparison of concurrent and retrospective verbal protocol analysis. Am. J. Psychol. 113(3), 387–404 (2000)CrossRefzbMATHGoogle Scholar
  11. 11.
    Lavery, D., Cockton, G., Atkinson, M.P.: Comparison of evaluation methods using structured usability problem reports. Behav. Inf. Technol. 16, 246–266 (1997)CrossRefzbMATHGoogle Scholar
  12. 12.
    Lazar, J., Olalere, A., Wentz, B.: Investigating the accessibility and usability of job application web sites for blind users. J. Usability Stud. 7, 68–87 (2012)Google Scholar
  13. 13.
    Lewis: Using the Thinking Aloud Method in Cognitive Interface Design, Technical report. IBM Research Center (1982)Google Scholar
  14. 14.
  15. 15.
    Nielsen, J.: Usability Engineering. Elsevier (1994)Google Scholar
  16. 16.
    Nørgaard, M., Hornbæk, K.: What do usability evaluators do in practice?: an explorative study of think-aloud testing. In: 6th Design Interactive systems conference, pp. 209–218, ACM, New York (2006)Google Scholar
  17. 17.
    Ohnemus, K.R., Biers, D.W.: Retrospective versus concurrent thinking-out-loud in usability testing. In: Human Factors and Ergonomics Society Annual Meeting, pp. 1127–1131 (1993)Google Scholar
  18. 18.
    Olmsted-Hawala, E.L., Murphy, E.D., Hawala, S., Ashenfelter, K.T.: Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability. In: SIGCHI Conference on Human Factors in Computing Systems, pp. 2381–2390. ACM, New York (2010)Google Scholar
  19. 19.
    Petrie, H., Kheir, O.: The relationship between accessibility and usability of websites. In: SIGHI conference on Human Factors in Computing Systems, pp. 397–406. ACM, New York (2007)Google Scholar
  20. 20.
    Power, C., Freire, A., Petrie, H., Swallow, D.: Guidelines are only half the story: accessibility problems encountered by blind users on the web. In: SIGHI Conference on Human Factors in Computing Systems, pp. 433–442, ACM, New York (2012)Google Scholar
  21. 21.
    Rogers, Y., Sharp, H., Preece, J.: Interaction Design: Beyond Human-Computer Interaction. John Wiley & Sons, Hoboken (2011)Google Scholar
  22. 22.
    Rømen, D., Svanæs, D.: Validating WCAG versions 1.0 and 2.0 through usability testing with disabled users. Univ. Access Inf. Soc. 11, 375–385 (2012)CrossRefGoogle Scholar
  23. 23.
    Shi, Q: A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests. In: 5th Nordic conference on Human-Computer Interaction: Building Bridges, pp. 344–352. ACM, New York (2008)Google Scholar
  24. 24.
    Strain, P., Shaikh, A.D., Boardman, R.: Thinking but not seeing: think-aloud for non-sighted users. In: CHI 2007 Extended Abstracts on Human Factors in Computing Systems, pp. 1851–1856. ACM, New York (2007)Google Scholar
  25. 25.
    van den Haak, M.J., De Jong, M.D., Schellens, P.J.: Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison. Gov. Inf. Q. 16, 1153–1170 (2004)Google Scholar
  26. 26.
    van den Haak, M.J., De Jong, M.D., Schellens, P.J.: Evaluating municipal websites: A methodological comparison of three think-aloud variants. Gov. Inf. Q. 26, 193–202 (2009)CrossRefGoogle Scholar
  27. 27.
    van den Haak, M.J., De Jong, M.D., Schellens, P.J.: Evaluation of an informational web site: three variants of the think-aloud method compared. Tech. Commun. 54, 58–71 (2007)zbMATHGoogle Scholar
  28. 28.
    van den Haak, M.J., De Jong, M.D., Schellens, P.J.: Retrospective vs concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav. Inf. Technol. 22, 339–351 (2003)CrossRefGoogle Scholar
  29. 29.
    van den Haak, M.J., De Jong, M.D.: Exploring two methods of usability testing: concurrent versus retrospective think-aloud protocols. In: IEEE International Professional Communication Conference. pp. 285–287 (2003)Google Scholar
  30. 30.
    Wright, R.B., Converse, S.A.: Method bias and concurrent verbal protocol in software usability testing. In: Human Factors and Ergonomics Society Annual Meeting, pp. 1220–1224 (1992)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • Andreas Savva
    • 1
    Email author
  • Helen Petrie
    • 1
  • Christopher Power
    • 1
  1. 1.Human Computer Interaction Research Group, Department of Computer ScienceUniversity of YorkYorkUK

Personalised recommendations