Human-Computer Interaction

INTERACT 2015: Human-Computer Interaction – INTERACT 2015 pp 55-71 | Cite as

Comparing Concurrent and Retrospective Verbal Protocols for Blind and Sighted Users

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9296)

Abstract

Verbal protocols are widely used in user studies for evaluating websites. This study investigated the effectiveness and efficiency of concurrent and retrospective verbal protocols (CVP and RVP) for both blind and sighted participants, as well as participant workload and attitudes towards these methods. Eight blind and eight sighted participants undertook both protocols in a website evaluation. RVP was more effective as measured by problems encountered for both groups, although it was no more efficient than CVP. The severity of problems identified by both protocols was equivalent. As measured on the NASA TLX, participants found RVP found more demanding than CVP. Sighted participants found rating problems during CVP more disruptive than blind participants. These results show that RVP is a more useful protocol for practitioners and researchers even though it takes more time and is more demanding for participants. It is equally applicable for both blind and sighted participants.

Keywords

User evaluation Think aloud protocol Concurrent verbal protocol Retrospective verbal protocol Web accessibility Web usability Blind users 

References

  1. 1.
    Boren, T., Ramey, J.: Thinking aloud: reconciling theory and practice. IEEE Trans. Prof. Commun. 43, 261–278 (2000)CrossRefGoogle Scholar
  2. 2.
    Borsci, S., Federici, S.: The partial concurrent thinking aloud: a new usability evaluation technique for blind users. Assistive technology from adapted equipment to inclusive environments—AAATE, vol. 25, pp. 421–425 (2009)Google Scholar
  3. 3.
    Bowers, V.A., Snyder, H.L.: Concurrent versus retrospective verbal protocol verbal for comparing window usability. In: Human Factors and Ergonomics Society Annual Meeting, pp. 1270–1274 (1990)Google Scholar
  4. 4.
    Chandrashekar, S., Stockman, T., Fels, D., Benedyk, R.: Using think aloud protocol with blind users: a case for inclusive usability evaluation methods. In: 8th international ACM SIGACCESS Conference on Computers and Accessibility, pp. 251–252. ACM, New York (2006)Google Scholar
  5. 5.
    Disability Rights Commission.: The Web: Access and inclusion for disabled people. The Stationery Office, London (2004)Google Scholar
  6. 6.
    Dumas, J.F., Redish, J.C.: A practical Guide to Usability Testing. Greenwood Publishing Group Inc., Westpost, CT, USA (1993)Google Scholar
  7. 7.
    Ericsson, K.A., Simon, H.A.: Protocol Analysis. MIT-press, Cambridge (1984)Google Scholar
  8. 8.
    Harrison, C., Petrie, H.: Severity of usability and accessibility problems in eCommerce and eGovernment websites. In: BryanKinns, N., Blandfor, A., Curzon, P., Nigay, L. (Eds.), People and Computers XX - Engage. pp. 255–262, Godalming: Springer-Verlag London Ltd (2007)Google Scholar
  9. 9.
    Krahmer, E., Ummelen, N.: Thinking about thinking aloud: a comparison of two verbal protocols for usability testing. IEEE Trans. Prof. Commun. 47, 105–117 (2004)CrossRefMATHGoogle Scholar
  10. 10.
    Kuusela, H., Paul, P.: A comparison of concurrent and retrospective verbal protocol analysis. Am. J. Psychol. 113(3), 387–404 (2000)CrossRefMATHGoogle Scholar
  11. 11.
    Lavery, D., Cockton, G., Atkinson, M.P.: Comparison of evaluation methods using structured usability problem reports. Behav. Inf. Technol. 16, 246–266 (1997)CrossRefMATHGoogle Scholar
  12. 12.
    Lazar, J., Olalere, A., Wentz, B.: Investigating the accessibility and usability of job application web sites for blind users. J. Usability Stud. 7, 68–87 (2012)Google Scholar
  13. 13.
    Lewis: Using the Thinking Aloud Method in Cognitive Interface Design, Technical report. IBM Research Center (1982)Google Scholar
  14. 14.
  15. 15.
    Nielsen, J.: Usability Engineering. Elsevier (1994)Google Scholar
  16. 16.
    Nørgaard, M., Hornbæk, K.: What do usability evaluators do in practice?: an explorative study of think-aloud testing. In: 6th Design Interactive systems conference, pp. 209–218, ACM, New York (2006)Google Scholar
  17. 17.
    Ohnemus, K.R., Biers, D.W.: Retrospective versus concurrent thinking-out-loud in usability testing. In: Human Factors and Ergonomics Society Annual Meeting, pp. 1127–1131 (1993)Google Scholar
  18. 18.
    Olmsted-Hawala, E.L., Murphy, E.D., Hawala, S., Ashenfelter, K.T.: Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability. In: SIGCHI Conference on Human Factors in Computing Systems, pp. 2381–2390. ACM, New York (2010)Google Scholar
  19. 19.
    Petrie, H., Kheir, O.: The relationship between accessibility and usability of websites. In: SIGHI conference on Human Factors in Computing Systems, pp. 397–406. ACM, New York (2007)Google Scholar
  20. 20.
    Power, C., Freire, A., Petrie, H., Swallow, D.: Guidelines are only half the story: accessibility problems encountered by blind users on the web. In: SIGHI Conference on Human Factors in Computing Systems, pp. 433–442, ACM, New York (2012)Google Scholar
  21. 21.
    Rogers, Y., Sharp, H., Preece, J.: Interaction Design: Beyond Human-Computer Interaction. John Wiley & Sons, Hoboken (2011)Google Scholar
  22. 22.
    Rømen, D., Svanæs, D.: Validating WCAG versions 1.0 and 2.0 through usability testing with disabled users. Univ. Access Inf. Soc. 11, 375–385 (2012)CrossRefGoogle Scholar
  23. 23.
    Shi, Q: A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests. In: 5th Nordic conference on Human-Computer Interaction: Building Bridges, pp. 344–352. ACM, New York (2008)Google Scholar
  24. 24.
    Strain, P., Shaikh, A.D., Boardman, R.: Thinking but not seeing: think-aloud for non-sighted users. In: CHI 2007 Extended Abstracts on Human Factors in Computing Systems, pp. 1851–1856. ACM, New York (2007)Google Scholar
  25. 25.
    van den Haak, M.J., De Jong, M.D., Schellens, P.J.: Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison. Gov. Inf. Q. 16, 1153–1170 (2004)Google Scholar
  26. 26.
    van den Haak, M.J., De Jong, M.D., Schellens, P.J.: Evaluating municipal websites: A methodological comparison of three think-aloud variants. Gov. Inf. Q. 26, 193–202 (2009)CrossRefGoogle Scholar
  27. 27.
    van den Haak, M.J., De Jong, M.D., Schellens, P.J.: Evaluation of an informational web site: three variants of the think-aloud method compared. Tech. Commun. 54, 58–71 (2007)MATHGoogle Scholar
  28. 28.
    van den Haak, M.J., De Jong, M.D., Schellens, P.J.: Retrospective vs concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav. Inf. Technol. 22, 339–351 (2003)CrossRefGoogle Scholar
  29. 29.
    van den Haak, M.J., De Jong, M.D.: Exploring two methods of usability testing: concurrent versus retrospective think-aloud protocols. In: IEEE International Professional Communication Conference. pp. 285–287 (2003)Google Scholar
  30. 30.
    Wright, R.B., Converse, S.A.: Method bias and concurrent verbal protocol in software usability testing. In: Human Factors and Ergonomics Society Annual Meeting, pp. 1220–1224 (1992)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • Andreas Savva
    • 1
  • Helen Petrie
    • 1
  • Christopher Power
    • 1
  1. 1.Human Computer Interaction Research Group, Department of Computer ScienceUniversity of YorkYorkUK

Personalised recommendations