Universal Access in the Information Society

, Volume 11, Issue 4, pp 375–385 | Cite as

Validating WCAG versions 1.0 and 2.0 through usability testing with disabled users

  • Dagfinn RømenEmail author
  • Dag Svanæs
Long Paper


The Web Content Accessibility Guidelines (WCAG) by the Web Accessibility Initiative (WAI) has become the de-facto standard for accessibility on the Web. WCAG version 1.0 has become significant both as a practical tool and as an academic set of principles and is presently the basis of Web accessibility evaluations and guidelines in many countries. WCAG version 2.0 was released in 2008. This paper reports on a study that empirically validated the usefulness of using WCAG as a heuristic for website accessibility. Through controlled usability tests of two websites with disabled users (N = 7) and a control group (N = 6), it was found that only 27% of the identified website accessibility problems could have been identified through the use of WCAG 1.0. A similar analysis of conformance to WCAG 2.0 showed a marginal 5% improvement concerning identified website accessibility problems. Compensating for the low number of test subjects with confidence tests gave results that were still low (42% for WCAG 1.0 and 49% for WCAG 2.0, with 95% confidence). It is concluded from this that the application of WAI accessibility guidelines is not sufficient to guarantee website accessibility. It is recommended that future versions of the accessibility guidelines should be based on empirical data and validated empirically and that WAI expand their definition of accessibility to include “usability for all” in accordance with ISO 9241-171:2008.


Web accessibility Accessibility guidelines WCAG Web usability Usability testing Validation 



We would like to thank the test participants and the organizations representing the disabled users. Also thanks to Terje Røsand at NTNU/NSEP for highly valuable technical assistance.


  1. 1.
    Caldwell, B., Cooper, M., Reid, L., Vanderheiden, G.: Web content accessibility guidelines 2.0. (2008). Accessed 29 July 2010
  2. 2.
    Center for Universal Design, University of North Carolina: Universal Design Principles. (2008). Accessed 25 July 2010
  3. 3.
    Chisholm, W., Vanderheiden, G., Jacobs, I.: Web content accessibility guidelines 1.0. (1999). Accessed 25 July 2010
  4. 4.
    Disability Rights Commission: The Web—Access and Inclusion for Disabled People. TSO, London (2004)Google Scholar
  5. 5.
    Dumas, J.: User-based evaluations. In: Jacko, J., Sears, A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 1093–1117. L. Erlbaum Associates Inc., Hillsdale (2000)Google Scholar
  6. 6.
    Gulliksen, J., Harker, S.: The software accessibility of human-computer interfaces—ISO Technical Specification 16071, Universal Access in the Information Society, 2004, vol. 3, No 1, pp. 6–16 (2004)Google Scholar
  7. 7.
    Harrison, C., Petrie, H.: Impact of usability and accessibility problems in e-commerce and e-government websites. In: Proceedings of HCI 2006, vol. 1. British Computer Society, London (2006)Google Scholar
  8. 8.
    Hyun, J., Choi, D., Kim, S.: An active step toward a web content accessible society. In: Proceedings of the 2005 international cross-disciplinary workshop on web accessibility (W4A). ACM, New York (2005)Google Scholar
  9. 9.
    ISO: ISO/IEC 25062:2006. Software Engineering-Software product quality requirements and evaluation (SQuaRE)—common industry format (CIF) for usability test reports (2006)Google Scholar
  10. 10.
    ISO: ISO/IEC 9241-171:2008. Ergonomics of human--system interaction—Part 171: guidance on software accessibility (2008)Google Scholar
  11. 11.
    IT-og telestyrelsen: Bedst paa Nettet (Best on the Web. In Danish). (2008). Accessed 16 July 2008
  12. 12.
    Lopes, R., Carrico, L.: The impact of accessibility assessment in macro scale universal usability studies of the web. In: Proceedings of the 2008 international cross-disciplinary workshop on Web accessibility (W4A), pp. 5–14 (2008)Google Scholar
  13. 13.
    Molich, R.: Usable web design (in Danish), Ingenioren boger, Copenhagen, Denmark (2000)Google Scholar
  14. 14. Kvalitetsvurdering av offentlige nettsteder (Quality assessment of public websites). (2008). Accessed 16 July 2008
  15. 15.
    Petrie, H., Kheir, O.: The relationship between accessibility and usability of websites. In: Proceedings of the SIGCHI conference on human factors in computing systems CHI07, pp. 397–406 (2007)Google Scholar
  16. 16.
    Petrie, H., King, N., Hamilton, F.: Accessibility of museum, library and archive websites: the MLA audit. (2005). Accessed 16 July 2008
  17. 17.
    Reid, L., Snow-Weaver, A.: WCAG 2.0: a web accessibility standard for the evolving web. ACM, New York (2008)Google Scholar
  18. 18.
    Rømen, D., Svanæs, D.: Evaluating web site accessibility: validating the WAI guidelines through usability testing with disabled users. In: Proceedings of the 5th Nordic conference on human-computer interaction: building bridges. ACM Press, Lund, Sweden (2008)Google Scholar
  19. 19.
    Velleman, E., Strobbe, C., Koch, J., Velasco, C., Snaprud, M.: A unified web evaluation methodology using WCAG. Lect. Notes Comput. Sci. 4556, 177 (2007)CrossRefGoogle Scholar
  20. 20.
    Web Guidelines: (2007). Accessed 29 July 2010
  21. 21.
    WAI website: Accessed 29 July 2010
  22. 22.
    Watanabe, T., Umegaki, M.: Capability survey of user agents with the UAAG 1.0 test suite and its impact on web accessibility. Univ. Access Inf. Soc. 6(3), 221–232 (2007)Google Scholar
  23. 23.
    WHO: WHO definition of blindness. (2010). Accessed 28 Oct 2010
  24. 24.
    W3C: Comparison of WCAG 1.0 Checkpoints to WCAG 2.0, in numerical order. (2008). Accessed 29 July 2010
  25. 25.
    W3C ATAG: Accessed 29 July 2010
  26. 26.
    W3C UAAG: Accessed 29 July 2010
  27. 27.
    W3C WAI-AIRA: Accessed 29 July 2010

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  1. 1.Department of Computer and Information ScienceNorwegian University of Science and TechnologyTrondheimNorway
  2. 2.Agency for Public Management and eGovernment (Difi)LeikangerNorway

Personalised recommendations