Skip to main content
SpringerLink
Log in

Validating WCAG versions 1.0 and 2.0 through usability testing with disabled users

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

The Web Content Accessibility Guidelines (WCAG) by the Web Accessibility Initiative (WAI) has become the de-facto standard for accessibility on the Web. WCAG version 1.0 has become significant both as a practical tool and as an academic set of principles and is presently the basis of Web accessibility evaluations and guidelines in many countries. WCAG version 2.0 was released in 2008. This paper reports on a study that empirically validated the usefulness of using WCAG as a heuristic for website accessibility. Through controlled usability tests of two websites with disabled users (N = 7) and a control group (N = 6), it was found that only 27% of the identified website accessibility problems could have been identified through the use of WCAG 1.0. A similar analysis of conformance to WCAG 2.0 showed a marginal 5% improvement concerning identified website accessibility problems. Compensating for the low number of test subjects with confidence tests gave results that were still low (42% for WCAG 1.0 and 49% for WCAG 2.0, with 95% confidence). It is concluded from this that the application of WAI accessibility guidelines is not sufficient to guarantee website accessibility. It is recommended that future versions of the accessibility guidelines should be based on empirical data and validated empirically and that WAI expand their definition of accessibility to include “usability for all” in accordance with ISO 9241-171:2008.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://www.trondheim.kommune.no/.

  2. http://www.malvik.kommune.no/.

  3. Due to the way the website accessibility problems were coded, the real SD might be a bit less. The resulting confidence interval is consequently conservative.

References

  1. Caldwell, B., Cooper, M., Reid, L., Vanderheiden, G.: Web content accessibility guidelines 2.0. http://www.w3.org/TR/WCAG20/ (2008). Accessed 29 July 2010

  2. Center for Universal Design, University of North Carolina: Universal Design Principles. http://www.design.ncsu.edu/cud/about_ud/about_ud.htm (2008). Accessed 25 July 2010

  3. Chisholm, W., Vanderheiden, G., Jacobs, I.: Web content accessibility guidelines 1.0. http://www.w3.org/TR/WCAG10/ (1999). Accessed 25 July 2010

  4. Disability Rights Commission: The Web—Access and Inclusion for Disabled People. TSO, London (2004)

  5. Dumas, J.: User-based evaluations. In: Jacko, J., Sears, A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 1093–1117. L. Erlbaum Associates Inc., Hillsdale (2000)

    Google Scholar 

  6. Gulliksen, J., Harker, S.: The software accessibility of human-computer interfaces—ISO Technical Specification 16071, Universal Access in the Information Society, 2004, vol. 3, No 1, pp. 6–16 (2004)

  7. Harrison, C., Petrie, H.: Impact of usability and accessibility problems in e-commerce and e-government websites. In: Proceedings of HCI 2006, vol. 1. British Computer Society, London (2006)

  8. Hyun, J., Choi, D., Kim, S.: An active step toward a web content accessible society. In: Proceedings of the 2005 international cross-disciplinary workshop on web accessibility (W4A). ACM, New York (2005)

  9. ISO: ISO/IEC 25062:2006. Software Engineering-Software product quality requirements and evaluation (SQuaRE)—common industry format (CIF) for usability test reports (2006)

  10. ISO: ISO/IEC 9241-171:2008. Ergonomics of human--system interaction—Part 171: guidance on software accessibility (2008)

  11. IT-og telestyrelsen: Bedst paa Nettet (Best on the Web. In Danish). http://bedstpaanettet.dk/ (2008). Accessed 16 July 2008

  12. Lopes, R., Carrico, L.: The impact of accessibility assessment in macro scale universal usability studies of the web. In: Proceedings of the 2008 international cross-disciplinary workshop on Web accessibility (W4A), pp. 5–14 (2008)

  13. Molich, R.: Usable web design (in Danish), Ingenioren boger, Copenhagen, Denmark (2000)

  14. Norge.no: Kvalitetsvurdering av offentlige nettsteder (Quality assessment of public websites). http://www.norge.no/kvalitet (2008). Accessed 16 July 2008

  15. Petrie, H., Kheir, O.: The relationship between accessibility and usability of websites. In: Proceedings of the SIGCHI conference on human factors in computing systems CHI07, pp. 397–406 (2007)

  16. Petrie, H., King, N., Hamilton, F.: Accessibility of museum, library and archive websites: the MLA audit. http://www.mla.gov.uk/website/policy/Diversity/Web_Accessibility (2005). Accessed 16 July 2008

  17. Reid, L., Snow-Weaver, A.: WCAG 2.0: a web accessibility standard for the evolving web. ACM, New York (2008)

    Google Scholar 

  18. Rømen, D., Svanæs, D.: Evaluating web site accessibility: validating the WAI guidelines through usability testing with disabled users. In: Proceedings of the 5th Nordic conference on human-computer interaction: building bridges. ACM Press, Lund, Sweden (2008)

  19. Velleman, E., Strobbe, C., Koch, J., Velasco, C., Snaprud, M.: A unified web evaluation methodology using WCAG. Lect. Notes Comput. Sci. 4556, 177 (2007)

    Article  Google Scholar 

  20. Web Guidelines: http://www.webrichtlijnen.nl/english/ (2007). Accessed 29 July 2010

  21. WAI website: http://www.w3.org/WAI/intro/accessibility. Accessed 29 July 2010

  22. Watanabe, T., Umegaki, M.: Capability survey of user agents with the UAAG 1.0 test suite and its impact on web accessibility. Univ. Access Inf. Soc. 6(3), 221–232 (2007)

    Google Scholar 

  23. WHO: WHO definition of blindness. http://www.who.int/mediacentre/factsheets/fs282/en/ (2010). Accessed 28 Oct 2010

  24. W3C: Comparison of WCAG 1.0 Checkpoints to WCAG 2.0, in numerical order. http://www.w3.org/WAI/WCAG20/from10/comparison (2008). Accessed 29 July 2010

  25. W3C ATAG: http://www.w3.org/TR/WAI-AUTOOLS/. Accessed 29 July 2010

  26. W3C UAAG: http://www.w3.org/WAI/intro/uaag.php. Accessed 29 July 2010

  27. W3C WAI-AIRA: http://www.w3.org/TR/wai-aria/. Accessed 29 July 2010

Download references

Acknowledgments

We would like to thank the test participants and the organizations representing the disabled users. Also thanks to Terje Røsand at NTNU/NSEP for highly valuable technical assistance.

Author information

Author notes

  1. Dagfinn Rømen

    Present address: Agency for Public Management and eGovernment (Difi), 6863, Leikanger, Norway

Authors and Affiliations

  1. Department of Computer and Information Science, Norwegian University of Science and Technology, 7491, Trondheim, Norway

    Dagfinn Rømen & Dag Svanæs

Authors
  1. Dagfinn Rømen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dag Svanæs
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dagfinn Rømen.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rømen, D., Svanæs, D. Validating WCAG versions 1.0 and 2.0 through usability testing with disabled users. Univ Access Inf Soc 11, 375–385 (2012). https://doi.org/10.1007/s10209-011-0259-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-011-0259-3

Keywords

Search

Navigation