Skip to main content
SpringerLink
Log in

Validating the effectiveness of EvalAccess when deploying WCAG 2.0 tests

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

While automatic tools are not intended to replace human judgment, they are crucial in order to develop accessible websites. The release of WCAG 2.0 has caused great expectation, as it is supposed to be precisely testable with automated review tools. Therefore, more effective tools could be developed. However, so far few tools applying WCAG 2.0 have been developed. This paper presents an evaluation framework which has been updated in order to evaluate the new tests. In addition, it describes a validation process carried out in order to verify the effectiveness of the new version of the evaluation tool. The effectiveness is validated by conducting a quantitative and qualitative analysis of the results obtained by applying both versions of the tool (the one implementing WCAG 1.0 and the one implementing WCAG 2.0) to a set of selected web pages, as well as by manual evaluation of an expert for detecting the possible false positives and false negatives produced by each tool.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://www.w3.org/WAI/ER/tests/.

  2. http://wave.webaim.org/.

  3. http://www.achecker.ca/checker/index.php.

  4. http://sipt07.si.ehu.es/evalaccess2/.

  5. http://fae.cita.uiuc.edu/.

  6. http://www.tawdis.net.

  7. XML-Schema of UGL: http://sipt07.si.ehu.es/evalaccess3/ugl.xsd. Its representation: http://sipt07.si.ehu.es/evalaccess3/ugl.png.

References

  1. Abascal, J., Nicolle, C.: Why inclusive design guidelines? In: Abascal, J., Nicolle, C. (eds.) Inclusive Design Guidelines for HCI, Chapter 1, pp. 3–13. Taylor & Francis, London (2001)

    Google Scholar 

  2. Abascal, J., Arrue, M., Fajardo, I., Garay, N., Tomás, J.: Use of guidelines to automatically verify web accessibility. Univ. Access Inf. Soc. 3(1), 71–79. (2004). (Springer)

    Article  Google Scholar 

  3. Abou-Zahra, S. (ed.): Web Accessibility Evaluation Tools: Overview. Available at http://www.w3.org/WAI/ER/tools/ (2006)

  4. Arrue, M., Vigo, M., Abascal, J.: Including heterogeneous web accessibility guidelines in the development process. In: Engineering Interactive Systems, EIS 2007, LNCS 4940, pp. 620–637. Springer (2008)

  5. Brajnik, G.: Towards valid quality models for websites. In: Proceedings of the 7th Conference on Human Factors and the Web (2001)

  6. Brajnik, G.: Comparing accessibility evaluation tools: a method for tool effectiveness. Univ. Access Inf. Soc. 3(3–4), 252–263 (2004). (Springer)

    Article  Google Scholar 

  7. Brajnik, G.: Validity and reliability of web accessibility guidelines. In: Proceedings of the 11th international ACM SIGACCESS Conference on Computers and Accessibility, ASSETS’09, pp. 131–138. ACM Press (2009)

  8. Caldwell, B., Cooper, M., Guarino Reid, L., Vanderheiden, G.: Web Content Accessibility Guidelines 2.0 (W3C Recommendation). Available at http://www.w3.org/TR/WCAG20/ (2008)

  9. Chisholm, W., Vanderheiden, G., Jacobs, I.: Web Content Accessibility Guidelines 1.0. (W3C Recommendation). Available at http://www.w3.org/TR/WAI-WEBCONTENT/ (1999)

  10. Clark, J.: To Hell with WCAG 2.0. A List Apart 217. Available at http://www.alistapart.com/articles/tohellwithwcag2 (2006)

  11. Diaper, D., Worman, L.: Two falls out of three in the automated accessibility assessment of world wide web sites: a-prompt v. bobby. In: People and Computers, vol. XVII, pp. 349–363. Springer (2003)

  12. Freed, G., Rothberg, M., Wlodkowski, T.: Making Educational Software and Web Sites Accessible. Available at http://ncam.wgbh.org/cdrom/guideline/ (2003)

  13. Fuertes, J.L., González, R., Gutiérrez, E., Martínez, L.: Hera-FFX: a Firefox add-on for semi-automatic web accessibility evaluation. In: Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibility, W4A’09, pp. 26–35. ACM Press (2009)

  14. IBM Accessibility Center: Developer guidelines for Web Accessibility. Available at http://www-306.ibm.com/able/guidelines/web/accessweb.html. Accessed Dec 2009

  15. Ivory, M.Y., Mankoff, J., Le, A.: Using automated tools to improve web site usage by users with diverse abilities. Inf. Technol. Soc. 1(3), 195–236 (2003)

    Google Scholar 

  16. Ivory, M.Y., Hearst, M.A.: The state of art in automating usability evaluations of user interfaces. ACM Comput. Surv. 33(4), 470–516. (2001) (ACM Press)

    Article  Google Scholar 

  17. Kurniawan, S., Zaphiris, P.: Research-derived web design guidelines for older people. In: Proceedings of the ACM SIGACCESS Conference on Computers and Accessibility, ASSETS’05, pp. 129–135. ACM Press (2005)

  18. Leporini, B., Paternò, F., Scorcia, A.: Flexible tool support for accessibility evaluation. Interact. Comput. 18(5), 869–890 (2006). (Elsevier)

    Article  Google Scholar 

  19. Luque, V., Delgado, C., Gaedke, M., Nussbaumer, M.: WCAG formalization with W3C standards. In: Special Interest Tracks and Posters of the 14th International Conference On World Wide Web, WWW’05, pp. 1146–1147. ACM Press (2005)

  20. Mbipom, G., Harper, S.: The transition from web content accessibility guidelines 1.0 to 2.0: what this means for evaluation and repair. In: Proceedings of the 27th ACM International Conference on Design of Communication, SIGDOC’09, pp. 37–44. ACM Press (2009)

  21. Rabin, J., McCathieNevile, C.: Mobile Web Best Practices (W3C Recommendation). http://www.w3.org/TR/mobile-bp/ (2008)

  22. Takata, Y., Nakamura, T., Seki, H.: Accessibility verification of WWW documents by an automatic guideline verification tool. In: Proceedings of the 37th Hawaii International Conference on System Sciences, HICSS’04, Track 4, vol. 4. IEEE Computer Society (2004)

  23. Vanderdonckt, J.: Development milestones towards a tool for working with guidelines. Interact. Comput. 12(2), 81–118 (1999). (Elsevier)

    Article  Google Scholar 

  24. Vanderdonckt, J., Bereikdar, A.: Automated web evaluation by guideline review. J. Web Eng. 4(2), 102–117. (Rinton Press) (2005)

    Google Scholar 

  25. Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., Abascal, J.: Quantitative metrics for measuring web accessibility. In: Proceedings of the International Cross-Disciplinary Conference on Web accessibility, W4A’07, pp. 99–107. ACM Press (2007)

Download references

Author information

Authors and Affiliations

  1. Informatika Fakultatea, University of the Basque Country, Manuel Lardizabal 1, 20018, Donostia, Spain

    Amaia Aizpurua, Myriam Arrue & Julio Abascal

  2. Computer Science School, University of Manchester, Kilburn Building, Oxford Road, Manchester, M13 9PL, UK

    Markel Vigo

Authors
  1. Amaia Aizpurua
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Myriam Arrue
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Markel Vigo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Julio Abascal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Julio Abascal.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Aizpurua, A., Arrue, M., Vigo, M. et al. Validating the effectiveness of EvalAccess when deploying WCAG 2.0 tests. Univ Access Inf Soc 10, 425–441 (2011). https://doi.org/10.1007/s10209-011-0226-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-011-0226-z

Keywords

Search

Navigation