Universal Access in the Information Society

, Volume 10, Issue 4, pp 425–441 | Cite as

Validating the effectiveness of EvalAccess when deploying WCAG 2.0 tests

  • Amaia Aizpurua
  • Myriam Arrue
  • Markel Vigo
  • Julio AbascalEmail author
Long Paper


While automatic tools are not intended to replace human judgment, they are crucial in order to develop accessible websites. The release of WCAG 2.0 has caused great expectation, as it is supposed to be precisely testable with automated review tools. Therefore, more effective tools could be developed. However, so far few tools applying WCAG 2.0 have been developed. This paper presents an evaluation framework which has been updated in order to evaluate the new tests. In addition, it describes a validation process carried out in order to verify the effectiveness of the new version of the evaluation tool. The effectiveness is validated by conducting a quantitative and qualitative analysis of the results obtained by applying both versions of the tool (the one implementing WCAG 1.0 and the one implementing WCAG 2.0) to a set of selected web pages, as well as by manual evaluation of an expert for detecting the possible false positives and false negatives produced by each tool.


Web accessibility guidelines Tool effectiveness Automatic evaluation 


  1. 1.
    Abascal, J., Nicolle, C.: Why inclusive design guidelines? In: Abascal, J., Nicolle, C. (eds.) Inclusive Design Guidelines for HCI, Chapter 1, pp. 3–13. Taylor & Francis, London (2001)Google Scholar
  2. 2.
    Abascal, J., Arrue, M., Fajardo, I., Garay, N., Tomás, J.: Use of guidelines to automatically verify web accessibility. Univ. Access Inf. Soc. 3(1), 71–79. (2004). (Springer)CrossRefGoogle Scholar
  3. 3.
    Abou-Zahra, S. (ed.): Web Accessibility Evaluation Tools: Overview. Available at (2006)
  4. 4.
    Arrue, M., Vigo, M., Abascal, J.: Including heterogeneous web accessibility guidelines in the development process. In: Engineering Interactive Systems, EIS 2007, LNCS 4940, pp. 620–637. Springer (2008)Google Scholar
  5. 5.
    Brajnik, G.: Towards valid quality models for websites. In: Proceedings of the 7th Conference on Human Factors and the Web (2001)Google Scholar
  6. 6.
    Brajnik, G.: Comparing accessibility evaluation tools: a method for tool effectiveness. Univ. Access Inf. Soc. 3(3–4), 252–263 (2004). (Springer)CrossRefGoogle Scholar
  7. 7.
    Brajnik, G.: Validity and reliability of web accessibility guidelines. In: Proceedings of the 11th international ACM SIGACCESS Conference on Computers and Accessibility, ASSETS’09, pp. 131–138. ACM Press (2009)Google Scholar
  8. 8.
    Caldwell, B., Cooper, M., Guarino Reid, L., Vanderheiden, G.: Web Content Accessibility Guidelines 2.0 (W3C Recommendation). Available at (2008)
  9. 9.
    Chisholm, W., Vanderheiden, G., Jacobs, I.: Web Content Accessibility Guidelines 1.0. (W3C Recommendation). Available at (1999)
  10. 10.
    Clark, J.: To Hell with WCAG 2.0. A List Apart 217. Available at (2006)
  11. 11.
    Diaper, D., Worman, L.: Two falls out of three in the automated accessibility assessment of world wide web sites: a-prompt v. bobby. In: People and Computers, vol. XVII, pp. 349–363. Springer (2003)Google Scholar
  12. 12.
    Freed, G., Rothberg, M., Wlodkowski, T.: Making Educational Software and Web Sites Accessible. Available at (2003)
  13. 13.
    Fuertes, J.L., González, R., Gutiérrez, E., Martínez, L.: Hera-FFX: a Firefox add-on for semi-automatic web accessibility evaluation. In: Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibility, W4A’09, pp. 26–35. ACM Press (2009)Google Scholar
  14. 14.
    IBM Accessibility Center: Developer guidelines for Web Accessibility. Available at Accessed Dec 2009
  15. 15.
    Ivory, M.Y., Mankoff, J., Le, A.: Using automated tools to improve web site usage by users with diverse abilities. Inf. Technol. Soc. 1(3), 195–236 (2003)Google Scholar
  16. 16.
    Ivory, M.Y., Hearst, M.A.: The state of art in automating usability evaluations of user interfaces. ACM Comput. Surv. 33(4), 470–516. (2001) (ACM Press)CrossRefGoogle Scholar
  17. 17.
    Kurniawan, S., Zaphiris, P.: Research-derived web design guidelines for older people. In: Proceedings of the ACM SIGACCESS Conference on Computers and Accessibility, ASSETS’05, pp. 129–135. ACM Press (2005)Google Scholar
  18. 18.
    Leporini, B., Paternò, F., Scorcia, A.: Flexible tool support for accessibility evaluation. Interact. Comput. 18(5), 869–890 (2006). (Elsevier)CrossRefGoogle Scholar
  19. 19.
    Luque, V., Delgado, C., Gaedke, M., Nussbaumer, M.: WCAG formalization with W3C standards. In: Special Interest Tracks and Posters of the 14th International Conference On World Wide Web, WWW’05, pp. 1146–1147. ACM Press (2005)Google Scholar
  20. 20.
    Mbipom, G., Harper, S.: The transition from web content accessibility guidelines 1.0 to 2.0: what this means for evaluation and repair. In: Proceedings of the 27th ACM International Conference on Design of Communication, SIGDOC’09, pp. 37–44. ACM Press (2009)Google Scholar
  21. 21.
    Rabin, J., McCathieNevile, C.: Mobile Web Best Practices (W3C Recommendation). (2008)
  22. 22.
    Takata, Y., Nakamura, T., Seki, H.: Accessibility verification of WWW documents by an automatic guideline verification tool. In: Proceedings of the 37th Hawaii International Conference on System Sciences, HICSS’04, Track 4, vol. 4. IEEE Computer Society (2004)Google Scholar
  23. 23.
    Vanderdonckt, J.: Development milestones towards a tool for working with guidelines. Interact. Comput. 12(2), 81–118 (1999). (Elsevier)CrossRefGoogle Scholar
  24. 24.
    Vanderdonckt, J., Bereikdar, A.: Automated web evaluation by guideline review. J. Web Eng. 4(2), 102–117. (Rinton Press) (2005)Google Scholar
  25. 25.
    Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., Abascal, J.: Quantitative metrics for measuring web accessibility. In: Proceedings of the International Cross-Disciplinary Conference on Web accessibility, W4A’07, pp. 99–107. ACM Press (2007)Google Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Amaia Aizpurua
    • 1
  • Myriam Arrue
    • 1
  • Markel Vigo
    • 2
  • Julio Abascal
    • 1
    Email author
  1. 1.Informatika FakultateaUniversity of the Basque CountryDonostiaSpain
  2. 2.Computer Science SchoolUniversity of ManchesterManchesterUK

Personalised recommendations