Beyond Conformance: The Role of Accessibility Evaluation Methods

  • Giorgio Brajnik
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5176)


The topic I want to address is the role that accessibility evaluation methods can play in helping the transition from accessibility viewed as standard conformance, to a user-centered accessibility. As we will see, this change sets additional requirements on how evaluations of websites should be carried out.

This paper first discusses different problems that occur while dealing with accessibility. We will see that different people have radically different views of accessibility and how it should be assessed.

The first requirement is a clear definition of what accessibility is and how it should be assessed. The accessibility model discussed in Section 2.1 has precisely this role.

Several existing evaluation methods are then reviewed and discussed, a simple taxonomy is presented, and differences that occur when evaluating accessibility rather than usability are pinpointed.


Assistive Technology Screen Reader Accessibility Problem Accessibility Model Accessibility Evaluation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Brajnik, G.: Modeling content and expression of learning objects in multimodal learning management systems. In: HCI International 2007, FUITEL: Future Interfaces in Technology Enhanced Learning, Beijing, China (July 2007)Google Scholar
  2. 2.
    Brajnik, G.: Measuring web accessibility by estimating severity of barriers. In: 2nd International Workshop on Web Usability and Accessibility IWWUA 2008, Auckland, New Zealand (September 2008)Google Scholar
  3. 3.
    Brajnik, G.: Web Accessibility Testing: When the Method is the Culprit. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A.I. (eds.) ICCHP 2006. LNCS, vol. 4061, pp. 156–163. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  4. 4.
    Brajnik, G.: Comparing accessibility evaluation tools: a method for tool effectiveness. Int. Journal on Universal Access in the Information Society 3(3-4), 252–263 (2004)CrossRefGoogle Scholar
  5. 5.
     Brajnik, G.: Web accessibility testing with barriers walkthrough (March 2006) (Visited May 2008),
  6. 6.
    Brajnik, G., Lomuscio, R.: SAMBA: a semi-automatic method for measuring barriers of accessibility. In: Trewin, S., Pontelli, E. (eds.) 9th Int. ACM SIGACCESS Conference on Computers and Accessibility, ASSETS, Tempe, AZ. ACM Press, New York (2007)Google Scholar
  7. 7.
    Brajnik, G., Toppano, E.: Creare siti web multimediali: fondamenti di analisi e progettazione, Italy. Addison-Wesley/Pearson Education (2007)Google Scholar
  8. 8.
    Brajnik, G., Mulas, A., Pitton, C.: Effects of sampling methods on web accessibility evaluations. In: Trewin, S., Pontelli, E. (eds.) 9th Int. ACM SIGACCESS Conference on Computers and Accessibility, ASSETS, Tempe, AZ. ACM Press, New York (2007)Google Scholar
  9. 9.
    Buillon, L., Vanderdonckt, J.: Retargeting web pages on other computing platforms with vaquita. In: van Deursen, Burd, A. (eds.) Proc. of IEEE Working Conf. on Reverse Engineering WCRE 2002, Richmond, October 2002, pp. 339–348. IEEE Computer Society Press, Los Alamitos (2002)CrossRefGoogle Scholar
  10. 10.
    College of Design, North Carolina University. Principles of Universal Design. The Center for Universal Design (Febuary 1997) (Visited May 2008),
  11. 11.
    Coyne, K.P., Nielsen, J.: How to conduct usability evaluations for accessibility: methodology guidelines for testing websites and intranets with users who use assistive technology. Nielsen Norman Group (October 2001),
  12. 12.
    Dey, A.: Accessibility evaluation practices - survey results (2004) (Visited May 2008),
  13. 13.
    DRC. Formal investigation report: web accessibility. Disability Rights Commission (April 2004) (Visited January 2006),
  14. 14.
    Gray, W.D., Salzman, M.C.: Damaged merchandise: a review of experiments that compare usability evaluation methods. Human–Computer Interaction 13(3), 203–261 (1998)CrossRefGoogle Scholar
  15. 15.
    Hartson, H.R., Andre, T.S., Williges, R.C.: Criteria for evaluating usability evaluation methods. Journal of Human-Computer Interaction 15(1), 145–181 (2003)CrossRefGoogle Scholar
  16. 16.
    Henry, S.L., Grossnickle, M.: Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, On-line book (2004),
  17. 17.
    Hertzum, M., Jacobsen, N.E.: The evaluator effect: a chilling fact about usability evaluation methods. Int. Journal of Human-Computer Interaction 1(4), 421–443 (2001)CrossRefGoogle Scholar
  18. 18.
    ISO. Ergonomics of human–system interaction — guidance on accessibility for human–computer interfaces. ISO/TS 16071. Technical report, International Standards Organization (2003),
  19. 19.
    Italian Government. Requisiti tecnici e i diversi livelli per l’accessibilità agli strumenti informatici (July 2005) (G. U. n. 183 8/8/2005),
  20. 20.
    Italian Parliament. Disposizioni per favorire l’accesso dei soggetti disabili agli strumenti informatici (January 2004) (Law n. 4, January 9 2004),
  21. 21.
    Kelly, B., Sloan, D., Phipps, L., Petrie, H., Hamilton, F.: Forcing standardization or accomodating diversity? A framework for applying the WCAG in the real world. In: Harper, S., Yesilada, Y., Goble, C. (eds.) W4A 2005: Proc. of the 2005 international cross-disciplinary conference on Web accessibility, Chiba, Japan, April 2005, pp. 46–54. ACM, New York (2005)CrossRefGoogle Scholar
  22. 22.
    Kelly, B., Sloan, D., Brown, S., Seale, J., Petrie, H., Lauke, P., Ball, S.: Accessibility 2.0: people, policies and processes. In: W4A 2007: Proc. of the 2007 international cross-disciplinary conference on Web accessibility (W4A), pp. 138–147. ACM, New York (2007)CrossRefGoogle Scholar
  23. 23.
    Lang, T.: Comparing website accessibility evaluation methods and learnings from usability evaluation methods (Visited May 2008) (2003),
  24. 24.
    Mankoff, J., Fait, H., Tran, T.: Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind. In: CHI 2005: Proc. of the SIGCHI conference on Human factors in computing systems, pp. 41–50. ACM, New York (2005)CrossRefGoogle Scholar
  25. 25.
    Nielsen, J.: Usability Engineering. Morgan Kaufmann, San Francisco (1994)Google Scholar
  26. 26.
    Nielsen, J.: Heuristic evaluation (2002) (Visited January 2008),
  27. 27.
    Nielsen Norman Group. Beyond ALT Text: Making the Web Easy to Use for Users with Disabilities (October 2001),
  28. 28.
    Nielsen Norman Group. Web usability for senior citizens (April 2002),
  29. 29.
    Petrie, H., Kheir, O.: The relationship between accessibility and usability of websites. In: Proc. CHI 2007, pp. 397–406. ACM, CA (2007)Google Scholar
  30. 30.
    Petrie, H., Hamilton, F., King, N., Pavan, P.: Remote usability evaluations with disabled people. In: CHI 2006: Proc. of the SIGCHI conference on Human factors in computing systems, pp. 1133–1141. ACM, New York (2006)CrossRefGoogle Scholar
  31. 31.
    Rubin, J.: Handbook of usability testing. Wiley, Technical Communication Library, Chichester (1994)Google Scholar
  32. 32.
    Sears, A.: Heuristic walkthroughs: finding the problems without the noise. Int. Journal of Human-Computer Interaction 9(3), 213–234 (1997)CrossRefGoogle Scholar
  33. 33.
    Slatin, J., Lewis, K.: Challenges of accessible web design: Standards, guidelines, and user testing. In: Technology and Persons with Disabilities Conference, Los Angeles, USA. CSUN, California State University Northridge (2002)Google Scholar
  34. 34.
    Slatin, J., Rush, S.: Maximum Accessibility: Making Your Web Site More Usable for Everyone. Addison-Wesley, Reading (2003)Google Scholar
  35. 35.
    Thatcher, J., Waddell, C., Henry, S., Swierenga, S., Urban, M., Burks, M., Regan, B., Bohman, P.: Constructing Accessible Web Sites. Glasshouse (2002)Google Scholar
  36. 36.
    Thatcher, J., Burks, M., Heilmann, C., Henry, S., Kirkpatrick, A., Lauke, P., Lawson, B., Regan, B., Rutter, R., Urban, M., Waddell, C.: Web Accessibility: Web Standards and Regulatory Compliance (2006); Friends of EDGoogle Scholar
  37. 37.
    Theofanos, M.F., Redish, J.: Bridging the gap: between accessibility and usability. Interactions 10(6), 36–51 (2003)CrossRefGoogle Scholar
  38. 38.
    Thevenin, D., Coutaz, J.: Plasticity of user interfaces: framework and research agenda. In: Sasse, A., Johnson, C. (eds.) Proceedings of Interact 1999, Edinburgh, UK, pp. 110–117 (1999)Google Scholar
  39. 39.
    U.S. Dept. of Justice. Section 508 of the Rehabilitation Act (2001),
  40. 40.
    U.S. Government. SEC. 508. ELECTRONIC AND INFORMATION TECHNOLOGY, 1998 Amendment to Section 508 of the Rehabilitation Act, (1998) (Visited May 2008),
  41. 41.
    Vigo, M., Kobsa, A., Arrue, M., Abascal, J.: User-tailored web accessibility evaluations. In: HyperText 2007, Manchester, UK, September 2007, pp. 95–104. ACM, New York (2007)Google Scholar
  42. 42.
    W3C/WAI. Conformance evaluation of web sites for accessibility (2008)(Visited May 2008),
  43. 43.
    W3C/WAI. How people with disabilities use the web. World Wide Web Consortium — Web Accessibility Initiative (March 2004) (Visited May 2008),
  44. 44.
    W3C/WAI. Web accessibility evaluation tools: Overview. World Wide Web Consortium — Web Accessibility Initiative (2006) (Visited May 2008),
  45. 45.
    W3C/WAI. Web content accessibility guidelines 1.0. World Wide Web Consortium — Web Accessibility Initiative (May 1999),
  46. 46.
    W3C/WAI. Web content accessibility guidelines 2.0 — w3c candidate recommendation 30 april 2008. World Wide Web Consortium — Web Accessibility Initiative (April 2008),

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Giorgio Brajnik
    • 1
  1. 1.Dip. di Matematica e InformaticaUniversità di UdineItaly

Personalised recommendations