Evaluating e-learning accessibility by automated and student-centered methods

  • Kari L. KumarEmail author
  • Ron Owston
Development Article


The use of learning technologies is becoming ubiquitous in higher education. As a result, there is a pressing need to develop methods to evaluate their accessibility to ensure that students do not encounter barriers to accessibility while engaging in e-learning. In this study, sample online units were evaluated for accessibility by automated tools and by student participants (in sessions moderated and unmoderated by researchers), and the data from these different methods of e-learning accessibility evaluation were compared. Nearly all students were observed encountering one or more barriers to accessibility while completing the online units, though the automated tools did not predict these barriers and instead predicted potential barriers that were not relevant to the study participants. These data underscore the need to carry out student-centered accessibility evaluation in addition to relying on automated tools and accessibility guideline conformance as measures of accessibility. Students preferred to participate in unmoderated sessions, and the data from the unmoderated sessions were comparable to that from the more traditional moderated sessions. Additional work is needed to further explore methods of student-centered evaluation, including different variations of unmoderated sessions.


Accessibility E-learning Educational technology Disabilities Higher education 


  1. Americans with Disabilities Act. (1990, 2008). ADA. Retrieved from
  2. Andreason, M. S., Villemann Nielsen, H., Ormholt Schroeder, S., & Stage, J. (2007). What happened to remote usability testing? An empirical study of three methods. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1405–1414). New York, NY: ACM.Google Scholar
  3. AODA Integrated Accessibility Standards. (2011). Retrieved from the Service Canada website
  4. Asuncion, J. V., Fichten, C. S., Ferraro, V., Chwojka, C., Barile, M., Nguyen, M. N., et al. (2010). Multiple perspectives on the accessibility of e-learning in Canadian colleges and universities. Assistive Technology, 22(4), 187–199.CrossRefGoogle Scholar
  5. Baravalle, A., & Lanfranchi, V. (2003). Remote web usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 364–368.CrossRefGoogle Scholar
  6. Bland, M. (2000). Multiple significance tests and the Bonferroni correction. An introduction to medical statistics (3rd ed.). New York, NY: Oxford Medical Publications.Google Scholar
  7. Bohman, P. R., & Anderson, S. (2005). A conceptual framework for accessibility tools to benefit users with cognitive disabilities. Paper presented at the W4A, Chiba, Japan.Google Scholar
  8. Bolt, N., & Tulathimutte, T. (2010). Remote research. Brooklyn, NY: Rosenfeld Media.Google Scholar
  9. Boren, T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communications, 43(3), 261–278.CrossRefGoogle Scholar
  10. Carnevale, D. (2005). Lawsuit charges online university does not accommodate learning-disabled students. The Chronicle of Higher Education, 51(49), 33.Google Scholar
  11. Cassidy, S. (2004). Learning styles: An overview of theories, models, and measures. Educational Psychology, 24(4), 419–444.CrossRefGoogle Scholar
  12. Chen, Y. T. (2014). A study to explore the effects of self-regulated learning environment for hearing-impaired students. Journal of Computer Assisted learning, 30(2), 97–109.CrossRefGoogle Scholar
  13. Clark, J. (2006, May 23). To hell with the WCAG2. A List Apart, 217. Retrieved from
  14. Denton, W., & Coysh, S. J. (2011). Usability testing of VuFind at an academic library. Library Hi Tech, 29(2), 301–319.CrossRefGoogle Scholar
  15. Dermody, K., & Majekodunmi, N. (2010). Online databases and the research experience for university students with print disabilities. Library Hi Tech, 29(1), 149–160.CrossRefGoogle Scholar
  16. Ellcessor, E. (2010). Bridging disability divides: A critical history of web content accessibility through 2001. Information, Communication & Society, 13(3), 289–308.CrossRefGoogle Scholar
  17. Evett, L., & Brown, D. (2005). Text formats and web design for visually impaired and dyslexic readers—clear text for all. Interacting with Computers, 17(4), 453–472.CrossRefGoogle Scholar
  18. Fichten, C. S., Ferraro, V., Asuncion, J. V., Chwojka, C., Barile, M., Nguyen, M. N., et al. (2009). Disabilities and e-learning problems and solutions: An exploratory study. Educational Technology & Society, 12(4), 241–256.Google Scholar
  19. Foley, A. (2011). Exploring the design, development and use of websites through accessibility and usability studies. Journal of Educational Multimedia and Hypermedia, 20(4), 361–385.Google Scholar
  20. Friedman, M. G., & Bryen, D. N. (2007). Web accessibility design recommendations for people with cognitive disabilities. Technology and Disability, 19(4), 205–212.Google Scholar
  21. Gerrard, C. (2007). Virtual learning environments: Enhancing the learning experience for students with disabilities. Campus-Wide Information Systems, 24(3), 199–206.CrossRefGoogle Scholar
  22. Goggin, G., & Newell, C. (2003). Digital disability: The social construction of disability in new media. New York, NY: Rowman & Littlefield.Google Scholar
  23. Goodall, B. (2008, January 3). Judge finds university didn’t discriminate online. Retrieved from
  24. Grabinger, S. (2010). A framework for supporting postsecondary learners with psychiatric disabilities in online environments. Electronic Journal of e-Learning, 8(2), 101–110.Google Scholar
  25. Habib, L., Berget, G., Sandnes, F. E., Sanderson, N., Kahn, P., Fagernes, S., et al. (2012). Dyslexic students in higher education and virtual learning environments: an exploratory study. Journal of Computer Assisted learning, 28(6), 574–584.CrossRefGoogle Scholar
  26. Hartson, H. R., Castillo, J. C., Kelso, J., & Neale, W. (1996). Remote evaluation: The network as an extension of the usability laboratory. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 228–235). New York, NY: ACM.Google Scholar
  27. Henry, S. L. (2007). Just ask: Integrating accessibility throughout design. Retrieved from
  28. Her Majesty’s Stationary Office. (2001). The Special Educational Needs and Disability Act 2001, London. Retrieved from
  29. Her Majesty’s Stationary Office. (2010). Equality Act 2010, London. Retrieved from
  30. Houck-Whitaker, J. (2005). Remote testing versus lab testing. Retrieved from
  31. Inside Higher Ed. (2014, February 17). Portland State settles complaint by deaf student. Retrieved from
  32. Joint Department of Justice and Department of Education. (2011). Frequently asked questions about the June 29, 2010, Dear Colleagues letter. Retrieved from
  33. Jung, S., Herlocker, J. L., Webster, J., Mellinger, M., & Frumkin, J. (2008). LibraryFind: System design and usability testing of academic metasearch system. Journal of the American Society for Information Science and Technology, 59(3), 375–389.CrossRefGoogle Scholar
  34. Kapsi, M., Vlachogiannis, E., Darzentas, J., & Spyrou, T. (2009a). A preliminary feedback for the WCAG 2.0-WCAG 1.0 Vs WCAG 2.0 evaluation study. Paper presented at the 2nd annual conference on pervasive technologies related to assistive environments (PETRA), Confu, Greece.Google Scholar
  35. Kapsi, M., Vlachogiannis, E., Darzentas, J., & Spyrou, T. (2009b). The usability of web accessibility guidelines: An approach for evaluation. In Proceedings of the 5th international conference on universal access in human computer interaction (pp. 716–724). Berlin: Springer.Google Scholar
  36. Kelly, B., Phipps, L., & Swift, E. (2004). Developing a holistic approach for e-learning accessibility. Canadian Journal of Learning and Technology, 30(3), 20.Google Scholar
  37. Kennedy, H., Evans, S., & Thomas, S. (2011). Can the web be made accessible for people with intellectual disabilities? The Information Society, 27(1), 29–39.CrossRefGoogle Scholar
  38. Kumar, K., & Owston, R. (2012). Learnings from post-secondary students on e-learning accessibility and accessibility evaluation methods. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2012 (pp. 663–669). Chesapeake, VA: AACE.Google Scholar
  39. Laff, M., & Rissenberg, M. (2007). Cognitive ability measures for accessible web content. Paper presented at the 4th international conference on universal access in human computer interaction (HCI), Beijing, China.Google Scholar
  40. Marks, D. (1997). Models of disability. Disability and Rehabilitation, 19(3), 85–91.CrossRefGoogle Scholar
  41. McCarthy, J. E., & Swierenga, S. J. (2010). What we know about dyslexia and web accessibility: A research review. Universal Access in the Information Society, 9(2), 147–152.CrossRefGoogle Scholar
  42. McFadden, E., Hager, D. R., Elie, C. J., & Blackwell, J. M. (2002). Remote usability evaluation: Overview and case studies. International Journal of Human-Computer Interaction, 14(3&4), 489–502.CrossRefGoogle Scholar
  43. Narasimhan, N. (Ed.). (2012). Web accessibility policy making: An international perspective. New York: Hemkunt Publishers Ltd.Google Scholar
  44. National Federation of the Blind. (2012, March 6). Florida State resolves litigation with students [Press release]. Retrieved from
  45. National Federation of the Blind. (2014, January 13). Blind student files discrimination suit against Miami University [Press release]. Retrieved from
  46. Nicolle, C., & Paulson, D. (2004). Making the Internet accessible by people with cognitive and communication impairments. Universal Access in the Information Society, 3(1), 48–56.CrossRefGoogle Scholar
  47. Nielsen, J. (1993). Usability engineering. Boston, MA: AP Professional.Google Scholar
  48. Oliver, M. (1996). Understanding disability: From theory to practice. Houndmills: Macmillan.CrossRefGoogle Scholar
  49. Olmsted-Hawala, E. L., Murphy, E. D., Hawala, S., & Ashenfelter, K. T. (2010a). Think-aloud protocols: A comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 2381–2390). New York, NY: ACM.Google Scholar
  50. Olmsted-Hawala, E. L., Murphy, E. D., Hawala, S., & Ashenfelter, K. T. (2010b). Think-aloud protocols: Analyzing three different think-aloud protocols with counts of verbalized frustrations in a usability study of an information-rich Web site. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 60–66). New York, NY: ACM.Google Scholar
  51. Petrie, H., Hamilton, F., King, N., & Pavan, P. (2006). Remote usability evaluations with disabled people. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1133–1141). New York, NY: ACM.Google Scholar
  52. Power, C., Petrie, H., & Mitchell, R. (2009). A framework for remote user evaluation of accessibility and usability of websites. In C. Stephanidis (Ed.), Universal access in human computer interaction. Addressing diversity (Vol. 5614, pp. 594–601)., Lecture Notes in Computer Science Berlin, Germany: Springer-Verlag.CrossRefGoogle Scholar
  53. Power, C., Petrie, H., Sakharov, V., & Swallow, D. (2010). Virtual learning environments: Another barrier to blended and e-learning. In Proceedings of the 12th international conference on computers helping people with special needs: part I (pp. 519–526). Berlin: Springer.Google Scholar
  54. Pretorius, M., & van Biljon, J. (2010). Learning management systems: ICT skills, usability and learnability. Interactive Technology and Smart Education, 7(1), 30–43.CrossRefGoogle Scholar
  55. Rehabilitation Act. (1998). Rehabilitation Act of 1973. Retrieved from
  56. Ribera, M., Porras, M., Boldu, M., Termens, M., Sule, A., & Paris, P. (2009). Web Content Accessibility Guidelines 2.0: A further step towards accessible digital information. Program: Electronic Library and Information Systems, 43(4), 392–406.CrossRefGoogle Scholar
  57. Roberts, J. B., Crittenden, L. A., & Crittenden, J. C. (2011). Students with disabilities and online learning: A cross-institutional study of perceived satisfaction with accessibility compliance and services. Internet and Higher Education, 14(4), 242–250.CrossRefGoogle Scholar
  58. Rubin, J., & Chisnell, D. (2008). Handbook of usability testing. Indianapolis, IN: Wiley.Google Scholar
  59. Seale, J. K. (2003). The challenge of researching accessibility practices within Higher Education: An exploration of “shared enterprises” or “political games”. Paper presented at the International Education Research Conference AARE and NZARE.Google Scholar
  60. Seeman, L. (2006). Re: Formal objection to WCAG 2.0. [Electronic mailing list message]. Retrieved from
  61. Selvaraj, P. (2004). Comparative study of synchronous remote and traditional in-lab usability evaluation methods. Unpublished Unpublished Master’s thesis, Virginia Polytechnic Institute and State University.Google Scholar
  62. Sloan, D., & Kelly, B. (2008). Reflections on the development of a holistic approach to web accessibility. Paper presented at the 2008 accessible design in the digital world conference (ADDW), York.Google Scholar
  63. Thatcher, J., Burks, M. R., Heilmann, C., Henry, S. L., Kirkpatrick, A., Lauke, P. H., et al. (2006). Web accessibility: Web standards and regulatory compliance. Retrieved from
  64. Thompson, T., Comden, D., Ferguson, S., Burgstahler, S., & Moore, E. (2013). Seeking predictors of web accessibility in U.S. higher education institutions. Information Technology and Disabilities, 13(1), 18.Google Scholar
  65. U.S. Department of Justice. (2010a). Letter of Resolution, D.J. No. 202-48-213 Princeton University. Retrieved from
  66. U.S. Department of Justice. (2010b). Settlement agreement between the United States of America, the National Federation of the Blind, Inc. (“NFB”), and the American Council of the Blind (“ACB”), and the Arizona Board of Regents (“ABOR”), for and on behalf of Arizona State University (“ASU”). Retrieved from
  67. U.S. Department of Justice. (2013, July 23). Justice department settles with Louisiana Tech University over inaccessible course materials [Press release]. Retrieved from
  68. Ustun, T. B., Chatterji, S., Bickenbach, J., Kostanjsek, N., & Schneider, M. (2003). The international classification for functioning, disability and health: A new tool for understanding disability and health. Disability and Rehabilitation, 25(11–12), 565–571.CrossRefGoogle Scholar
  69. Vanderheiden, G. (2000). Fundamental principles and priority setting for universal usability. In Proceedings of the 2000 conference on universal usability (pp. 32–38). New York, NY: ACM.Google Scholar
  70. Vigo, M., & Brajnik, G. (2011). Automatic web accessibility metrics: Where we are and where we can go. Interacting with Computers, 23(2), 137–155.CrossRefGoogle Scholar
  71. W3C. (2008). Web Content Accessibility Guidelines 2.0: W3C Recommendation 11 December 2008. Retrieved from
  72. Woodfine, B. P., Nunes, M. B., & Wright, D. J. (2008). Text-based synchronous e-learning and dyslexia: Not necessarily the perfect match! Computers & Education, 50(3), 703–717.CrossRefGoogle Scholar

Copyright information

© Association for Educational Communications and Technology 2015

Authors and Affiliations

  1. 1.University of Manitoba, Extended EducationWinnipegCanada
  2. 2.Faculty of Education, Winters CollegeYork UniversityTorontoCanada

Personalised recommendations