Evaluating e-learning accessibility by automated and student-centered methods


The use of learning technologies is becoming ubiquitous in higher education. As a result, there is a pressing need to develop methods to evaluate their accessibility to ensure that students do not encounter barriers to accessibility while engaging in e-learning. In this study, sample online units were evaluated for accessibility by automated tools and by student participants (in sessions moderated and unmoderated by researchers), and the data from these different methods of e-learning accessibility evaluation were compared. Nearly all students were observed encountering one or more barriers to accessibility while completing the online units, though the automated tools did not predict these barriers and instead predicted potential barriers that were not relevant to the study participants. These data underscore the need to carry out student-centered accessibility evaluation in addition to relying on automated tools and accessibility guideline conformance as measures of accessibility. Students preferred to participate in unmoderated sessions, and the data from the unmoderated sessions were comparable to that from the more traditional moderated sessions. Additional work is needed to further explore methods of student-centered evaluation, including different variations of unmoderated sessions.

This is a preview of subscription content, access via your institution.

Fig. 1


  1. Americans with Disabilities Act. (1990, 2008). ADA. Retrieved from http://www.ada.gov/pubs/ada.htm.

  2. Andreason, M. S., Villemann Nielsen, H., Ormholt Schroeder, S., & Stage, J. (2007). What happened to remote usability testing? An empirical study of three methods. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1405–1414). New York, NY: ACM.

  3. AODA Integrated Accessibility Standards. (2011). Retrieved from the Service Canada website http://www.e-laws.gov.on.ca/html/source/regs/english/2011/elaws_src_regs_r11191_e.htm#BK15.

  4. Asuncion, J. V., Fichten, C. S., Ferraro, V., Chwojka, C., Barile, M., Nguyen, M. N., et al. (2010). Multiple perspectives on the accessibility of e-learning in Canadian colleges and universities. Assistive Technology, 22(4), 187–199.

    Article  Google Scholar 

  5. Baravalle, A., & Lanfranchi, V. (2003). Remote web usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 364–368.

    Article  Google Scholar 

  6. Bland, M. (2000). Multiple significance tests and the Bonferroni correction. An introduction to medical statistics (3rd ed.). New York, NY: Oxford Medical Publications.

    Google Scholar 

  7. Bohman, P. R., & Anderson, S. (2005). A conceptual framework for accessibility tools to benefit users with cognitive disabilities. Paper presented at the W4A, Chiba, Japan.

  8. Bolt, N., & Tulathimutte, T. (2010). Remote research. Brooklyn, NY: Rosenfeld Media.

    Google Scholar 

  9. Boren, T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communications, 43(3), 261–278.

    Article  Google Scholar 

  10. Carnevale, D. (2005). Lawsuit charges online university does not accommodate learning-disabled students. The Chronicle of Higher Education, 51(49), 33.

    Google Scholar 

  11. Cassidy, S. (2004). Learning styles: An overview of theories, models, and measures. Educational Psychology, 24(4), 419–444.

    Article  Google Scholar 

  12. Chen, Y. T. (2014). A study to explore the effects of self-regulated learning environment for hearing-impaired students. Journal of Computer Assisted learning, 30(2), 97–109.

    Article  Google Scholar 

  13. Clark, J. (2006, May 23). To hell with the WCAG2. A List Apart, 217. Retrieved from http://www.alistapart.com/articles/tohellwithwcag2.

  14. Denton, W., & Coysh, S. J. (2011). Usability testing of VuFind at an academic library. Library Hi Tech, 29(2), 301–319.

    Article  Google Scholar 

  15. Dermody, K., & Majekodunmi, N. (2010). Online databases and the research experience for university students with print disabilities. Library Hi Tech, 29(1), 149–160.

    Article  Google Scholar 

  16. Ellcessor, E. (2010). Bridging disability divides: A critical history of web content accessibility through 2001. Information, Communication & Society, 13(3), 289–308.

    Article  Google Scholar 

  17. Evett, L., & Brown, D. (2005). Text formats and web design for visually impaired and dyslexic readers—clear text for all. Interacting with Computers, 17(4), 453–472.

    Article  Google Scholar 

  18. Fichten, C. S., Ferraro, V., Asuncion, J. V., Chwojka, C., Barile, M., Nguyen, M. N., et al. (2009). Disabilities and e-learning problems and solutions: An exploratory study. Educational Technology & Society, 12(4), 241–256.

    Google Scholar 

  19. Foley, A. (2011). Exploring the design, development and use of websites through accessibility and usability studies. Journal of Educational Multimedia and Hypermedia, 20(4), 361–385.

    Google Scholar 

  20. Friedman, M. G., & Bryen, D. N. (2007). Web accessibility design recommendations for people with cognitive disabilities. Technology and Disability, 19(4), 205–212.

    Google Scholar 

  21. Gerrard, C. (2007). Virtual learning environments: Enhancing the learning experience for students with disabilities. Campus-Wide Information Systems, 24(3), 199–206.

    Article  Google Scholar 

  22. Goggin, G., & Newell, C. (2003). Digital disability: The social construction of disability in new media. New York, NY: Rowman & Littlefield.

    Google Scholar 

  23. Goodall, B. (2008, January 3). Judge finds university didn’t discriminate online. Retrieved from http://chronicle.com/blogs/wiredcampus/judge-finds-university-didnt-discriminate-online/3582.

  24. Grabinger, S. (2010). A framework for supporting postsecondary learners with psychiatric disabilities in online environments. Electronic Journal of e-Learning, 8(2), 101–110.

    Google Scholar 

  25. Habib, L., Berget, G., Sandnes, F. E., Sanderson, N., Kahn, P., Fagernes, S., et al. (2012). Dyslexic students in higher education and virtual learning environments: an exploratory study. Journal of Computer Assisted learning, 28(6), 574–584.

    Article  Google Scholar 

  26. Hartson, H. R., Castillo, J. C., Kelso, J., & Neale, W. (1996). Remote evaluation: The network as an extension of the usability laboratory. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 228–235). New York, NY: ACM.

  27. Henry, S. L. (2007). Just ask: Integrating accessibility throughout design. Retrieved from http://www.uiaccess.com/JustAsk/index.html.

  28. Her Majesty’s Stationary Office. (2001). The Special Educational Needs and Disability Act 2001, London. Retrieved from http://www.legislation.gov.uk/ukpga/2001/10/contents.

  29. Her Majesty’s Stationary Office. (2010). Equality Act 2010, London. Retrieved from http://www.legislation.gov.uk/ukpga/2010/15/section/92.

  30. Houck-Whitaker, J. (2005). Remote testing versus lab testing. Retrieved from http://www.boltpeters.com/articles/versus.html.

  31. Inside Higher Ed. (2014, February 17). Portland State settles complaint by deaf student. Retrieved from http://www.insidehighered.com/quicktakes/2014/02/17/portland-state-settles-complaint-deaf-student#sthash.YQTpkfRK.dpbs.

  32. Joint Department of Justice and Department of Education. (2011). Frequently asked questions about the June 29, 2010, Dear Colleagues letter. Retrieved from http://www2.ed.gov/about/offices/list/ocr/docs/dcl-ebook-faq-201105.html.

  33. Jung, S., Herlocker, J. L., Webster, J., Mellinger, M., & Frumkin, J. (2008). LibraryFind: System design and usability testing of academic metasearch system. Journal of the American Society for Information Science and Technology, 59(3), 375–389.

    Article  Google Scholar 

  34. Kapsi, M., Vlachogiannis, E., Darzentas, J., & Spyrou, T. (2009a). A preliminary feedback for the WCAG 2.0-WCAG 1.0 Vs WCAG 2.0 evaluation study. Paper presented at the 2nd annual conference on pervasive technologies related to assistive environments (PETRA), Confu, Greece.

  35. Kapsi, M., Vlachogiannis, E., Darzentas, J., & Spyrou, T. (2009b). The usability of web accessibility guidelines: An approach for evaluation. In Proceedings of the 5th international conference on universal access in human computer interaction (pp. 716–724). Berlin: Springer.

  36. Kelly, B., Phipps, L., & Swift, E. (2004). Developing a holistic approach for e-learning accessibility. Canadian Journal of Learning and Technology, 30(3), 20.

    Google Scholar 

  37. Kennedy, H., Evans, S., & Thomas, S. (2011). Can the web be made accessible for people with intellectual disabilities? The Information Society, 27(1), 29–39.

    Article  Google Scholar 

  38. Kumar, K., & Owston, R. (2012). Learnings from post-secondary students on e-learning accessibility and accessibility evaluation methods. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2012 (pp. 663–669). Chesapeake, VA: AACE.

  39. Laff, M., & Rissenberg, M. (2007). Cognitive ability measures for accessible web content. Paper presented at the 4th international conference on universal access in human computer interaction (HCI), Beijing, China.

  40. Marks, D. (1997). Models of disability. Disability and Rehabilitation, 19(3), 85–91.

    Article  Google Scholar 

  41. McCarthy, J. E., & Swierenga, S. J. (2010). What we know about dyslexia and web accessibility: A research review. Universal Access in the Information Society, 9(2), 147–152.

    Article  Google Scholar 

  42. McFadden, E., Hager, D. R., Elie, C. J., & Blackwell, J. M. (2002). Remote usability evaluation: Overview and case studies. International Journal of Human-Computer Interaction, 14(3&4), 489–502.

    Article  Google Scholar 

  43. Narasimhan, N. (Ed.). (2012). Web accessibility policy making: An international perspective. New York: Hemkunt Publishers Ltd.

    Google Scholar 

  44. National Federation of the Blind. (2012, March 6). Florida State resolves litigation with students [Press release]. Retrieved from https://nfb.org/node/931.

  45. National Federation of the Blind. (2014, January 13). Blind student files discrimination suit against Miami University [Press release]. Retrieved from https://nfb.org/blind-student-files-discrimination-suit-against-miami-university.

  46. Nicolle, C., & Paulson, D. (2004). Making the Internet accessible by people with cognitive and communication impairments. Universal Access in the Information Society, 3(1), 48–56.

    Article  Google Scholar 

  47. Nielsen, J. (1993). Usability engineering. Boston, MA: AP Professional.

    Google Scholar 

  48. Oliver, M. (1996). Understanding disability: From theory to practice. Houndmills: Macmillan.

    Google Scholar 

  49. Olmsted-Hawala, E. L., Murphy, E. D., Hawala, S., & Ashenfelter, K. T. (2010a). Think-aloud protocols: A comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 2381–2390). New York, NY: ACM.

  50. Olmsted-Hawala, E. L., Murphy, E. D., Hawala, S., & Ashenfelter, K. T. (2010b). Think-aloud protocols: Analyzing three different think-aloud protocols with counts of verbalized frustrations in a usability study of an information-rich Web site. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 60–66). New York, NY: ACM.

  51. Petrie, H., Hamilton, F., King, N., & Pavan, P. (2006). Remote usability evaluations with disabled people. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1133–1141). New York, NY: ACM.

  52. Power, C., Petrie, H., & Mitchell, R. (2009). A framework for remote user evaluation of accessibility and usability of websites. In C. Stephanidis (Ed.), Universal access in human computer interaction. Addressing diversity (Vol. 5614, pp. 594–601)., Lecture Notes in Computer Science Berlin, Germany: Springer-Verlag.

    Google Scholar 

  53. Power, C., Petrie, H., Sakharov, V., & Swallow, D. (2010). Virtual learning environments: Another barrier to blended and e-learning. In Proceedings of the 12th international conference on computers helping people with special needs: part I (pp. 519–526). Berlin: Springer.

  54. Pretorius, M., & van Biljon, J. (2010). Learning management systems: ICT skills, usability and learnability. Interactive Technology and Smart Education, 7(1), 30–43.

    Article  Google Scholar 

  55. Rehabilitation Act. (1998). Rehabilitation Act of 1973. Retrieved from http://www2.ed.gov/policy/speced/reg/narrative.html.

  56. Ribera, M., Porras, M., Boldu, M., Termens, M., Sule, A., & Paris, P. (2009). Web Content Accessibility Guidelines 2.0: A further step towards accessible digital information. Program: Electronic Library and Information Systems, 43(4), 392–406.

    Article  Google Scholar 

  57. Roberts, J. B., Crittenden, L. A., & Crittenden, J. C. (2011). Students with disabilities and online learning: A cross-institutional study of perceived satisfaction with accessibility compliance and services. Internet and Higher Education, 14(4), 242–250.

    Article  Google Scholar 

  58. Rubin, J., & Chisnell, D. (2008). Handbook of usability testing. Indianapolis, IN: Wiley.

    Google Scholar 

  59. Seale, J. K. (2003). The challenge of researching accessibility practices within Higher Education: An exploration of “shared enterprises” or “political games”. Paper presented at the International Education Research Conference AARE and NZARE.

  60. Seeman, L. (2006). Re: Formal objection to WCAG 2.0. [Electronic mailing list message]. Retrieved from http://lists.w3.org/Archives/Public/w3c-wai-gl/2006AprJun/0368.html.

  61. Selvaraj, P. (2004). Comparative study of synchronous remote and traditional in-lab usability evaluation methods. Unpublished Unpublished Master’s thesis, Virginia Polytechnic Institute and State University.

  62. Sloan, D., & Kelly, B. (2008). Reflections on the development of a holistic approach to web accessibility. Paper presented at the 2008 accessible design in the digital world conference (ADDW), York.

  63. Thatcher, J., Burks, M. R., Heilmann, C., Henry, S. L., Kirkpatrick, A., Lauke, P. H., et al. (2006). Web accessibility: Web standards and regulatory compliance. Retrieved from http://library.books24x7.com.ezproxy.library.yorku.ca/assetviewer.aspx?bookid=14675&chunkid=335181448&rowid=1352&noteMenuToggle=0&leftMenuState=1.

  64. Thompson, T., Comden, D., Ferguson, S., Burgstahler, S., & Moore, E. (2013). Seeking predictors of web accessibility in U.S. higher education institutions. Information Technology and Disabilities, 13(1), 18.

    Google Scholar 

  65. U.S. Department of Justice. (2010a). Letter of Resolution, D.J. No. 202-48-213 Princeton University. Retrieved from http://www.ada.gov/princeton.htm.

  66. U.S. Department of Justice. (2010b). Settlement agreement between the United States of America, the National Federation of the Blind, Inc. (“NFB”), and the American Council of the Blind (“ACB”), and the Arizona Board of Regents (“ABOR”), for and on behalf of Arizona State University (“ASU”). Retrieved from http://www.ada.gov/arizona_state_university.htm.

  67. U.S. Department of Justice. (2013, July 23). Justice department settles with Louisiana Tech University over inaccessible course materials [Press release]. Retrieved from http://www.justice.gov/opa/pr/2013/July/13-crt-831.html.

  68. Ustun, T. B., Chatterji, S., Bickenbach, J., Kostanjsek, N., & Schneider, M. (2003). The international classification for functioning, disability and health: A new tool for understanding disability and health. Disability and Rehabilitation, 25(11–12), 565–571.

    Article  Google Scholar 

  69. Vanderheiden, G. (2000). Fundamental principles and priority setting for universal usability. In Proceedings of the 2000 conference on universal usability (pp. 32–38). New York, NY: ACM.

  70. Vigo, M., & Brajnik, G. (2011). Automatic web accessibility metrics: Where we are and where we can go. Interacting with Computers, 23(2), 137–155.

    Article  Google Scholar 

  71. W3C. (2008). Web Content Accessibility Guidelines 2.0: W3C Recommendation 11 December 2008. Retrieved from http://www.w3.org/TR/WCAG20/.

  72. Woodfine, B. P., Nunes, M. B., & Wright, D. J. (2008). Text-based synchronous e-learning and dyslexia: Not necessarily the perfect match! Computers & Education, 50(3), 703–717.

    Article  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Kari L. Kumar.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kumar, K.L., Owston, R. Evaluating e-learning accessibility by automated and student-centered methods. Education Tech Research Dev 64, 263–283 (2016). https://doi.org/10.1007/s11423-015-9413-6

Download citation


  • Accessibility
  • E-learning
  • Educational technology
  • Disabilities
  • Higher education