The use of learning technologies is becoming ubiquitous in higher education. As a result, there is a pressing need to develop methods to evaluate their accessibility to ensure that students do not encounter barriers to accessibility while engaging in e-learning. In this study, sample online units were evaluated for accessibility by automated tools and by student participants (in sessions moderated and unmoderated by researchers), and the data from these different methods of e-learning accessibility evaluation were compared. Nearly all students were observed encountering one or more barriers to accessibility while completing the online units, though the automated tools did not predict these barriers and instead predicted potential barriers that were not relevant to the study participants. These data underscore the need to carry out student-centered accessibility evaluation in addition to relying on automated tools and accessibility guideline conformance as measures of accessibility. Students preferred to participate in unmoderated sessions, and the data from the unmoderated sessions were comparable to that from the more traditional moderated sessions. Additional work is needed to further explore methods of student-centered evaluation, including different variations of unmoderated sessions.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Americans with Disabilities Act. (1990, 2008). ADA. Retrieved from http://www.ada.gov/pubs/ada.htm.
Andreason, M. S., Villemann Nielsen, H., Ormholt Schroeder, S., & Stage, J. (2007). What happened to remote usability testing? An empirical study of three methods. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1405–1414). New York, NY: ACM.
AODA Integrated Accessibility Standards. (2011). Retrieved from the Service Canada website http://www.e-laws.gov.on.ca/html/source/regs/english/2011/elaws_src_regs_r11191_e.htm#BK15.
Asuncion, J. V., Fichten, C. S., Ferraro, V., Chwojka, C., Barile, M., Nguyen, M. N., et al. (2010). Multiple perspectives on the accessibility of e-learning in Canadian colleges and universities. Assistive Technology, 22(4), 187–199.
Baravalle, A., & Lanfranchi, V. (2003). Remote web usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 364–368.
Bland, M. (2000). Multiple significance tests and the Bonferroni correction. An introduction to medical statistics (3rd ed.). New York, NY: Oxford Medical Publications.
Bohman, P. R., & Anderson, S. (2005). A conceptual framework for accessibility tools to benefit users with cognitive disabilities. Paper presented at the W4A, Chiba, Japan.
Bolt, N., & Tulathimutte, T. (2010). Remote research. Brooklyn, NY: Rosenfeld Media.
Boren, T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communications, 43(3), 261–278.
Carnevale, D. (2005). Lawsuit charges online university does not accommodate learning-disabled students. The Chronicle of Higher Education, 51(49), 33.
Cassidy, S. (2004). Learning styles: An overview of theories, models, and measures. Educational Psychology, 24(4), 419–444.
Chen, Y. T. (2014). A study to explore the effects of self-regulated learning environment for hearing-impaired students. Journal of Computer Assisted learning, 30(2), 97–109.
Clark, J. (2006, May 23). To hell with the WCAG2. A List Apart, 217. Retrieved from http://www.alistapart.com/articles/tohellwithwcag2.
Denton, W., & Coysh, S. J. (2011). Usability testing of VuFind at an academic library. Library Hi Tech, 29(2), 301–319.
Dermody, K., & Majekodunmi, N. (2010). Online databases and the research experience for university students with print disabilities. Library Hi Tech, 29(1), 149–160.
Ellcessor, E. (2010). Bridging disability divides: A critical history of web content accessibility through 2001. Information, Communication & Society, 13(3), 289–308.
Evett, L., & Brown, D. (2005). Text formats and web design for visually impaired and dyslexic readers—clear text for all. Interacting with Computers, 17(4), 453–472.
Fichten, C. S., Ferraro, V., Asuncion, J. V., Chwojka, C., Barile, M., Nguyen, M. N., et al. (2009). Disabilities and e-learning problems and solutions: An exploratory study. Educational Technology & Society, 12(4), 241–256.
Foley, A. (2011). Exploring the design, development and use of websites through accessibility and usability studies. Journal of Educational Multimedia and Hypermedia, 20(4), 361–385.
Friedman, M. G., & Bryen, D. N. (2007). Web accessibility design recommendations for people with cognitive disabilities. Technology and Disability, 19(4), 205–212.
Gerrard, C. (2007). Virtual learning environments: Enhancing the learning experience for students with disabilities. Campus-Wide Information Systems, 24(3), 199–206.
Goggin, G., & Newell, C. (2003). Digital disability: The social construction of disability in new media. New York, NY: Rowman & Littlefield.
Goodall, B. (2008, January 3). Judge finds university didn’t discriminate online. Retrieved from http://chronicle.com/blogs/wiredcampus/judge-finds-university-didnt-discriminate-online/3582.
Grabinger, S. (2010). A framework for supporting postsecondary learners with psychiatric disabilities in online environments. Electronic Journal of e-Learning, 8(2), 101–110.
Habib, L., Berget, G., Sandnes, F. E., Sanderson, N., Kahn, P., Fagernes, S., et al. (2012). Dyslexic students in higher education and virtual learning environments: an exploratory study. Journal of Computer Assisted learning, 28(6), 574–584.
Hartson, H. R., Castillo, J. C., Kelso, J., & Neale, W. (1996). Remote evaluation: The network as an extension of the usability laboratory. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 228–235). New York, NY: ACM.
Henry, S. L. (2007). Just ask: Integrating accessibility throughout design. Retrieved from http://www.uiaccess.com/JustAsk/index.html.
Her Majesty’s Stationary Office. (2001). The Special Educational Needs and Disability Act 2001, London. Retrieved from http://www.legislation.gov.uk/ukpga/2001/10/contents.
Her Majesty’s Stationary Office. (2010). Equality Act 2010, London. Retrieved from http://www.legislation.gov.uk/ukpga/2010/15/section/92.
Houck-Whitaker, J. (2005). Remote testing versus lab testing. Retrieved from http://www.boltpeters.com/articles/versus.html.
Inside Higher Ed. (2014, February 17). Portland State settles complaint by deaf student. Retrieved from http://www.insidehighered.com/quicktakes/2014/02/17/portland-state-settles-complaint-deaf-student#sthash.YQTpkfRK.dpbs.
Joint Department of Justice and Department of Education. (2011). Frequently asked questions about the June 29, 2010, Dear Colleagues letter. Retrieved from http://www2.ed.gov/about/offices/list/ocr/docs/dcl-ebook-faq-201105.html.
Jung, S., Herlocker, J. L., Webster, J., Mellinger, M., & Frumkin, J. (2008). LibraryFind: System design and usability testing of academic metasearch system. Journal of the American Society for Information Science and Technology, 59(3), 375–389.
Kapsi, M., Vlachogiannis, E., Darzentas, J., & Spyrou, T. (2009a). A preliminary feedback for the WCAG 2.0-WCAG 1.0 Vs WCAG 2.0 evaluation study. Paper presented at the 2nd annual conference on pervasive technologies related to assistive environments (PETRA), Confu, Greece.
Kapsi, M., Vlachogiannis, E., Darzentas, J., & Spyrou, T. (2009b). The usability of web accessibility guidelines: An approach for evaluation. In Proceedings of the 5th international conference on universal access in human computer interaction (pp. 716–724). Berlin: Springer.
Kelly, B., Phipps, L., & Swift, E. (2004). Developing a holistic approach for e-learning accessibility. Canadian Journal of Learning and Technology, 30(3), 20.
Kennedy, H., Evans, S., & Thomas, S. (2011). Can the web be made accessible for people with intellectual disabilities? The Information Society, 27(1), 29–39.
Kumar, K., & Owston, R. (2012). Learnings from post-secondary students on e-learning accessibility and accessibility evaluation methods. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2012 (pp. 663–669). Chesapeake, VA: AACE.
Laff, M., & Rissenberg, M. (2007). Cognitive ability measures for accessible web content. Paper presented at the 4th international conference on universal access in human computer interaction (HCI), Beijing, China.
Marks, D. (1997). Models of disability. Disability and Rehabilitation, 19(3), 85–91.
McCarthy, J. E., & Swierenga, S. J. (2010). What we know about dyslexia and web accessibility: A research review. Universal Access in the Information Society, 9(2), 147–152.
McFadden, E., Hager, D. R., Elie, C. J., & Blackwell, J. M. (2002). Remote usability evaluation: Overview and case studies. International Journal of Human-Computer Interaction, 14(3&4), 489–502.
Narasimhan, N. (Ed.). (2012). Web accessibility policy making: An international perspective. New York: Hemkunt Publishers Ltd.
National Federation of the Blind. (2012, March 6). Florida State resolves litigation with students [Press release]. Retrieved from https://nfb.org/node/931.
National Federation of the Blind. (2014, January 13). Blind student files discrimination suit against Miami University [Press release]. Retrieved from https://nfb.org/blind-student-files-discrimination-suit-against-miami-university.
Nicolle, C., & Paulson, D. (2004). Making the Internet accessible by people with cognitive and communication impairments. Universal Access in the Information Society, 3(1), 48–56.
Nielsen, J. (1993). Usability engineering. Boston, MA: AP Professional.
Oliver, M. (1996). Understanding disability: From theory to practice. Houndmills: Macmillan.
Olmsted-Hawala, E. L., Murphy, E. D., Hawala, S., & Ashenfelter, K. T. (2010a). Think-aloud protocols: A comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 2381–2390). New York, NY: ACM.
Olmsted-Hawala, E. L., Murphy, E. D., Hawala, S., & Ashenfelter, K. T. (2010b). Think-aloud protocols: Analyzing three different think-aloud protocols with counts of verbalized frustrations in a usability study of an information-rich Web site. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 60–66). New York, NY: ACM.
Petrie, H., Hamilton, F., King, N., & Pavan, P. (2006). Remote usability evaluations with disabled people. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1133–1141). New York, NY: ACM.
Power, C., Petrie, H., & Mitchell, R. (2009). A framework for remote user evaluation of accessibility and usability of websites. In C. Stephanidis (Ed.), Universal access in human computer interaction. Addressing diversity (Vol. 5614, pp. 594–601)., Lecture Notes in Computer Science Berlin, Germany: Springer-Verlag.
Power, C., Petrie, H., Sakharov, V., & Swallow, D. (2010). Virtual learning environments: Another barrier to blended and e-learning. In Proceedings of the 12th international conference on computers helping people with special needs: part I (pp. 519–526). Berlin: Springer.
Pretorius, M., & van Biljon, J. (2010). Learning management systems: ICT skills, usability and learnability. Interactive Technology and Smart Education, 7(1), 30–43.
Rehabilitation Act. (1998). Rehabilitation Act of 1973. Retrieved from http://www2.ed.gov/policy/speced/reg/narrative.html.
Ribera, M., Porras, M., Boldu, M., Termens, M., Sule, A., & Paris, P. (2009). Web Content Accessibility Guidelines 2.0: A further step towards accessible digital information. Program: Electronic Library and Information Systems, 43(4), 392–406.
Roberts, J. B., Crittenden, L. A., & Crittenden, J. C. (2011). Students with disabilities and online learning: A cross-institutional study of perceived satisfaction with accessibility compliance and services. Internet and Higher Education, 14(4), 242–250.
Rubin, J., & Chisnell, D. (2008). Handbook of usability testing. Indianapolis, IN: Wiley.
Seale, J. K. (2003). The challenge of researching accessibility practices within Higher Education: An exploration of “shared enterprises” or “political games”. Paper presented at the International Education Research Conference AARE and NZARE.
Seeman, L. (2006). Re: Formal objection to WCAG 2.0. [Electronic mailing list message]. Retrieved from http://lists.w3.org/Archives/Public/w3c-wai-gl/2006AprJun/0368.html.
Selvaraj, P. (2004). Comparative study of synchronous remote and traditional in-lab usability evaluation methods. Unpublished Unpublished Master’s thesis, Virginia Polytechnic Institute and State University.
Sloan, D., & Kelly, B. (2008). Reflections on the development of a holistic approach to web accessibility. Paper presented at the 2008 accessible design in the digital world conference (ADDW), York.
Thatcher, J., Burks, M. R., Heilmann, C., Henry, S. L., Kirkpatrick, A., Lauke, P. H., et al. (2006). Web accessibility: Web standards and regulatory compliance. Retrieved from http://library.books24x7.com.ezproxy.library.yorku.ca/assetviewer.aspx?bookid=14675&chunkid=335181448&rowid=1352¬eMenuToggle=0&leftMenuState=1.
Thompson, T., Comden, D., Ferguson, S., Burgstahler, S., & Moore, E. (2013). Seeking predictors of web accessibility in U.S. higher education institutions. Information Technology and Disabilities, 13(1), 18.
U.S. Department of Justice. (2010a). Letter of Resolution, D.J. No. 202-48-213 Princeton University. Retrieved from http://www.ada.gov/princeton.htm.
U.S. Department of Justice. (2010b). Settlement agreement between the United States of America, the National Federation of the Blind, Inc. (“NFB”), and the American Council of the Blind (“ACB”), and the Arizona Board of Regents (“ABOR”), for and on behalf of Arizona State University (“ASU”). Retrieved from http://www.ada.gov/arizona_state_university.htm.
U.S. Department of Justice. (2013, July 23). Justice department settles with Louisiana Tech University over inaccessible course materials [Press release]. Retrieved from http://www.justice.gov/opa/pr/2013/July/13-crt-831.html.
Ustun, T. B., Chatterji, S., Bickenbach, J., Kostanjsek, N., & Schneider, M. (2003). The international classification for functioning, disability and health: A new tool for understanding disability and health. Disability and Rehabilitation, 25(11–12), 565–571.
Vanderheiden, G. (2000). Fundamental principles and priority setting for universal usability. In Proceedings of the 2000 conference on universal usability (pp. 32–38). New York, NY: ACM.
Vigo, M., & Brajnik, G. (2011). Automatic web accessibility metrics: Where we are and where we can go. Interacting with Computers, 23(2), 137–155.
W3C. (2008). Web Content Accessibility Guidelines 2.0: W3C Recommendation 11 December 2008. Retrieved from http://www.w3.org/TR/WCAG20/.
Woodfine, B. P., Nunes, M. B., & Wright, D. J. (2008). Text-based synchronous e-learning and dyslexia: Not necessarily the perfect match! Computers & Education, 50(3), 703–717.
About this article
Cite this article
Kumar, K.L., Owston, R. Evaluating e-learning accessibility by automated and student-centered methods. Education Tech Research Dev 64, 263–283 (2016). https://doi.org/10.1007/s11423-015-9413-6
- Educational technology
- Higher education