Skip to main content
SpringerLink
Log in

Comparative Analysis of Accessibility Testing Tools and Their Limitations in RIAs

  • Conference paper
  • First Online:
HCI International 2022 – Late Breaking Papers: HCI for Health, Well-being, Universal Access and Healthy Aging (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13521))

Included in the following conference series:

Abstract

Accessibility is a required quality for websites today, and several tools exist to test for this quality. These tools are highly advantageous, but sadly they also have some limitations. A particular set of challenges they face is in the evaluation of Rich Internet Applications (RIAs). In this paper, we carry out an experiment to compare and analyze different accessibility testing tools as they evaluate 10 educational websites. We judged these tools based on their error detection, guideline coverage, speed, similarity to one another, and their relative performance when evaluating RIAs. The experiment findings revealed the strength and limitations of each tool. The results of this experiment also exposed that there are many guidelines and success criteria that accessibility testing tools are not able to cover, and that some evaluation tools are similar to each other in terms of the results they produce. Lastly, this experiment highlights a discrepancy in the behavior of the tools when evaluating RIAs compared to when evaluating static websites, although some more than others. This experiment has some limitations which we presented. As a future work, we intend to work with an expert to determine the accuracy of the results produced from the experiment. We also intend to delve deeper into the limitations of these tools and come up with possible solutions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.w3.org/WAI/standards-guidelines/wcag/.

  2. 2.

    https://www.section508.gov/.

  3. 3.

    https://www.accessi.org/bitv.

  4. 4.

    https://www.access-board.gov/ada/.

  5. 5.

    https://www.w3.org/.

  6. 6.

    https://www.w3.org/TR/WCAG20/.

  7. 7.

    https://www.w3.org/TR/WCAG21/.

  8. 8.

    https://achecker.achecks.ca/checker/index.php.

  9. 9.

    https://fae.disability.illinois.edu/.

  10. 10.

    https://www.tawdis.net/resumen.

  11. 11.

    https://wave.webaim.org/.

References

  1. Al-Ahmad, A., Ahmaro, I.Y., Mustafa, M.: Comparison between web accessibility evaluation tools, January 2013. https://www.researchgate.net/publication/279181298

  2. Alsaeedi, A.: Comparing web accessibility evaluation tools and evaluating the accessibility of webpages: proposed frameworks. Information 11(1), 40 (2020)

    Article  Google Scholar 

  3. Baazeem, I.S., Al-Khalifa, H.S.: Advancements in web accessibility evaluation methods: how far are we? In: Proceedings of the 17th International Conference on Information Integration and Web-based Applications & Services. iiWAS 2015, 11–13 December 2015. ACM Press, New York. https://doi.org/10.1145/2837185.2843850

  4. Brajnik, G., Lomuscio, R.: Samba: a semi-automatic method for measuring barriers of accessibility. In: Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility, Assets 2007, October 15–17 2007, pp. 43–50. ACM Press, New York. https://doi.org/10.1145/1296843.1296853

  5. Campoverde-Molina, M., Lujan-Mora, S., Garcia, L.V.: Empirical studies on web accessibility of educational websites: a systematic literature review. IEEE Access 8, 91676–91700 (2020)

    Article  Google Scholar 

  6. Fortes, R.P., Antonelli, H.L., de Lima Salgado, A.: Accessibility and usability evaluation of rich internet applications. In: Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web, Webmedia 2016, 08–11 November 2016, pp. 7–8. ACM Press, New York. https://doi.org/10.1145/2976796.2988221

  7. Human Network Contributor: Websites fail to comply with accessibility for people with disabilities. November 2020. https://isemag.com/2020/11/telecom-98-percent-of-websites-fail-to-comply-with-accessibility-requirements-for-people-with-disabilities/

  8. Investis Digital: Accessibility testing - manual or automated? January 2020. https://www.investisdigital.com/blog/web-design-and-development/accessibility-testing-manual-or-automated. Accessed 23 Oct 2021

  9. Ismail, A., Kuppusamy, K.: Web accessibility investigation and identification of major issues of higher education websites with statistical measures: a case study of college websites. J. King Saud Univ. Comput. Inf. Sci. (2019)

    Google Scholar 

  10. Kumar, A.: 5 free must have web accessibility testing tools - leader in offshore accessibility testing — section 508 compliance — wcag conformance — barrierbreak, October 2021. https://www.barrierbreak.com/5-free-must-have-web-accessibility-testing-tools/

  11. Law Office of Lainey Feingold: Digital accessibility laws around the globe, May 2013. https://www.lflegal.com/2013/05/gaad-legal/. Accessed 23 Oct 2021

  12. Linaje, M., Lozano-Tello, A., Perez-Toledano, M.A., Preciado, J.C., Rodriguez-Echeverria, R., Sanchez-Figueroa, F.: Providing ria user interfaces with accessibility properties. J. Symb. Comput. 46, 207–217 (2011). https://doi.org/10.1016/j.jsc.2010.08.008

    Article  MATH  Google Scholar 

  13. Lopes, R., Gomes, D., Carriço, L.: Web not for all: A large scale study of web accessibility. In: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A). W4A 2010. Association for Computing Machinery, New York (2010). https://doi.org/10.1145/1805986.1806001. https://doi.org/10.1145/1805986.1806001

  14. Mankoff, J., Fait, H., Tran, T.: Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind. In: In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2005, pp. 41–50. ACM Press, New York (2005)

    Google Scholar 

  15. Misfud, J.: 8 free web-based website accessibility evaluation tools - usability geek. https://usabilitygeek.com/10-free-web-based-web-site-accessibility-evaluation-tools/

  16. Mucha, J.M.: Combination of automatic and manual testing for web accessibility. Master’s thesis, Grimstad, Norway (2018). Accessed 05 Aug 2019

    Google Scholar 

  17. Neova Tech Solutions: 7 web accessibility testing tools you should know, April 2020. https://www.neovasolutions.com/2020/04/08/7-web-accessibility-testing-tools-you-should-know/. Accessed 03 Mar 2022

  18. Smith, S.: Website accessibility standards should be higher to help disabled people - vox, May 2019. https://www.vox.com/the-goods/2019/2/5/18210912/websites-ada-compliance-lawsuits. Accessed 04 Mar 2022

  19. Software Testing Help: Top 20 accessibility testing tools for web applications, Feb 2022. https://www.softwaretestinghelp.com/accessibility-testing-tools/. Accessed 03 Mar 2022

  20. Timbi-Sisalima, C., Amor, C.I.M., Tortosa, S.O., Hilera, J.R., Aguado-Delgado, J.: Comparative analysis of online web accessibility evaluation tools. In: Proceedings of the 25th International Conference on Information Systems Development, ISD 2016, pp. 562–573. University of Economics, ACM Press, New York (2016)

    Google Scholar 

  21. Usablenet: Quick guide to manual accessibility testing and why it’s important, June 2018. https://blog.usablenet.com/quick-guide-to-manual-accessibility-testing-and-why-its-important. Accessed 02 Mar 2022

  22. Usablenet: Automated wcag testing is not enough for web accessibility ada compliance [blog], July 2020. https://blog.usablenet.com/automated-wcag-testing-is-not-enough-for-web-accessibility-ada-compliance. Accessed 02 Mar 2022

  23. Vigo, M., Brajnik, G.: Automatic web accessibility metrics: where we are and where we can go. Interact. Comput. 23, 137–155 (2011). https://doi.org/10.1016/j.intcom.2011.01.001

    Article  Google Scholar 

  24. Vigo, M., Brown, J., Conway, V.: Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. In: Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, w4A 2013, pp. 1–10. ACM Press, New York (2013). https://doi.org/10.1145/2461121.2461124

  25. W3C: Web accessibility evaluation tools list. https://www.w3.org/WAI/ER/tools/

  26. W3C: Web content accessibility guidelines (wcag) 2.0, December 2008. https://www.w3.org/TR/WCAG20/. Accessed 04 Mar 2022

  27. W3C: Web content accessibility guidelines (wcag) 2.1, June 2018. https://www.w3.org/TR/WCAG21/. Accessed 04 2022

  28. W3C Web Accessibility Initiative (WAI): Wcag 2 overview, July 2005. https://www.w3.org/WAI/standards-guidelines/wcag/. Accessed 04 Mar 2022

  29. Watanabe, W.M., Fortes, R.P.M., Dias, A.L.: Using acceptance tests to validate accessibility requirements in ria. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A 2012, 16–17 April 2012. ACM Press, New York. https://doi.org/10.1145/2207016.2207022

  30. World Wide Web Consortium: Wai-aria, January 2016. https://www.w3.org/WAI/standards-guidelines/aria/. Accessed 24 Aug 2019

  31. World Wide Web Consortium: Accessible rich internet applications (wai-aria) 1.1. https://www.w3.org/TR/wai-aria/. Accessed 05 Sept 2019

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of North Texas, Denton, TX, USA

    Obianuju Okafor, Wajdi Aljedaani & Stephanie Ludi

Authors
  1. Obianuju Okafor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wajdi Aljedaani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephanie Ludi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Obianuju Okafor .

Editor information

Editors and Affiliations

  1. Purdue University West Lafayette, West Lafayette, IN, USA

    Vincent G. Duffy

  2. Tsinghua University, Beijing, China

    Qin Gao

  3. Chongqing University, Chongqing, China

    Jia Zhou

  4. Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Margherita Antona

  5. University of Crete and Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Okafor, O., Aljedaani, W., Ludi, S. (2022). Comparative Analysis of Accessibility Testing Tools and Their Limitations in RIAs. In: Duffy, V.G., Gao, Q., Zhou, J., Antona, M., Stephanidis, C. (eds) HCI International 2022 – Late Breaking Papers: HCI for Health, Well-being, Universal Access and Healthy Aging. HCII 2022. Lecture Notes in Computer Science, vol 13521. Springer, Cham. https://doi.org/10.1007/978-3-031-17902-0_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-17902-0_34

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-17901-3

  • Online ISBN: 978-3-031-17902-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation