A checklist for assessing blind users’ usability of educational smartphone applications

  • Yekyung Lee
  • Jihyun LeeEmail author
Long Paper


The purpose of this study is to develop a checklist which specifically evaluates blind users’ usability of educational smartphone applications. To carry out this task, researchers developed checklist items based on the previous usability literature, evaluation tools, and research on e-learning and Web accessibility for users with/without blindness. As a result, a checklist with 29 items covering three levels of interface design (structure, behavior, presentation) was developed. In order to accomplish this, usability principles were first categorized into these three levels and then transformed to become relevant to the blind user. The initial version of the usability checklist items was reviewed and evaluated for their representativeness and comprehensibility by interface design experts and teachers of blind learners. Content validity index (CVI) and Cronbach \(\alpha\) values were calculated to check the validity and reliability of the tool. The revised second version was reviewed in the same way by a group of blind users, and CVI and Cronbach \(\alpha\) values were calculated as well. The final version was implemented by the blind user group for evaluating two learning applications. Reviewers’ comments were reflected in the second and final version as well. Evaluation results indicated low usability for both applications even when accessibility requirements were met.


Usability Accessibility Smartphone application Touchscreen Checklist Evaluation 



The study is part of the ENABLE Network of ICT Supported Learning for Disabled People Project (2011-4437/001-001). The project was funded with support from the Lifelong Learning Programme of the European Commission.


  1. 1.
    American Educational Research Association: American Psychological Association and National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing (US): Standards for educational and psychological testing. Amer Educational Research Assn, Washington, DC (1999)Google Scholar
  2. 2.
    Ardito, C., Costabile, M.F., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., Rossano, V.: An approach to usability evaluation of e-learning applications. Univ. Access Inf. Soc. 4(3), 270–283 (2006)CrossRefGoogle Scholar
  3. 3.
    Babu, R.: Understanding challenges in non-visual interaction with travel sites: an exploratory field study with blind users. First Monday 18(12) (2013)Google Scholar
  4. 4.
    Babu, R., Singh, R., Ganesh, J.: Understanding blind users’ web accessibility and usability problems. AIS Trans. Hum. Comput. Interact. 2(3), 73–94 (2010)Google Scholar
  5. 5.
    Baxley, B.: Universal model of a user interface. In: Proceedings of the 2003 Conference on Designing for User Experiences, pp. 1–14. ACM (2003)Google Scholar
  6. 6.
    Billi, M., Burzagli, L., Catarci, T., Santucci, G., Bertini, E., Gabbanini, F., Palchetti, E.: A unified methodology for the evaluation of accessibility and usability of mobile applications. Univ. Access Inf. Soc. 9(4), 337–356 (2010)CrossRefGoogle Scholar
  7. 7.
    Buzzi, M.C., Buzzi, M., Leporini, B.: Accessing e-learning systems via screen reader: an example. In: International Conference on Human-Computer Interaction, pp. 21–30. Springer (2009)Google Scholar
  8. 8.
    Caldwell, B., Cooper, M., Reid, L.G., Vanderheiden, G.: Web content accessibility guidelines (WCAG) 2.0. WWW Consortium (W3C) (2008)Google Scholar
  9. 9.
    Calvo, R., Iglesias, A., Moreno, L.: Accessibility barriers for users of screen readers in the Moodle learning content management system. Univ. Access Inf. Soc. 13(3), 315–327 (2014)CrossRefGoogle Scholar
  10. 10.
    Di Blas, N., Paolini, P., Speroni, M.: Usable accessibility to the web for blind users. In: Proceedings of 8th ERCIM Workshop: User Interfaces for All, Vienna. Citeseer (2004)Google Scholar
  11. 11.
    Dix, A., Finlay, J., Abowd, G.D., Beale, R.: Human–Computer Interaction, 3rd edn. Pearson Prentice Hall, Upper Saddle River (2003)zbMATHGoogle Scholar
  12. 12.
    Ericsson: Ericsson mobility report 2015. (2016/03/01)
  13. 13.
    Federici, S., Micangeli, A., Ruspantini, I., Borgianni, S., Corradi, F., Pasqualotto, E., Olivetti Belardinelli, M.: Checking an integrated model of web accessibility and usability evaluation for disabled people. Disabil. Rehabil. 27(13), 781–790 (2005)CrossRefGoogle Scholar
  14. 14.
    Gall, M.D., Gall, J.P., Borg, W.R.: Educational Research: An Introduction, 8th edn. Pearson, New York (2006)Google Scholar
  15. 15.
    Harrison, R., Flood, D., Duce, D.: Usability of mobile applications: literature review and rationale for a new usability model. J. Interact. Sci. 1(1), 1–16 (2013)CrossRefGoogle Scholar
  16. 16.
    Inostroza, R., Rusu, C., Roncagliolo, S., Jimenez, C., Rusu, V.: Usability heuristics for touchscreen-based mobile devices. In: 2012 Ninth International Conference on Information Technology: New Generations (ITNG), pp. 662–667. IEEE (2012)Google Scholar
  17. 17.
    International Organization for Standardization: ISO 9241-11: Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs)—Part 11: Guidance on Usability (1998)Google Scholar
  18. 18.
    Kane, S.K., Wobbrock, J.O., Ladner, R.E.: Usable gestures for blind people: understanding preference and performance. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 413–422. ACM (2011)Google Scholar
  19. 19.
    Lazar, J., Allen, A., Kleinman, J., Malarkey, C.: What frustrates screen reader users on the web: a study of 100 blind users. Int. J. Hum. Comput. Interact. 22(3), 247–269 (2007)CrossRefGoogle Scholar
  20. 20.
    Lazar, J., Olalere, A., Wentz, B.: Investigating the accessibility and usability of job application web sites for blind users. J. Usability Stud. 7(2), 68–87 (2012)Google Scholar
  21. 21.
    Leem, J., Lim, B., Sung, E.: Developmental study on the quality certification standards of smart education contents. J. Educ. Inf. Media 20(3), 327–353 (2014)Google Scholar
  22. 22.
    Leporini, B., Buzzi, M.C., Buzzi, M.: Interacting with mobile devices via voiceover: usability and accessibility issues. In: Proceedings of the 24th Australian Computer–Human Interaction Conference, pp. 339–348. ACM (2012)Google Scholar
  23. 23.
    Leporini, B., Paternò, F.: Applying web usability criteria for vision-impaired users: does it really improve task performance? Int. J. Hum. Comput. Interact. 24(1), 17–47 (2008)CrossRefGoogle Scholar
  24. 24.
    McMillan, J.H., Schumacher, S.: Research in Education: A Conceptual Introduction, 5th edn. Addison-Wesley, New York (2001)Google Scholar
  25. 25.
    Mehlenbacher, B., Bennett, L., Bird, T., Ivey, M., Lucas, J., Morton, J., Whitman, L.: Usable e-learning: a conceptual model for evaluation and design. In: Proceedings of HCI International 2005: 11th International Conference on Human-Computer Interaction, vol. 4, pp. 1–10 (2005)Google Scholar
  26. 26.
    Mi, N., Cavuoto, L.A., Benson, K., Smith-Jackson, T., Nussbaum, M.A.: A heuristic checklist for an accessible smartphone interface design. Univ. Access Inf. Soc. 13(4), 351–365 (2014)CrossRefGoogle Scholar
  27. 27.
    Murphy, E., Kuber, R., McAllister, G., Strain, P., Yu, W.: An empirical investigation into the difficulties experienced by visually impaired internet users. Univ. Access Inf. Soc. 7(1–2), 79–91 (2008)CrossRefGoogle Scholar
  28. 28.
    Nielsen, J.: Usability Engineering. Morgan Kaufman, San Diego (1993)zbMATHGoogle Scholar
  29. 29.
    Nokelainen, P.: An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. Educ. Technol. Soc. 9(2), 178–197 (2006)Google Scholar
  30. 30.
    Park, K., Goh, T., So, H.J.: Toward accessible mobile application design: developing mobile application accessibility guidelines for people with visual impairment. In: Proceedings of HCI Korea, pp. 31–38. Hanbit Media, Inc. (2014)Google Scholar
  31. 31.
    Park, K., Kim, H.J., So, H.J.: Are massive open online courses (moocs) really open to everyone?: A study of accessibility evaluation from the perspective of universal design for learning. In: Proceedings of HCI Korea, pp. 29–36. Hanbit Media, Inc. (2016)Google Scholar
  32. 32.
    Petrie, H., Kheir, O.: The relationship between accessibility and usability of websites. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 397–406. ACM (2007)Google Scholar
  33. 33.
    Pew Research Center: U.S. smartphone use in 2015. (2016/02/24)
  34. 34.
    Rubio, D.M., Berg-Weger, M., Tebb, S.S., Lee, E.S., Rauch, S.: Objectifying content validity: conducting a content validity study in social work research. Soc. Work Res. 27(2), 94–104 (2003)CrossRefGoogle Scholar
  35. 35.
    Rzvanolu, K.: Research and Design Innovations for Mobile User Experience, 1st edn. IGI Global, Hershey (2013)Google Scholar
  36. 36.
    Shneiderman, B.: Designing the User Interface. Addison Wesley Longman, Reading (1998)Google Scholar
  37. 37.
    Shneiderman, B., Plaisant, C., Cohen, M., Jacobs, S.: Designing the User Interface: Strategies for Effective Human–Computer Interaction, 5th edn. Addison-Wesley, Reading (2009)Google Scholar
  38. 38.
    World Wide Web Consortium: Mobile Accessibility: How WCAG 2.0 and Other W3C/WAI Guidelines Apply to Mobile. (2015/11/20)

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  1. 1.Sogang University, Graduate School of EducationSeoulKorea
  2. 2.Seoul National University, School of DentistrySeoulKorea

Personalised recommendations