Abstract
This paper presents a systematic literature review characterizing the methodological properties of usability studies conducted on educational and learning technologies in the past 20 years. PRISMA guidelines were followed to identify, select, and review relevant research and report results. Our rigorous review focused on (1) categories of educational and learning technologies that have been the focus of usability evaluation, (2) specific usability evaluation methods used, (3) outcome measures, and (4) research limitations. Findings revealed a diverse range of usability evaluation methods employed for different types of educational/learning technologies and the contexts in which those methods were used, with the majority of usability studies being performed on e-learning technologies within higher education contexts. Specific methods, instrumentation, and types of usability research found to be dominant in reviewed studies were further analyzed and classified, with findings suggesting inquiry methods using questionnaires were most prevalent. Prevalent outcome measures were also synthesized, with findings suggesting that the majority of usability research focuses on issues of technological usability, with very few studies considering pedagogical and socio-cultural aspects of usability. A number of limitations were found, including conceptual and procedural flaws, fundamental misunderstanding of usability evaluation methods, and inappropriate application of usability methods, suggesting potentially problematic and unreliable results. These findings are discussed in-depth, and implications for future research are provided.
Similar content being viewed by others
References
Abulfaraj, A., & Steele, A. (2020). Coherent heuristic evaluation (CoHE): toward increasing the effectiveness of heuristic evaluation for novice evaluators. In A. Marcus & E. Rosenzweig (Eds.), International Conference on Human-Computer Interaction (pp. 3–20). Springer.
Ackerman, R., Parush, A., Nassar, F., & Shtub, A. (2016). Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing. Computers in Human Behavior, 54, 101–113. https://doi.org/10.1016/j.chb.2015.07.041
Alaswad, Z., & Nadolny, L. (2015). Designing for Game-Based Learning: The Effective Integration of Technology to Support Learning. Journal of Educational Technology Systems, 43(4), 389–402. https://doi.org/10.1177/0047239515588164
Albert, B., & Tullis, T. (2013). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Newnes.
Ally, M., & Prieto-Blázquez, J. (2014). What is the future of mobile learning in education? International Journal of Educational Technology in Higher Education, 11(1), 142–151. https://doi.org/10.7238/rusc.v11i1.2033
Alroobaea, R. S., Al-Badi, A. H., & Mayhew, P. J. (2013). AlRoobaea, R., Al-Badi, A. H., & Mayhew, P. J. (2013, March). A framework for generating domain-specific heuristics for evaluating online educational websites-Further validation. In 2nd International Conference on Human Computer Interaction and Learning Technology (ICHCILT 2013), Abu Dhabi, UAE.
Ardito, C., Costabile, M. F., Marsico, M. D., Lanzilotti, R., Levialdi, S., Roselli, T., & Rossano, V. (2006). An approach to usability evaluation of e-learning applications. Universal Access in the Information Society, 4(3), 270–283. https://doi.org/10.1007/s10209-005-0008-6
Arruda Filho, E. J. M., Nogueira, A. C. L., & da Costa, E. M. S. (2021). Social Influence effect on consumers’ intention to adopt mobile banking services. Information Systems Management. https://doi.org/10.1080/10580530.2021.1965678
Babatunde, F. O., MacDermid, J., Grewal, R., Macedo, L., & Szekeres, M. (2020). Development and Usability Testing of a Web-Based and Therapist-Assisted Coping Skills Program for Managing Psychosocial Problems in Individuals With Hand and Upper Limb Injuries: Mixed Methods Study. JMIR Human Factors, 7(2), e17088. https://doi.org/10.2196/17088
Barbosa, D., Schneider, G., Mossmann, J., & Santos, P. D. (2020). Usability and gameplay evaluation on mobile games: A user-centred application proposal. International Journal of Learning Technology, 15(2), 107–129. https://doi.org/10.1504/IJLT.2020.109570
Barrie, M., Socha, J. J., Mansour, L., & Patterson, E. S. (2019). Mixed reality in medical education: a narrative literature review. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care. https://doi.org/10.1177/2327857919081006
Batra, S., & Bishu, R. R. (2007). Web usability and evaluation: issues and concerns. In N. Aykin (Ed.), International Conference on Usability and Internationalization (pp. 243–249). Springer.
Benson, L., Elliott, D., Grant, M., Holschuh, D., Kim, B., Kim, H., Lauber, E., Loh, S., & Reeves, T. C. (2002). Usability and instructional design heuristics for e-learning evaluation (pp. 1615–1621). Association for the Advancement of Computing in Education (AACE).
Bernard, L., Raina, S., Taylor, B., & Kaza, S. (2021). Minimizing Cognitive Overload in Cybersecurity Learning Materials: An Experimental Study Using Eye-Tracking. IFIP World Conference on Information Security Education (pp. 47–63). Springer.
Bernhaupt, R. (2010). User experience evaluation in entertainment. In R. Bernhaupt (Ed.), Evaluating user experience in games (pp. 3–7). Springer.
Bernsen, N. O., & Dybkjær, L. (2009). Multimodal usability. Springer Science & Business Media. https://doi.org/10.1007/978-1-84882-553-6
Bertolino, A., Braione, P., Angelis, G. D., Gazzola, L., Kifetew, F., Mariani, L.,…Tonella, P. (2021). A Survey of Field-based Testing Techniques. ACM Computing Surveys (CSUR), 54(5), 1–39.
Bourges-Waldegg, P., Moreno, L., & Rojano, T. (2000). The role of usability on the implementation and evaluation of educational technology. In Proceedings of the 33rd Annual Hawaii International Conference on System Sciences (pp. 7-pp). IEEE. https://doi.org/10.1109/HICSS.2000.926722
Bowman, D. A., Gabbard, J. L., & Hix, D. (2002). A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods. Presence: Teleoperators and Virtual Environments, 11(4), 404–424. https://doi.org/10.1162/105474602760204309
Buzhardt, J., Abbott, M., Greenwood, C., & Tapia, Y. (2004). Usability Testing of the ClassWide Peer Tutoring-Learning Management System. Journal of Special Education Technology, 20(1), 19–29. https://doi.org/10.1177/016264340502000102
Calhoun, C., Sahay, S., & Wilson, M. (2021). Instructional Design Evaluation. Design for Learning. https://edtechbooks.org/id/instructional_design_evaluation
Cassino, R., & Tucci, M. (2011). Developing usable web interfaces with the aid of automatic verification of their formal specification. Journal of Visual Languages & Computing, 22(2), 140–149. https://doi.org/10.1016/j.jvlc.2010.12.001
Cayola, L., & Macías, J. A. (2018). Systematic guidance on usability methods in user-centered software development. Information and Software Technology, 97, 163–175.
Chang, V. (2016). Review and discussion: E-learning for academia and industry. International Journal of Information Management, 36(3), 476–485. https://doi.org/10.1016/j.ijinfomgt.2015.12.007
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Conn, V. S., Valentine, J. C., Cooper, H. M., & Rantz, M. J. (2003). Grey Literature in Meta-Analyses. Nursing Research, 52(4), 256–261. https://doi.org/10.1097/00006199-200307000-00008
Connan, V., Marcon, M. A., Mahmud, F. H., Assor, E., Martincevic, I., Bandsma, R. H., Vresk, L., & Walsh, C. M. (2019). Online education for gluten-free diet teaching: Development and usability testing of an e-learning module for children with concurrent celiac disease and type 1 diabetes. Pediatric Diabetes, 20(3), 293–303. https://doi.org/10.1111/pedi.12815
Costabile, M. F., De Marsico, M., Lanzilotti, R., Plantamura, V. L., & Roselli, T. (2005). On the usability evaluation of e-learning applications. In Proceedings of the 38th Annual Hawaii International Conference on System Sciences (pp. 6b-6b). IEEE. https://doi.org/10.1109/HICSS.2005.468
Coursaris, C. K., & Kim, D. J. (2011). A meta-analytical review of empirical mobile usability studies. Journal of Usability Studies, 6(3), 117–171.
Criollo-C, S., Guerrero-Arias, A., Jaramillo-Alcázar, Á., & Luján-Mora, S. (2021). Mobile learning technologies for education: benefits and pending issues. Applied Sciences, 11(9), 4111. https://doi.org/10.3390/app11094111
Crowther, M. S., Keller, C. C., & Waddoups, G. L. (2004). Improving the quality and effectiveness of computer-mediated instruction through usability evaluations. British Journal of Educational Technology, 35(3), 289–303. https://doi.org/10.1111/j.0007-1013.2004.00390.x
Dabbagh, N., Fake, H., & Zhang, Z. (2019). Student Perspectives of Technology use for Learning in Higher Education. Revista Iberoamericana De Educación a Distancia, 22(1), 127. https://doi.org/10.5944/ried.22.1.22102
Darren, B., & Thom, N. (2015). New tools and techniques for understanding non-conscious consumer decisions. Applied Marketing Analytics, 1(3), 214–220.
Davids, M. R., Chikte, U. M. E., & Halperin, M. L. (2013). An efficient approach to improve the usability of e-learning resources: The role of heuristic evaluation. Advances in Physiology Education, 37(3), 242–248. https://doi.org/10.1152/advan.00043.2013
Davis, R., Gardner, J., & Schnall, R. (2020). A review of usability evaluation methods and their use for testing eHealth HIV interventions. Current HIV/AIDS Reports, 17(3), 203–218. https://doi.org/10.1007/s11904-020-00493-3
de Abreu Cybis, W., Betiol, A. H., & Faust, R. (2015). Ergonomia e Usabilidade 3ª edição: Conhecimentos, Métodos e Aplicações. Novatec Editora.
de Lima Salgado, A., & Freire, A. P. (2014). Lima Salgado, A. D., & Freire, A. P. (2014, June). Heuristic evaluation of mobile usability: A mapping study. In: International Conference on Human-Computer Interaction (pp. 178–188). Springer, Cham. https://doi.org/10.1007/978-3-319-07227-2_18
Derouin, R. E., Fritzsche, B. A., & Salas, E. (2005). E-Learning in Organizations. Journal of Management, 31(6), 920–940. https://doi.org/10.1177/0149206305279815
Dingli, A., & Cassar, S. (2014). An intelligent framework for website usability. Advances in Human-Computer Interaction. https://doi.org/10.1155/2014/479286
Falkowska, J., Sobecki, J., & Pietrzak, M. (2016). Eye tracking usability testing enhanced with EEG analysis. International Conference of Design, User Experience, and Usability (pp. 399–411). Cham: Springer.
Fernandez, A., Insfran, E., & Abrahão, S. (2011). Usability evaluation methods for the web: A systematic mapping study. Information and Software Technology, 53(8), 789–817. https://doi.org/10.1016/j.infsof.2011.02.007
Flagg, B. N. (2013). Formative Evaluation for Educational Technologies. Routledge.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378.
Fleiss, J. L., Levin, B., & Paik, M. C. (2013). Statistical methods for rates and proportions. John Wiley & Sons.
Frøkjær, E., Hertzum, M., & Hornbæk, K. (2000). Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 345–352). https://doi.org/10.1145/332040.332455
Gladman, T., Tylee, G., Gallagher, S., Mair, J., Rennie, S. C., & Grainger, R. (2020). A tool for rating the value of health education mobile apps to enhance student learning (MARuL): Development and usability study. JMIR MHealth and UHealth, 8(7), e18015. https://doi.org/10.2196/18015
Goldberg, J., & Wichansky, A. (2003). Eye tracking in usability evaluation: A practitioner’s guide. The Mind’s Eye (pp. 493–516). Elsevier.
Grant, M. M. (2019). Difficulties in defining mobile learning: Analysis, design characteristics, and implications. Educational Technology Research and Development, 67(2), 361–388. https://doi.org/10.1007/s11423-018-09641-4
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015). The role of google scholar in evidence reviews and its applicability to grey literature searching. PLoS ONE, 10(9), e0138237. https://doi.org/10.1371/journal.pone.0138237
Hadjerrouit, S. (2012). Investigating technical and pedagogical usability issues of collaborative learning with Wikis. Informatics in Education, 11(1), 45–64.
Haq, R., Li, B., Jovicic, A., Dastur, D., Trinkaus, M., & Kong, A. (2018). Web-based oncology educational tool for medical trainees on oncology rotation—results of a pilot study. Journal of Cancer Education, 33(4), 788–797. https://doi.org/10.1007/s13187-016-1151-x
Hasan, L., Morris, A., & Probets, S. (2013). E-commerce websites for developing countries – a usability evaluation framework. Online Information Review, 37(2), 231–251. https://doi.org/10.1108/OIR-10-2011-0166
Hinkle, D., Wiersma, W., & Jurs, S. (2003). Applied statistics for the behavioral sciences. Houghton Mifflin.
Hornbæk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79–102. https://doi.org/10.1016/j.ijhcs.2005.06.002
Huizenga, J. C., ten Dam, G. T. M., Voogt, J. M., & Admiraal, W. F. (2017). Teacher perceptions of the value of game-based learning in secondary education. Computers & Education, 110, 105–115. https://doi.org/10.1016/j.compedu.2017.03.008
Ingrassia, P. L., Mormando, G., Giudici, E., Strada, F., Carfagna, F., Lamberti, F., & Bottino, A. (2020). Augmented reality learning environment for basic life support and defibrillation training: Usability study. Journal of Medical Internet Research, 22(5), e14910. https://doi.org/10.2196/14910
Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
ISO. (2011). ISO/IEC 25010:2011(en), Systems and software engineering—Systems and software Quality Requirements and Evaluation (SQuaRE)—System and software quality models. (n.d.). Retrieved September 9, 2021, from https://www.iso.org/obp/ui/#iso:std:iso-iec:25010:ed-1:v1:en:sec:4.4
ISO. (2019). ISO 9241–210:2019(en), Ergonomics of human-system interaction—Part 210: Human-centred design for interactive systems. Retrieved September 9, 2021, from https://www.iso.org/obp/ui/#iso:std:iso:9241:-210:ed-2:v1:en
Issa, T., & Isaias, P. (2015). Usability and human computer interaction (HCI). In T. Issa & P. Isaias (Eds.), Sustainable design (pp. 19–36). London: Springer.
Ivory, M. Y., & Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys, 33(4), 470–516. https://doi.org/10.1145/503112.503114
Jackson, J. L., & Kuriyama, A. (2019). How often do systematic reviews exclude articles not published in english? Journal of General Internal Medicine, 34(8), 1388–1389. https://doi.org/10.1007/s11606-019-04976-x
Jahnke, I., Schmidt, M., Pham, M., & Singh, K. (2020). Sociotechnical-Pedagogical Usability for Designing and Evaluating Learner Experience in Technology-Enhanced Environments. Learner and User Experience Research.
Jeffries, R., & Desurvire, H. (1992). Usability testing vs heuristic evaluation: was there a contest? ACM SIGCHI Bulletin, 24(4), 39–41.
Kahiigi, E. K., Ekenberg, L., Hansson, H., Danielson, F. F. T., & Danielson, M. (2008). Exploring the e-Learning State of Art. Electronic Journal of E-Learning, 6(2), 149–160.
Kavitha, V., & Lohani, R. (2019). A critical study on the use of artificial intelligence, e-Learning technology and tools to enhance the learners experience. Cluster Comput, 22, 6985–6989. https://doi.org/10.1007/s10586-018-2017-2
Kearney, M., Schuck, S., Burden, K., & Aubusson, P. (2012). Viewing mobile learning from a pedagogical perspective. Research in Learning Technology. https://doi.org/10.3402/rlt.v20i0.14406
Keegan, D., & Ireland, D. (2005). The incorporation of mobile learning into mainstream education and training. In World Conference on Mobile Learning, Cape Town (Vol. 11, pp. 1–17).
Khajouei, R., Ameri, A., & Jahani, Y. (2018). Evaluating the agreement of users with usability problems identified by heuristic evaluation. International Journal of Medical Informatics, 117, 13–18. https://doi.org/10.1016/j.ijmedinf.2018.05.012
Khajouei, R., & Farahani, F. (2020). A combination of two methods for evaluating the usability of a hospital information system. BMC Medical Informatics and Decision Making, 20(1), 84. https://doi.org/10.1186/s12911-020-1083-6
Khanum, M. A., & Trivedi, M. C. (2013). Exploring verbalization and collaboration during usability evaluation with children in context. arXiv preprint arXiv:1306.4094.
King, L. A. (2009). Visual navigation patterns and cognitive load. In International Conference on Foundations of Augmented Cognition (pp. 254–259). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02812-0_30
Kirkwood, A., & Price, L. (2014). Technology-enhanced learning and teaching in higher education: What is ‘enhanced’ and how do we know? A critical literature review. Learning, Media and Technology, 39(1), 6–36. https://doi.org/10.1080/17439884.2013.770404
Koszalka, T. A., Russ-Eft, D. F., & Reiser, R. (2013). Instructional Designer Competencies: The Standards (Fourth Edition). IAP.
Kraemer, H. C., Kupfer, D. J., Clarke, D. E., Narrow, W. E., & Regier, D. A. (2012). DSM-5: How reliable is reliable enough? American Journal of Psychiatry, 169(1), 13–15.
Krug, S. (2009). Rocket surgery made easy: The do-it-yourself guide to finding and fixing usability problems. New Riders.
Kumar, B. A., & Goundar, M. S. (2019). Usability heuristics for mobile learning applications. Education and Information Technologies, 24(2), 1819–1833. https://doi.org/10.1007/s10639-019-09860-z
Kumar, B. A., Goundar, M. S., & Chand, S. S. (2020). A framework for heuristic evaluation of mobile learning applications. Education and Information Technologies, 25(4), 3189–3204. https://doi.org/10.1007/s10639-020-10112-8
Kumar, B. A., & Mohite, P. (2018). Usability of mobile learning applications: A systematic literature review. Journal of Computers in Education, 5(1), 1–17. https://doi.org/10.1007/s40692-017-0093-6
Lai, J. W. M., & Bower, M. (2019). How is the use of technology in education evaluated? A systematic review. Computers & Education, 133, 27–42. https://doi.org/10.1016/j.compedu.2019.01.010
Lam, J., Yau, J., & Cheung, S. K. S. (2010). A review of mobile learning in the mobile age. In International Conference on Hybrid Learning (pp. 306–315). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-14657-2_28
Lau, R. W. H., Yen, N. Y., Li, F., & Wah, B. (2014). Recent development in multimedia e-learning technologies. World Wide Web, 17(2), 189–198. https://doi.org/10.1007/s11280-013-0206-8
Law, E.L.-C., & Abrahão, S. (2014). Interplay between User Experience (UX) evaluation and system development. International Journal of Human-Computer Studies, 72(6), 523–525. https://doi.org/10.1016/j.ijhcs.2014.03.003
Law, E.L.-C., & Sun, X. (2012). Evaluating user experience of adaptive digital educational games with Activity Theory. International Journal of Human-Computer Studies, 70(7), 478–497. https://doi.org/10.1016/j.ijhcs.2012.01.007
Lee, M. J. W., Georgieva, M., Alexander, B., Craig, E., & Richter, J. (2021). State of XR & Immersive Learning Outlook Report 2021. Walnut, CA: Immersive Learning Research Network.
Leow, M. C., Wang, L. Y. K., Lau, S. H., & Tan, C. K. (2016). Usability of RPG-based Learning Framework. International Journal of Human-Computer Interaction, 32(8), 643–653. https://doi.org/10.1080/10447318.2016.1183863
Lim, C. J., & Lee, S. (2007). Pedagogical usability checklist for ESL/EFL e-learning websites. Journal of Convergence and Information Technology, 2(3), 67–76.
Liu, D., Bhagat, K. K., Gao, Y., Chang, T.-W., & Huang, R. (2017). The potentials and trends of virtual reality in education. In Virtual, augmented, and mixed realities in education (pp. 105–130). Springer, Singapore. https://doi.org/10.1007/978-981-10-5490-7_7
Lyons, L., & Mallavarapu, A. (2021). Collective Usability: Using Simulation Tools to Explore Embodied Design Challenges in Immersive, Shared Mixed-Reality Experiences. Educational Technology & Society, 24(2), 120–135. https://www.jstor.org/stable/27004936
Lyzara, R., Purwandari, B., & Zulfikar, M. F. (2019). E-government usability evaluation: Insights from a systematic literature review. In Proceedings of the 2nd International Conference on Software Engineering and Information Management (pp. 249–253).
Machado Neto, O., & Pimentel, M. da G. (2013). Heuristics for the assessment of interfaces of mobile devices. In Proceedings of the 19th Brazilian Symposium on Multimedia and the Web. https://doi.org/10.1145/2526188.2526237
Machado, A., Oliveira, A., Jácome, C., Pereira, M., Moreira, J., Rodrigues, J., Aparício, J., Jesus, L. M. T., & Marques, A. (2018). Usability of Computerized Lung Auscultation-Sound Software (CLASS) for learning pulmonary auscultation. Medical & Biological Engineering & Computing, 56(4), 623–633. https://doi.org/10.1007/s11517-017-1697-8
Mankoff, J., Dey, A. K., Hsieh, G., Kientz, J., Lederer, S., & Ames, M. (2003). Heuristic evaluation of ambient displays. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 169–176). Association for Computing Machinery, New York. https://doi.org/10.1145/642611.642642
Maramba, I., Chatterjee, A., & Newman, C. (2019). Methods of usability testing in the development of eHealth applications: A scoping review. International Journal of Medical Informatics, 126, 95–104. https://doi.org/10.1016/j.ijmedinf.2019.03.018
Marell-Olsson, E., & Jahnke, I. (2019). Wearable technology in a dentistry study program: Potential and challenges of smart glasses for learning at the workplace. In I. Buchem, R. Klamma, & F. Wild (Eds.), Perspectives on wearable enhanced learning (WELL) (pp. 433–451). Springer.
Martin, F., & Ertzberger, J. (2013). Here and now mobile learning: An experimental study on the use of mobile technology. Computers & Education, 68, 76–85. https://doi.org/10.1016/j.compedu.2013.04.021
Martin, S., Diaz, G., Sancristobal, E., Gil, R., Castro, M., & Peire, J. (2011). New technology trends in education: Seven years of forecasts and convergence. Computers & Education, 57(3), 1893–1906. https://doi.org/10.1016/j.compedu.2011.04.003
Martín-Gutiérrez, J., Mora, C. E., Añorbe-Díaz, B., & González-Marrero, A. (2017). Virtual technologies trends in education. Eurasia Journal of Mathematics, Science and Technology Education, 13(2), 469–486.
McDonald, J. K., & Yanchar, S. C. (2020). Towards a view of originary theory in instructional design. Educational Technology Research and Development, 68(2), 633–651. https://doi.org/10.1007/s11423-019-09734-8
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282.
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., Group, T. P. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLOS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097
Moore, J. L., Dickson-Deane, C., & Liu, M. Z. (2014). Designing CMS courses from a pedagogical usability perspective. In A. Benson & D. Whitworth (Eds.), Perspectives in instructional technology and distance education: Research on course management systems in higher education (pp. 143–169). Charlotte, NC: Information Age.
Moore, J. L., Dickson-Deane, C., & Galyen, K. (2011). e-Learning, online learning, and distance learning environments: Are they the same? The Internet and Higher Education, 14(2), 129–135.
Moradian, S., Krzyzanowska, M. K., Maguire, R., Morita, P. P., Kukreti, V., Avery, J., Liu, G., Cafazzo, J., & Howell, D. (2018). Usability evaluation of a mobile phone-based system for remote monitoring and management of chemotherapy-related side effects in cancer patients: Mixed-methods study. JMIR Cancer, 4(2), e10932. https://doi.org/10.2196/10932
Moubayed, A., Injadat, M., Nassif, A. B., Lutfiyya, H., & Shami, A. (2018). E-Learning: Challenges and research opportunities using machine learning & data analytics. IEEE Access, 6, 39117–39138. https://doi.org/10.1109/ACCESS.2018.2851790
Nakamura, W., Teixeira de Oliveira, E., & Conte, T. (2018). Applying design science research to develop a technique to evaluate the usability and user experience of learning management systems. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), 29(2), 953. https://doi.org/10.5753/cbie.sbie.2018.953
Navarro, C. X., Molina, A. I., Redondo, M. A., & Juárez-Ramírez, R. (2016). Framework to evaluate M-learning systems: A Technological and pedagogical approach. IEEE Revista Iberoamericana De Tecnologias Del Aprendizaje, 11(1), 33–40. https://doi.org/10.1109/RITA.2016.2518459
Nayebi, F., Desharnais, J.-M., & Abran, A. (2012). The state of the art of mobile application usability evaluation. In 25th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE) (pp. 1–4). IEEE. https://doi.org/10.1109/CCECE.2012.6334930
Nicholson, P. (2007). A History of E-Learning. In B. Fernández-Manjón, J. M. Sánchez-Pérez, J. A. Gómez-Pulido, M. A. Vega-Rodríguez, & J. Bravo-Rodríguez (Eds.), Computers and Education. Springer.
Nielsen, J. (1994b). Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI ’94), pp. 152–158.
Nielsen, J. (1994a). 10 Usability Heuristics for User Interface Design. Nielsen Norman Group. https://www.nngroup.com/articles/ten-usability-heuristics/
Nielsen, J. (1990). Evaluating hypertext usability. In D. H. Jonassen & H. Mandl (Eds.), Designing hypermedia for learning (pp. 147–168). Springer.
Nokelainen, P. (2004). Conceptual definition of the technical and pedagogical usability criteria for digital learning material. In EdMedia+ Innovate Learning (pp. 4249–4254). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/11688/
Nokelainen, P. (2005). The technical and pedagogical usability criteria for digital learning material. In EdMedia+ Innovate Learning (pp. 1011–1016). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/20212/
Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. Journal of Educational Technology & Society, 9(2), 178–197.
Nyang’or, J., Villiers, R. D., & Ssemugabi, S. (2013). A framework for usability evaluation of an offline e-learning tutorial and its application in usability testing. In EdMedia+ Innovate Learning (pp. 1097–1105). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/112097/
Oliver, M. (2000). An introduction to the evaluation of learning technology. Journal of Educational Technology & Society, 3(4), 20–30.
Oztekin, A., Delen, D., Turkyilmaz, A., & Zaim, S. (2013). A machine learning-based usability evaluation method for eLearning systems. Decision Support Systems, 56, 63–73.
Page, M. J., McKenzie, J. E., Bossuyt, P. M., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Systematic Reviews, 10, 89. https://doi.org/10.1186/s13643-021-01626-4
Park, B. K., Kim, J. Y., & Rogers, V. E. (2020). Development and usability evaluation of a facebook-based intervention program for childhood cancer patients: mixed methods study. Journal of Medical Internet Research, 22(7), e18779. https://doi.org/10.2196/18779
Paz, F., & Pow-Sang, J. A. (2014). Current trends in usability evaluation methods: A systematic review. In 2014 7th International Conference on Advanced Software Engineering and Its Applications, 2014, pp. 11–15. https://doi.org/10.1109/ASEA.2014.10
Paz, F., & Pow-Sang, J. A. (2016). A systematic mapping review of usability evaluation methods for software development process. International Journal of Software Engineering and Its Applications, 10(1), 165–178.
Pérez-Sanagustín, M., Nussbaum, M., Hilliger, I., Alario-Hoyos, C., Heller, R. S., Twining, P., & Tsai, C.-C. (2017). Research on ICT in K-12 schools – A review of experimental and survey-based studies in computers & education 2011 to 2015. Computers & Education, 104, A1–A15. https://doi.org/10.1016/j.compedu.2016.09.006
Perry, G. T., Kulpa, C. C., Pinheiro, E., & Eichler, M. L. (2012). Lessons from an educational game usability evaluation. International Journal of Interactive Mobile Technologies (IJIM), 6(2), 23. https://doi.org/10.3991/ijim.v6i2.1888
Pham, M., Singh, K., & Jahnke, I. (2021). Socio-technical-pedagogical usability of online courses for older adult learners. Interactive Learning Environments. https://doi.org/10.1080/10494820.2021.1912784
Piasecki, J., Waligora, M., & Dranseika, V. (2018). Google Search as an Additional Source in Systematic Reviews. Science and Engineering Ethics, 24(2), 809–810. https://doi.org/10.1007/s11948-017-0010-4
Plantak Vukovac, D., Kirinic, V., & Klicek, B. (2010). A comparison of usability evaluation methods for e-learning systems. DAAAM International Scientific Book. https://doi.org/10.2507/daaam.scibook.2010.27
Plass, J. L., Homer, B. D., & Kinzer, C. K. (2015). Foundations of Game-Based Learning. Educational Psychologist, 50(4), 258–283. https://doi.org/10.1080/00461520.2015.1122533
Polly, D., Allman, B., Casto, A., & Norwood, J. (2018). Sociocultural perspectives of learning. In R. E. West (Eds.), Foundations of learning and instructional design technology: The past, present, and future of learning and instructional design technology. EdTech Books. Retrieved from https://edtechbooks.org/lidtfoundations/sociocultural_perspectives_of_learning
Punchoojit, L., & Hongwarittorrn, N. (2017). Usability Studies on Mobile User Interface Design Patterns: A Systematic Literature Review. Advances in Human-Computer Interaction, 2017, e6787504. https://doi.org/10.1155/2017/6787504
Quiñones, D., & Rusu, C. (2017). How to develop usability heuristics: A systematic literature review. Computer Standards & Interfaces, 53, 89–122. https://doi.org/10.1016/j.csi.2017.03.009
Quiñones, D., Rusu, C., & Rusu, V. (2018). A methodology to develop usability/user experience heuristics. Computer Standards & Interfaces, 59, 109–129. https://doi.org/10.1016/j.csi.2018.03.002
Rajanen, D., Clemmensen, T., Iivari, N., Inal, Y., Rızvanoğlu, K., Sivaji, A., & Roche, A. (2017). UX professionals’ definitions of usability and UX–A comparison between Turkey, Finland, Denmark, France and Malaysia. In R. Bernhaupt, G. Dalvi, A. Joshi, D. K. Balkrishan, J. O’Neill, & M. Winckler (Eds.), Ifip conference on human-computer interaction (pp. 218–239). Springer.
Rebelo, F., Noriega, P., Duarte, E., & Soares, M. (2012). Using virtual reality to assess user experience. Human Factors, 54(6), 964–982. https://doi.org/10.1177/0018720812465006
Reeves, T. C. (1994). Evaluating what really matters in computer-based education. In M. Wild & D. Kirkpatrick (Eds.), Computer education: New perspectives (pp. 219–246). MASTEC.
Reeves, T. C., & Hedberg, J. C. (2003). Interactive learning systems evaluation. Educational Technology Publications.
Resnik, L. (2011). Development and testing of new upper-limb prosthetic devices: Research designs for usability testing. The Journal of Rehabilitation Research and Development, 48(6), 697. https://doi.org/10.1682/JRRD.2010.03.0050
Robinson, H. A., Sheffield, A., Phillips, A. S., & Moore, M. (2017). “Introduction to Teaching Online”: Usability Evaluation of Interactivity in an Online Social Constructivist Course. TechTrends, 61(6), 533–540. https://doi.org/10.1007/s11528-017-0187-z
Rodrigues, H., Almeida, F., Figueiredo, V., & Lopes, S. L. (2019). Tracking e-learning through published papers: A systematic review. Computers & Education, 136, 87–98. https://doi.org/10.1016/j.compedu.2019.03.007
Rosell-Aguilar, F. (2017). State of the app: A taxonomy and framework for evaluating language learning mobile applications. CALICO Journal, 34(2), 243–258.
Rosenbaum, S. (2000). Not just a hammer: when and how to employ multiple methods in usability programs. In Proceedings of UPA.
Rosenbaum, S. (2008). The Future of Usability Evaluation: Increasing Impact on Value. In E. L. C. Law, E. T. Hvannberg, & G. Cockton (Eds.), Maturing Usability: Quality in Software, Interaction and Value (pp. 344–378). Springer.
Ruiz, J. G., Mintzer, M. J., & Leipzig, R. M. (2006). The impact of E-learning in medical education. Academic Medicine: Journal of the Association of American Medical Colleges, 81(3), 207–212. https://doi.org/10.1097/00001888-200603000-00002
Russell, J. D., & Blake, B. L. (1988). Formative and Summative Evaluation of Instructional Products and Learners. Educational Technology, 28(9), 22–28. http://www.jstor.org/stable/44426917
Santoso, H. B., Schrepp, M., Isal, R. Y. K., Utomo, A. Y., & Priyogi, B. (2016). Measuring User experience of the student-centered e-Learning environment. Journal of Educators Online, 13(1), 58–79.
Scanlon, E., & Issroff, K. (2005). Activity theory and higher education: Evaluating learning technologies. Journal of Computer Assisted Learning, 21(6), 430–439. https://doi.org/10.1111/j.1365-2729.2005.00153.x
Schmidt, M., Tawfik, A. A., Jahnke, I., & Earnshaw, Y. (2020). Learner and User Experience Research. EdTech Books. https://edtechbooks.org/ux
Schmidt, M. & Tawfik, A. A. (2022). Activity Theory as a Lens for Developing and Applying Personas and Scenarios in Learning Experience Design. The Journal of Applied Instructional Design, 11(1). https://edtechbooks.org/jaid_11_1/activity_theory_as_a
Schmidt, M., & Huang, R. (2022). Defining learning experience design: Voices from the field of learning design & technology. TechTrends, 66, 141–158. https://doi.org/10.1007/s11528-021-00656-y
Schnall, R., Cho, H., & Liu, J. (2018). Health Information Technology Usability Evaluation Scale (Health-ITUES) for usability assessment of mobile health technology: Validation study. JMIR mHealth and uHealth, 6(1), e8851. https://doi.org/10.2196/mhealth.8851
Schober, P., Boer, C., & Schwarte, L. A. (2018). Correlation coefficients: Appropriate use and interpretation. Anesthesia & Analgesia, 126(5), 1763–1768.
Seppälä, P., & Alamäki, H. (2003). Mobile learning in teacher training. Journal of Computer Assisted Learning, 19(3), 330–335. https://doi.org/10.1046/j.0266-4909.2003.00034.x
Sharples, M. (2000). The design of personal mobile technologies for lifelong learning. Computers & Education, 34(3), 177–193. https://doi.org/10.1016/S0360-1315(99)00044-5
Shekhar, A., & Marsden, N. (2018). Cognitive Walkthrough of a learning management system with gendered personas. In Proceedings of the 4th Conference on Gender & IT. https://doi.org/10.1145/3196839.3196869
Shilbayeh, S. A. R., & Ismail, S. A. E. R. (2020). Patient experience with an educational mobile health application: A pilot study on usability and feasibility in a Saudi population. Cogent Psychology, 7(1), 1843883. https://doi.org/10.1080/23311908.2020.1843883
Silius, K., Tervakari, A. M., & Pohjolainen, S. (2003). A multidisciplinary tool for the evaluation of usability, pedagogical usability, accessibility and informational quality of web-based courses. The Eleventh International PEG Conference: Powerful ICT for Teaching and Learning, 28, 1–10.
Spitzer, R. L., Williams, J. B., & Endicott, J. (2012). Standards for DSM-5 reliability. American Journal of Psychiatry, 169(5), 537–537.
Squires, D., & Preece, J. (1996). Usability and learning: Evaluating the potential of educational software. Computers & Education, 27(1), 15–22. https://doi.org/10.1016/0360-1315(96)00010-3
Squires, D., & Preece, J. (1999). Predicting quality in educational software. Interacting with Computers, 11(5), 467–483.
Srikong, M., & Wannapiroon, P. (2020). Immersive technology for medical education: Technology enhance immersive learning experiences. Siriraj Medical Journal, 72(3), 265–271. https://doi.org/10.33192/Smj.2020.36
Stern, C., Jordan, Z., & McArthur, A. (2014). Developing the review question and inclusion criteria. AJN the American Journal of Nursing, 114(4), 53–56.
Stevens, D., & Kitchenham, A. (2011). An analysis of mobile learning in education, business, and medicine. In A. Kitchenham (Ed.), Models for interdisciplinary mobile learning: Delivering information to students (pp. 1–25). IGI Global.
Storey, M. A., Phillips, B., Maczewski, M., & Wang, M. (2002). Evaluating the usability of Web-based learning tools. Journal of Educational Technology & Society, 5(3), 91–100.
Sugar, W. (2014). Development and formative evaluation of multimedia case studies for Instructional Design and Technology students. TechTrends, 58(5), 36–52. https://doi.org/10.1007/s11528-014-0785-z
Tan, W., Liu, D., & Bishu, R. (2009). Web evaluation: Heuristic evaluation vs. user testing. International Journal of Industrial Ergonomics, 39(4), 621–627. https://doi.org/10.1016/j.ergon.2008.02.012
Tawfik, A. A., Gatewood, J., Gish-Lieberman, J. J., & Hampton, A. J. (2021). Toward a definition of learning experience design. Technology, Knowledge and Learning, 27(1), 309–334. https://doi.org/10.1007/s10758-020-09482-2
Tessmer. (1993). Planning and Conducting Formative Evaluations. Routledge. https://doi.org/10.4324/9780203061978
Tondello, G. F., Kappen, D. L., Ganaba, M., & Nacke, L. E. (2019). Gameful design heuristics: a gamification inspection tool. In M. Kurosu (Ed.), International Conference on Human-Computer Interaction (pp. 224–244). Springer.
Turner, C. W., Lewis, J. R., & Nielsen, J. (2006). Determining usability test sample size. International Encyclopedia of Ergonomics and Human Factors, 3(2), 3084–3088.
Van Der Linden, J., Amadieu, F., Vayre, E., & Van De Leemput, C. (2019). User experience and social influence: A new perspective for UX theory. In A. Marcus & W. Wang (Eds.), International Conference on Human-Computer Interaction (pp. 98–112). Springer.
Walji, M. F., Kalenderian, E., Piotrowski, M., Tran, D., Kookal, K. K., Tokede, O., White, J. M., Vaderhobli, R., Ramoni, R., Stark, P. C., Kimmes, N. S., Lagerweij, M., & Patel, V. L. (2014). Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. International Journal of Medical Informatics, 83(5), 361–367. https://doi.org/10.1016/j.ijmedinf.2014.01.010
Wang, J. Y., Wu, H. K., & Hsu, Y. S. (2017). Using mobile applications for learning: Effects of simulation design, visual-motor integration, and spatial ability on high school students’ conceptual understanding. Computers in Human Behavior, 66, 103–113.
Weston, T. (2004). Formative evaluation for implementation: evaluating educational technology applications and lessons. American Journal of Evaluation, 25(1), 51–64. https://doi.org/10.1177/109821400402500104
Yáñez Gómez, R., Cascado Caballero, D., & Sevillano, J.-L. (2014). Heuristic evaluation on mobile interfaces: A New Checklist. The Scientific World Journal, 2014, 1–19. https://doi.org/10.1155/2014/434326
Yang, T., Linder, J., & Bolchini, D. (2012). DEEP: Design-oriented evaluation of perceived usability. International Journal of Human-Computer Interaction, 28(5), 308–346.
Yukselturk, E., Altıok, S., & Başer, Z. (2018). Using game-based learning with kinect technology in foreign language education course. Journal of Educational Technology & Society, 21(3), 159–173.
Yun, S. J., Kang, M.-G., Yang, D., Choi, Y., Kim, H., Oh, B.-M., & Seo, H. G. (2020). Cognitive training using fully immersive, enriched environment virtual reality for patients with mild cognitive impairment and mild dementia: feasibility and usability study. JMIR Serious Games, 8(4), e18127. https://doi.org/10.2196/18127
Zabed Ahmed, S. M. (2008). A comparison of usability techniques for evaluating information retrieval system interfaces. Performance Measurement and Metrics, 9(1), 48–58. https://doi.org/10.1108/14678040810869422
Zaharias, P., & Koutsabasis, P. (2012). Heuristic evaluation of e-learning courses: A comparative analysis of two e-learning heuristic sets. Campus-Wide Information Systems, 29(1), 45–60. https://doi.org/10.1108/10650741211192046
Zali, Z., & Fadzlah, A. F. A. (2016). An initial theoretical usability evaluation model for assessing defence mobile e-based application system. International Conference on Information and Communication Technology (ICICTM), 2016, 198–202. https://doi.org/10.1109/ICICTM.2016.7890800
Zardari, B. A., Hussain, Z., Arain, A. A., Rizvi, W. H., & Vighio, M. S. (2021). QUEST e-learning portal: Applying heuristic evaluation, usability testing and eye tracking. Universal Access in the Information Society, 20(3), 531–543. https://doi.org/10.1007/s10209-020-00774-z
Zeng, J., Parks, S., & Shang, J. (2020). To learn scientifically, effectively, and enjoyably: A review of educational games. Human Behavior and Emerging Technologies, 2(2), 186–195. https://doi.org/10.1002/hbe2.188
Zhang, J., Johnson, T. R., Patel, V. L., Paige, D. L., & Kubose, T. (2003). Using usability heuristics to evaluate patient safety of medical devices. Journal of Biomedical Informatics, 36(1), 23–30. https://doi.org/10.1016/S1532-0464(03)00060-1
Zurita, G., Baloian, N., Peñafiel, S., & Jerez, O. (2019). Applying pedagogical usability for designing a mobile learning application that support reading comprehension. Multidisciplinary Digital Publishing Institute Proceedings, 31(1), 6. https://doi.org/10.3390/proceedings2019031006
Acknowledgements
We thank Dr. Noah Glaser for his guidance on the methodology used in this research study.
Funding
Research reported in this manuscript was not funded.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Ethical approval
The research study described in this manuscript did not involve human subjects and/or animals other than the authors themselves.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lu, J., Schmidt, M., Lee, M. et al. Usability research in educational technology: a state-of-the-art systematic review. Education Tech Research Dev 70, 1951–1992 (2022). https://doi.org/10.1007/s11423-022-10152-6
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-022-10152-6