Assessing the Performance of a Tourism MOOC Using the Kirkpatrick Model: A Supplier’s Point of View

Conference paper


This paper presents the evaluation methods and results of a pilot tourism MOOC (Massive Open Online Course) called eTourism: Communication Perspectives, based on the Kirkpatrick model. It assigned twelve indicators to the model’s four levels of evaluation (reaction, learning, behaviour, results). Indicators include: self-efficacy and motivation, satisfaction, relevance, course performance, collaborative learning, higher-order learning, reflective and integrative learning, skills development, post-course practices, corporate social responsibility, public relations, and marketing. With various measurement tools such as pre-, in- and post-course surveys, post-course interviews, and analytics data by the host platform, the paper explains the available data with the twelve indicators and provides meaningful performance assessment for the MOOC. Results show that the MOOC was successful in all four levels according to the twelve indicators. The limitations and the future directions are also discussed at the end of the study.


MOOCs Massive Open Online Course Kirkpatrick model Tourism 


  1. Alario-Hoyos, C., Pérez-Sanagustín, M., Delgado-Kloos, C., & Munoz-Organero, M. (2014). Delving into participants’ profiles and use of social tools in MOOCs. IEEE Transactions on Learning Technologies, 3(7), 260–266.CrossRefGoogle Scholar
  2. Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341–347.CrossRefGoogle Scholar
  3. Bradley, K., & Connors, E. (2007). Training evaluation model: Evaluating and improving criminal justice training. Retrieved August 18, 2016, from
  4. Class Central. (2015). By the numbers: MOOCS in 2015. Retrieved August 22, 2016, from
  5. Creelman, A., Ehlers, U., & Ossiannilsson, E. (2014). Perspectives on MOOC quality-an account of the EFQUEL MOOC quality project. INNOQUAL-International Journal for Innovation and Quality in Learning, 2(3), 78–87.Google Scholar
  6. Cross, S. (2013). Evaluation of the OLDS MOOC curriculum design course: Participant perspectives, expectations and experiences. Retrieved September 4, 2016, from
  7. Douglas, K. A., Mihalec-Adkins, B. P., Hicks, N. M., & Diefes-Dux, H. A. (2016). Learners in advanced nanotechnology MOOCs: Understanding their intention and motivation. Retrieved October 18, 2016 from
  8. Downes, S. (2013). The quality of massive open online courses. Retrieved August 21, 2016, from
  9. Gamage, D., Fernando, S., & Perera, I. (2015). Quality of MOOCs: A review of literature on effectiveness and quality aspects. Education, 121, 2.Google Scholar
  10. Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. The International Review of Research in Open and Distributed Learning, 15(1).Google Scholar
  11. Khalil, M., Brunner, H., & Ebner, M. (2015). Evaluation grid for xMOOCs. International Journal of Emerging Technologies in Learning (iJET), 10(4).Google Scholar
  12. Khalil, H., & Ebner, M. (2014). MOOCs completion rates and possible methods to improve retention-a literature review. In EdMedia: World Conference on Educational Media and Technology (Vol. 2014, No. 1, pp. 1305–1313).Google Scholar
  13. Kirkpatrick, D. (1975). Evaluating training programs. Tata McGraw-Hill Education.Google Scholar
  14. Kirkpatrick Partners. (2016). The Kirkpatrick model. Retrieved August 18, 2016, from
  15. Lesjak, B., & Florjancic, V. (2014). Evaluation of MOOC: Hands-On project or creative use of ict in teaching. In Human Capital without Borders: Knowledge and Learning for Quality of Life; Proceedings of the Management, Knowledge and Learning International Conference 2014 (pp. 1147–1155). ToKnowPress.Google Scholar
  16. Lin, J., Cantoni, L., & Kalbaska, N. (2016). How to develop and evaluate an eTourism MOOC: An experience in progress. e-Review of Tourism Research (eRTR), 7, 1–5.Google Scholar
  17. Lin, J., Kalbaska, N., Cantoni, L., & Murphy, J. (2016). A new framework to describe and analyse MOOC design: Multiple case study of hospitality and tourism MOOCs. Revised and resubmitted for publication.Google Scholar
  18. Murphy, J., Tracey, J. B., & Horton-Tognazzini, L. (2016). MOOC camp: A flipped classroom and blended learning model. In Information and Communication Technologies in Tourism 2016 (pp. 653–665). Springer International Publishing.Google Scholar
  19. Onah, D. F., Sinclair, J., & Boyatt, R. (2014). Exploring the use of MOOC discussion forums. In Proceedings of London International Conference on Education (pp. 1–4). LICE.Google Scholar
  20. Parra, J. (2016). Moving beyond MOOC mania: Lessons from a faculty-designed MOOC. Current Issues in Emerging eLearning, 3(1), 10.Google Scholar
  21. Poce, A. (2015). Developing critical perspectives on technology in education: A tool for MOOC evaluation. European Journal of Open, Distance and E-learning, 18(1).Google Scholar
  22. Rodrigo, C., Read, T., Santamaría, M., & Sánchez-Elvira, A. (2014). OpenupEdLabel for MOOC quality assurance: UNED COMA initial self-evaluation. In Actas del V Congreso Internacional sobre Calidad y Accesibilidad en la Formación Virtual (CAFVIR 2014) (pp. 551–555).Google Scholar
  23. Tracey, J. B., Murphy, J., & Horton-Tognazzini, L. (2016). A framework for evaluating MOOCs in applied hospitality and tourism settings. In Information and Communication Technologies in Tourism 2016 (pp. 667–679). Springer International Publishing.Google Scholar
  24. Wintrup, J., Wakefield, K., & Davis, H. C. (2015). Engaged learning in MOOCs: A study using the UK engagement survey.Google Scholar
  25. Yousef, A. M. F., Chatti, M. A., Ahmad, I., Schroeder, U., & Wosnitza, M. (2015). An evaluation of learning analytics in a blended MOOC environment. In Proceedings of the Third European MOOCs Stakeholders Summit EMOOCs (pp. 122–130).Google Scholar
  26. Yousef, A. M. F., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2015). A usability evaluation of a blended MOOC environment: An experimental case study. The International Review of Research in Open and Distributed Learning, 16(2).Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.The Faculty of Communication SciencesUniversità della Svizzera ItalianaluganoSwitzerland

Personalised recommendations