The purpose of this study is to explore the pedagogical design of massive open online courses (MOOCs) using evidence-based e-learning principles. MOOCs have become an important part of discourse in higher education. However, there has been shared concern on the quality of MOOCs as learning systems for engaging learners as well as fulfilling their needs. The researchers conducted a design review of 40 computer science MOOCs from two major MOOC providers. The findings indicate a relatively low application of the principles in general, with the exception of those related to the organization and presentation of content. MOOC platforms and the difficulty level of MOOCs used the application of e-learning principles and guidelines differently. Implications for future research and design of MOOCs are discussed.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Altman, D. G. (1991). Practical statistics for medical research. London: Chapman and Hall.
Barak, M., Watted, A., & Haick, H. (2016). Motivation to learn in massive open online courses: Examining aspects of language and social engagement. Computers & Education, 94, 49–60. https://doi.org/10.1016/j.compedu.2015.11.010.
Beaven, T., Hauck, M., Comas-Quinn, A., Lewis, T., & de los Arcos, B. (2014). MOOCs: Striking the right balance between facilitation and self-determination. Journal of Online Learning and Teaching, 10(1), 31–43.
Burd, E. L., Smith, S. P., & Reisman, S. (2015). Exploring business models for MOOCs in Higher Education. Innovative Higher Education, 40, 37–49.
Castaño-Muñoz, J., Kreijns, K., Kalz, M., & Punie, Y. (2017). Does digital competence and occupational setting influence MOOC participation? Evidence from a cross-course survey. Journal of Computing in Higher Education, 29, 28–46.
Chen, C.-M., & Wu, C.-H. (2015). Effects of different video lecture types on sustained attention, emotion, cognitive load, and learning performance. Computers & Education, 80, 108–121. https://doi.org/10.1016/j.compedu.2014.08.015.
Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 3, 7.
Christensen, G., Steinmetz, A., Alcorn, B., Bennett, A., Woods, D., & Emanuel, E. (2013). The MOOC phenomenon: Who takes massive open online courses and why? Retrieved January 7, 2018 from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2350964.
Chu, H.-C., & Hwang, G.-J. (2010). Development of a project-based cooperative learning environment for computer programming courses. International Journal of Innovation and Learning, 8(3), 256–266.
Chukwuemeka, E. J., Yoila, A. O., & Iscioglu, E. (2015). Instructional design quality: An evaluation of Open Education Europa Networks’ open courses using the first principles of instruction. International Journal of Science and Research, 4(11), 878–884.
Clark, R. C., & Mayer, R. E. (2011). E-Learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning. San Francisco, CA: John Wiley & Sons.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–222.
Creelman, A., Ehlers, U., & Ossiannilsson, E. (2014). Perspectives on MOOC quality: An account of the EFQUEL MOOC quality project. The International Journal for Innovation and Quality in Learning, 2(3), 78–87.
Davis, L. L. (1992). Instrument review: Getting the most from a panel of experts. Applied Nursing Research, 5, 194–197.
Dillahunt, T., Wang, Z., & Teasley, S. D. (2014). Democratizing higher education: Exploring MOOC use among those who cannot afford formal education. The International Review of Research in Open and Distance Learning. https://doi.org/10.19173/irrodl.v15i5.1841.
Egloffstein, M., & Ifenthaler, D. (2017). Employee perspectives on MOOCs for workplace learning. TechTrends, 61(1), 65–67.
Gonzalez, G. (2006). A systematic approach to active and cooperative learning in CS1 and its effects on CS2. In Proceedings of the 37th SIGCSE Technical Symposium on Computer Science Education (SIGCSE’06) (pp. 133–137). New York, NY: ACM Press.
Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38–43.
Gruba, P., & Sondergaard, H. (2001). A constructivist approach to communication skills instruction in computer science. Computer Science Education, 11(3), 203–219. https://doi.org/10.1076/csed.126.96.36.19933.
Guo, P. J., Kim, J., & Rubin, R. (2014). How video production affects student engagement: An empirical study of MOOC videos. In Proceedings of the First ACM Conference on Learning @ Scale Conference (pp. 41–50). New York, NY: ACM Press.
Hathaway, K. L. (2013). An application of the seven principles of good practice to online courses. Research in Higher Education Journal, 22, 1.
Hew, K. F. (2018). Unpacking the strategies of ten highly rated MOOCs: Implications for engaging students in large online courses. Teachers College Record, 120, 1–40.
Howarth, J., D’Alessandro, S., Johnson, L., & White, L. (2017). MOOCs to university: A consumer goal and marketing perspective. Journal of Marketing for Higher Education, 27(1), 144–158.
Jansen, D., Rosewell, J., & Kear, K. (2017). Quality frameworks for MOOCs. In M. J. Kinshuk & M. K. Khribi (Eds.), Open education: From OERs to MOOCs (pp. 261–281). Berlin: Springer.
Konrad, A. (2017). Coursera fights to keep the promise of MOOCs alive with corporate customer push. Forbes. Retrieved January 7, 2018 from https://www.forbes.com/sites/alexkonrad/2017/12/20/coursera-goes-corporate-to-keep-alive-promise-of-moocs/#40181021543c.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563–575.
Linn, M., & Dalbey, J. (1989). Cognitive consequences of programming instruction [reprinted]. In E. Soloway & J. C. Spohrer (Eds.), Studying the novice programmer (pp. 57–81). Hillsdale, NJ: Lawrence Erlbaum Associates.
Littlejohn, A., Hooda, N., Milligan, C., & Mustain, P. (2016). Learning in MOOCs: Motivations and self-regulated learning in MOOCs. Internet and Higher Education, 29, 40–48.
Lowenthal, P., & Hodges, C. (2015). In search of quality: Using quality matters to analyze the quality of massive, open, online courses (MOOCs). The International Review of Research in Open and Distributed Learning, 16(5), 83–101. https://doi.org/10.19173/irrodl.v16i5.2348.
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of Massive Open Online Courses (MOOCs). Computers & Education, 80, 77–83.
Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York, NY: Cambridge University Press.
Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43–59.
Milligan, C., & Littlejohn, A. (2014). Supporting professional learning in a Massive Open Online Course. The International Review of Research in Open and Distance Learning. https://doi.org/10.19173/irrodl.v15i5.1855.
National Academies of Sciences, Engineering, and Medicine. (2018). Assessing and responding to the growth of computer science undergraduate enrollments. Washington, D.C.: National Academies Press.
Ossiannilsson, E., Williams, K., Camilleri, A. F., & Brown, M. (2015). Quality models in online and open education around the globe: State of the art and recommendations. Oslo: International Council for Open and Distance Education.
Quality Matters. (2014). Introduction to the quality matters program. Retrieved January 10, 2018 from https://www.qualitymatters.org/sites/default/files/Introduction%20to%20the%20Quality%20Matters%20Program%20HyperlinkedFinal2014.pdf.
Robins, A., Rountree, J., & Rountree, N. (2003). Learning and teaching programming: A review and discussion. Computer Science Education, 13(2), 137–172.
Rohs, M., & Ganz, M. (2015). MOOCs and the claim of education for all: A disillusion by empirical data. International Review of Research in Open and Distributed Learning., 16(6), 5. https://doi.org/10.19173/irrodl.v16i6.2033.
Rosewell, J., & Jansen, D. (2014). The OpenupEd quality label: Benchmarks for MOOCs. INNOQUAL: The International Journal for Innovation and Quality in Learning, 2(3), 88–100.
Sanchez-Gordon, S., & Luján-Mora, S. (2018). Technological innovations in large-scale teaching: Five roots of massive open online courses. Journal of Educational Computing Research, 56(5), 623–644. https://doi.org/10.1177/0735633117727597.
Scagnoli, N. I., Choo, J., & Tian, J. (2019). Students’ insights on the use of video lectures in online classes. British Journal of Educational Technology, 50(1), 399–414. https://doi.org/10.1111/bjet.12572.
Shah, D. (2017). A product at every price: A review of MOOC stats and trends in 2017. Class Central. Retrieved January 7, 2018 from https://www.class-central.com/report/moocs-stats-and-trends-2017/.
Spector, J. M. (2014). Remarks on MOOCs and Mini-MOOCs. Educational Technology and Research Development, 62, 385–392.
Stracke, C. M. (2019). Quality frameworks and learning design for open education. International Review of Research in Open and Distributed Learning, 20(2), 180–203.
Tawfik, A. A., Reeves, T. D., Stich, A. E., Gill, A., Hong, C., McDade, J., et al. (2017). The nature and level of learner-learner interaction in a chemistry massive open online course (MOOC). Journal of Computing in Higher Education, 29(3), 411–431. https://doi.org/10.1007/s12528-017-9135-3.
Tirrell, T., & Quick, D. (2012). Chickering’s seven principles of good practice: Student attrition in community college online courses. Community College Journal of Research and Practice, 36(8), 580–590.
Toven-Lindsey, B., Rhoads, R. A., & Lozano, J. B. (2015). Virtually unlimited classrooms: Pedagogical practices in massive open online courses. Internet and Higher Education, 24, 1–12.
Veletsianos, G., & Shepherdson, P. (2016). A systematic analysis and synthesis of the empirical MOOC literature published in 2013–2015. The International Review of Research in Open and Distributed Learning. https://doi.org/10.19173/irrodl.v17i2.2448.
Waltz, C. F., Strickland, O., & Lenz, E. R. (2010). Measurement in nursing and health research (5th ed.). New York, NY: Springer.
Waters, J. (2015). How nanodegrees are disrupting higher education, campus technology. Retrieved January 11, 2018 from http://campustechnology.com/articles/2015/08/05/how-nanodegrees-are-disrupting-higher-education.aspx.
Watson, S. L., Loizzo, J., Watson, W. R., Mueller, C., Lim, J., & Ertmer, P. A. (2016). Instructional design, facilitation, and perceived learning outcomes: An exploratory case study of a human trafficking MOOC for attitudinal change. Educational Technology Research and Development, 64, 1273–1300.
Watson, W. R., Watson, S. L., & Janakiraman, S. (2017). Instructional quality of massive open online courses: A review of attitudinal change MOOCs. International Journal of Learning Technology, 12(3), 219–224.
Wexler, E. (2015). Moocs are still rising at least in numbers. The Chronicle of Higher Education. Retrieved January 7, 2018 from http://www.chronicle.com/blogs/wiredcampus/moocs-are-still-rising-at-least-in-numbers/57527.
Williams, L., & Kessler, R. (2002). Pair programming illuminated. Boston, MA: Addison-Wesley Professional.
Wing, J. (2011). Research notebook: Computational thinking—What and why? The Link Magazine, 6. Retrieved January 11, 2018 from https://www.cs.cmu.edu/link/research-notebook-computational-thinking-what-and-why.
Yang, M., Shao, Z., Liu, Q., & Liu, C. (2017). Understanding the quality factors that influence the continuance intention of students toward participation in MOOCs. Educational Technology Research and Development, 65, 1195–1214.
Yilmaz, A. B., Ünal, M., & Çakir, H. (2017). Evaluating MOOCs according to instructional design principles. Journal of Learning and Teaching in Digital Age, 2(2), 26–35.
Yoila, A. O., & Chukwuemeka, E. J. (2015). Instructional design quality evaluation of eastern mediterranean university open courses. International Journal of Scientific Research in Science, Engineering and Technology, 1(6), 1–7.
Zarb, M., & Hughes, J. (2015). Breaking the communication barrier: Guidelines to aid communication within pair programming. Computer Science Education, 25, 120–151. https://doi.org/10.1080/08993408.2015.1033125.
Zhang, Q., Peck, K. L., Hristova, A., Jablokow, K. W., Hoffman, V., Park, E., et al. (2016). Exploring the communication preferences of MOOC learners and the value of preference-based groups: Is grouping enough? Educational Technology Research and Development, 64, 809–837. https://doi.org/10.1007/s12528-017-9135-3.
Zhenghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. J. (2015). Who’s benefiting from MOOCs, and why. Harvard Business Review. Retrieved January 8, 2018 from https://hbr.org/2015/09/whos-benefiting-from-moocs-and-why.
Zhu, M., Bonk, C. J., & Sari, A. R. (2018a). Instructor experiences designing MOOCs in higher education: Pedagogical, resource, and logistical considerations and challenges. Online Learning, 22(4), 203–241. https://doi.org/10.24059/olj.v22i4.1495.
Zhu, M., Sari, A., & Lee, M. M. (2018b). A systematic review of research methods and topics of the empirical MOOC literature (2014-2016). The Internet and Higher Education, 37, 31–39.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Oh, E.G., Chang, Y. & Park, S.W. Design review of MOOCs: application of e-learning design principles. J Comput High Educ 32, 455–475 (2020). https://doi.org/10.1007/s12528-019-09243-w
- E-learning design
- Instructional quality
- MOOCs design