Skip to main content

Analyzing learners’ engagement and behavior in MOOCs on programming with the Codeboard IDE

Educational Technology Research and Development volume 68, pages 2505–2528 (2020)Cite this article

Abstract

Massive Open Online Courses (MOOCs) can be enhanced with the so-called learning-by-doing, designing the courses in a way that the learners are involved in a more active way in the learning process. Within the options for increasing learners’ interaction in MOOCs, it is possible to integrate (third-party) external tools as part of the instructional design of the courses. In MOOCs on computer sciences, there are, for example, web-based Integrated Development Environments (IDEs) which can be integrated and that allow learners to do programming tasks directly in their browsers without installing desktop software. This work focuses on analyzing the effect on learners’ engagement and behavior of integrating a third-party web-based IDE, Codeboard, in three MOOCs on Java programming with the purpose of promoting learning-by-doing (learning by coding in this case). In order to measure learners’ level of engagement and behavior, data was collected from Codeboard on the number of compilations, executions and code generated, and compared between learners who registered in Codeboard to save and keep a record of their projects (registered learners) and learners who did not register in Codeboard and did not have access to these extra features (anonymous learners). The results show that learners who registered in Codeboard were more engaged than learners who did not register (in terms of number of compilations and executions), spent more time coding and did more changes in the base code provided by the teachers. The main implication of this study suggests the need for a trade-off between designing MOOCs that allow a very easy and anonymous access to external tools aimed for a more active learning, and forcing learners to give a step forward in terms of commitment in exchange for benefitting from additional features of the external tool used.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1

References

  • Alario-Hoyos, C., Bote-Lorenzo, M. L., Gómez-Sánchez, E., Asensio-Pérez, J. I., Vega-Gorgojo, G., & Ruiz-Calleja, A. (2013). GLUE!: An architecture for the integration of external tools in Virtual Learning Environments. Computers & Education, 60(1), 122–137.

    Article  Google Scholar 

  • Alario-Hoyos, C., Estévez Ayres, I. M., Gallego Romero, J. M., Delgado Kloos, C., Fernández Panadero, M. C., Crespo García, R., et al. (2018). A study of learning-by-doing in MOOCs through the integration of third-party external tools: Comparison of synchronous and asynchronous running modes. Journal of Universal Computer Science, 24(8), 1015–1033.

    Google Scholar 

  • Alario-Hoyos, C., Pérez-Sanagustín, M., Delgado-Kloos, C., Parada, G. H. A., & Muñoz-Organero, M. (2014). Delving into participants’ profiles and use of social tools in MOOCs. IEEE Transactions on Learning Technologies, 7(3), 260–266.

    Article  Google Scholar 

  • Alario-Hoyos, C. & Wilson, S. (2010). Comparison of the main alternatives to the integration of external tools in different platforms. In Proceedings of the International Conference of Education, Research and Innovation, ICERI (pp. 3466–3476).

  • Aleven, V., Baker, R., Blomberg, N., Andres, J. M., Sewall, J., Wang, Y., & Popescu, O. (2017). Integrating MOOCs and Intelligent Tutoring Systems: edX, GIFT, and CTAT. In 5th Annual Generalized Intelligent Framework for Tutoring Users Symposium (pp. 11–21).

  • Ally, M. (2008). Foundations of educational theory for online learning. In T. Anderson (Ed.), The theory and practice of online learning (2nd ed., pp. 15–44). Athabasca: Athabasca University Press.

    Google Scholar 

  • Antonucci, P., Estler, C., Nikolić, D., Piccioni, M., & Meyer, B. (2015). An incremental hint system for automated programming assignments. In 2015 ACM Conference on Innovation and Technology in Computer Science Education (pp. 320–325). ACM.

  • Bali, M. (2014). MOOC pedagogy: Gleaning good practice from existing MOOCs. Journal of Online Learning and Teaching, 10(1), 44–56.

    Google Scholar 

  • Ben-Ari, M. M. (2013). MOOCs on introductory programming: A travelogue. ACM Inroads, 4(2), 58–61.

    Article  Google Scholar 

  • Brusilovsky, P., Kouchnirenko, A., Miller, P., & Tomek, I. (1994). Teaching programming to novices: A review of approaches and tools. In 1994 World Conference on Educational Multimedia and Hypermedia (ED-MEDIA) (pp. 103–110).

  • Chi, H., Allen, C., & Jones, E. (2016). Integrating Computing to STEM Curriculum via CodeBoard. In International Conference on Computational Science and Its Applications (pp. 512–529). Springer, Cham.

  • Codeboard. (2020). Retrieved April 2020 from: https://codeboard.io/

  • De Freitas, S. I., Morgan, J., & Gibson, D. (2015). Will MOOCs transform learning and teaching in higher education? Engagement and course retention in online learning provision. British Journal of Educational Technology, 46(3), 455–471.

    Article  Google Scholar 

  • De Lucia, A., Scanniello, G., & Tortora, G. (2004). Identifying Clones in Dynamic Web Sites Using Similarity Thresholds. In Sixth International Conference on Enterprise Information Systems (pp. 391–396).

  • Derval, G., Gego, A., Reinbold, P., Frantzen, B., & Van Roy, P. (2015). Automatic grading of programming exercises in a MOOC using the INGInious platform. In European MOOCs Stakeholder Summit on experiences and best practices in and around MOOCs (EMOOCS’15) (pp. 86–91).

  • España-Boquera, S., Guerrero-López, D., Hermida-Pérez, A., Silva, J., & Benlloch-Dualde, J. V. (2017). Analyzing the learning process (in Programming) by using data collected from an online IDE. In 2017 16th International Conference on Information Technology Based Higher Education and Training (ITHET) (pp. 1–4). IEEE.

  • Evans, B. J., Baker, R. B., & Dee, T. S. (2016). Persistence patterns in massive open online courses (MOOCs). The Journal of Higher Education, 87(2), 206–242.

    Article  Google Scholar 

  • Ferguson, R., & Sharples, M. (2014). Innovative pedagogy at massive scale: Teaching and learning in MOOCs. In European Conference on Technology Enhanced Learning (EC-TEL 2014) (pp. 98–111). Springer, Cham.

  • Fontenla, J., Pérez, R., & Caeiro, M. (2011). Using IMS Basic LTI to integrate games in LMSs: Lessons from Game•Tel. In 2011 IEEE Global Engineering Education Conference (EDUCON) (pp. 299–306). IEEE.

  • Forment, M. A., Guerrero, M. J. C., Mayol, E., Piguillem, J., Galanis, N., García-Peñalvo, F. J., et al. (2012). Docs4Learning: Getting Google Docs to work within the LMS with IMS BLTI. Journal of Universal Computer Science, 18(11), 1483–1500.

    Google Scholar 

  • Freire, M., del Blanco, Á., & Fernández-Manjón, B. (2014). Serious games as edX MOOC activities. In 2014 IEEE Global Engineering Education Conference (EDUCON) (pp. 867–871). IEEE.

  • Funieru, L. M., & Lăzăroiu, F. (2016). Massive open online courses (MOOCs): A comparative analysis of the main platforms. Informatica Economică, 20(2), 35–45.

    Article  Google Scholar 

  • Fwa, H. L., & Marshall, L. (2018). Modeling engagement of programming students using unsupervised machine learning technique. GSTF Journal on Computing, 6(1), 1–6.

    Google Scholar 

  • Gilbert, M. A. (2015). edX E-learning course development. Birmingham: Packt Publishing Ltd.

    Google Scholar 

  • Godwin-Jones, R. (2014). Global reach and local practice: The promise of MOOCS. Language Learning & Technology, 18(3), 5–15.

    Google Scholar 

  • Gütl, C., Rizzardini, R. H., Chang, V., & Morales, M. (2014). Attrition in MOOC: Lessons learned from drop-out students. In International workshop on learning technology for education in cloud (pp. 37–48). Springer, Cham.

  • Hansen, J. D., & Reich, J. (2015). Democratizing education? Examining access and usage patterns in massive open online courses. Science, 350(6265), 1245–1248.

    Article  Google Scholar 

  • Hansen, S., & Eddy, E. (2007). Engagement and frustration in programming projects. ACM SIGCSE Bulletin, 39(1), 271–275.

    Article  Google Scholar 

  • Henson, K. T. (2003). Foundations for learner-centered education: A knowledge base. Education, 124(1), 5–16.

    Google Scholar 

  • Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47(2), 320–341.

    Article  Google Scholar 

  • Hill, P. (2013). Emerging student patterns in MOOCs: A graphical view. e-Literate. Retrieved April 2020 from: https://eliterate.us/emerging_student_patterns_in_moocs_graphical_view.

  • Hollands, F. M., & Tirthali, D. (2014). Resource requirements and costs of developing and delivering MOOCs. The International Review of Research in Open and Distributed Learning, 15(5), 113–133.

    Article  Google Scholar 

  • IMS LTI (2019). Learning tools interoperability core specification, IMS final release, Version 1.3. Retrieved April 2020, from: https://www.imsglobal.org/spec/lti/v1p3/.

  • Jurado, F., & Redondo, M. A. (2016). IMS-LTI and web-services for integrating Moodle to an eclipse-based distributed environment for learning to program. International Journal of Engineering Education, 32(2), 1007–1014.

    Google Scholar 

  • Khalil, H., & Ebner, M. (2014). MOOCs completion rates and possible methods to improve retention: A literature review. In EdMedia+ Innovate Learning (pp. 1305–1313). AACE.

  • Kim, K. J., & Bonk, C. J. (2006). The future of online teaching and learning in higher education. Educause Quarterly, 29(4), 22–30.

    Google Scholar 

  • Király, S., Nehéz, K., & Hornyák, O. (2017). Some aspects of grading Java code submissions in MOOCs. Research in Learning Technology, 25, 1–16.

    Article  Google Scholar 

  • Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the third international conference on learning analytics and knowledge (pp. 170–179).

  • Koedinger, K. R., Kim, J., Jia, J. Z., McLaughlin, E. A., & Bier, N. L. (2015). Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In Second ACM conference on learning@ scale (pp. 111–120). ACM.

  • Krugel, J. & Hubwieser, P. (2017). Computational thinking as springboard for learning object-oriented programming in an interactive MOOC. In 2017 IEEE Global Engineering Education Conference (EDUCON) (pp. 1709–1712). IEEE.

  • Kumar, S., Gankotiya, A. K., & Dutta, K. (2011). A comparative study of Moodle with other e-learning systems. In 2011 3rd International Conference on Electronics Computer Technology (Vol. 5, pp. 414–418). IEEE.

  • Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10, 707–710.

    Google Scholar 

  • Li, W., Gao, M., Li, H., Xiong, Q., Wen, J., & Wu, Z. (2016). Dropout prediction in MOOCs using behavior features and multi-view semi-supervised learning. In 2016 International Joint Conference on Neural Networks (IJCNN) (pp. 3130–3137). IEEE.

  • Lister, R. & Leaney, J. (2003). Introductory programming, criterion-referencing, and bloom. In 34th SIGCSE technical symposium on Computer science education (pp. 143–147).

  • Maldonado-Mahauad, J., Pérez-Sanagustín, M., Kizilcec, R. F., Morales, N., & Munoz-Gama, J. (2018). Mining theory-based patterns from Big data: Identifying self-regulated learning strategies in Massive Open Online Courses. Computers in Human Behavior, 80, 179–196.

    Article  Google Scholar 

  • Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MOOCs). Computers & Education, 80, 77–83.

    Article  Google Scholar 

  • Matthíasdóttir, Á. (2006). How to teach programming languages to novice students? Lecturing or not. In International Conference on Computer Systems and Technologies-CompSysTech (Vol. 6, pp. 15–16).

  • Meyer, B. (2017). Fourteen years of software engineering at ETH Zurich. arXiv:1712.05078 (pp. 1–118).

  • Morales-Chan, M., de la Roca, M., Alario-Hoyos, C., Barchino-Plata, R., Medina, J. A., & Hernández-Rizzardini, R. (2017). Perceived usefulness and motivation students towards the use of a cloud-based tool to support the learning process in a Java MOOC. In 2018 International Conference MOOC-Maker (MOOC-Maker) (pp. 73–82).

  • Noh, S. Y., Kim, S., & Jung, C. (2006). A lightweight program similarity detection model using XML and Levenshtein distance. In FECS (pp. 3–9).

  • Norman, D. A., & Spohrer, J. C. (1996). Learner-centered education. Communications of the ACM, 39(4), 24–27.

    Article  Google Scholar 

  • Moodle Plugins. (2020). Retrieved April 2020, from: https://moodle.org/plugins/.

  • Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231.

    Article  Google Scholar 

  • Pu, Y., Narasimhan, K., Solar-Lezama, A., & Barzilay, R. (2016). sk_p: a neural program corrector for MOOCs. In 2016 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity (pp. 39–40). ACM.

  • Rai, L., & Chunrao, D. (2016). Influencing factors of success and failure in MOOC and general analysis of learner behavior. International Journal of Information and Education Technology, 6(4), 262–268.

    Article  Google Scholar 

  • Ramesh, A., Goldwasser, D., Huang, B., Daumé III, H., & Getoor, L. (2013). Modeling learner engagement in MOOCs using probabilistic soft logic. In NIPS workshop on data driven education (pp. 1–7).

  • Reich, J., & Ruipérez-Valiente, J. A. (2019). The MOOC pivot. Science, 363(6423), 130–131.

    Article  Google Scholar 

  • Sarpong, K. A. M., Arthur, J. K., & Amoako, P. Y. O. (2013). Causes of failure of students in computer programming courses: The teacher-learner Perspective. International Journal of Computer Applications, 77(12), 27–32.

    Article  Google Scholar 

  • Severance, C., Hanss, T., & Hardin, J. (2010). IMS Learning Tools Interoperability: Enabling a mash-up approach to teaching and learning tools. Technology, Instruction, Cognition and Learning, 7(3–4), 245–262.

    Google Scholar 

  • Shah, D. (2019). By the numbers: MOOCs in 2019. Retrieved April 2020, from https://www.classcentral.com/report/mooc-stats-2019/.

  • Sheneamer, A., & Kalita, J. (2015). Code clone detection using coarse and fine-grained hybrid approaches. In 2015 IEEE seventh international conference on intelligent computing and information systems (ICICIS) (pp. 472–480). IEEE.

  • Sirkiä, T., & Haaranen, L. (2017). Improving online learning activity interoperability with acos server. Software Practice and Experience, 47(11), 1657–1676.

    Article  Google Scholar 

  • Škoríc, I., Pein, B., & Orehovački, T. (2016). Selecting the most appropriate web IDE for learning programming using AHP. In 2016 39th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) (pp. 877–882). IEEE.

  • Staubitz, T., Klement, H., Renz, J., Teusner, R., & Meinel, C. (2015). Towards practical programming exercises and automated assessment in Massive Open Online Courses. In 2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) (pp. 23–30). IEEE.

  • Staubitz, T., Klement, H., Teusner, R., Renz, J., & Meinel, C. (2016). CodeOcean: A versatile platform for practical programming exercises in online environments. In 2016 IEEE Global Engineering Education Conference (EDUCON) (pp. 314–323). IEEE.

  • Su, Y. S., Ding, T. J., & Lai, C. F. (2017). Analysis of students engagement and learning performance in a social community supported computer programming course. Eurasia Journal of Mathematics, Science and Technology Education, 13(9), 6189–6201.

    Article  Google Scholar 

  • Sunar, A. S., White, S., Abdullah, N. A., & Davis, H. C. (2016). How learners’ interactions sustain engagement: A MOOC case study. IEEE Transactions on Learning Technologies, 10(4), 475–487.

    Article  Google Scholar 

  • Tang, T., Rixner, S., & Warren, J. (2014). An environment for learning interactive programming. In 45th ACM technical symposium on Computer science education (pp. 671–676). ACM.

  • Volchek, D., Romanov, A., & Mouromtsev, D. (2017). Towards the semantic MOOC: Extracting, enriching and interlinking e-learning data in open edX platform. In International Conference on Knowledge Engineering and the Semantic Web (pp. 295–305). Springer, Cham.

  • Wang, X., Yang, D., Wen, M., Koedinger, K., & Rosé, C. P. (2015). Investigating How Student's Cognitive Behavior in MOOC Discussion Forums Affect Learning Gains. In 2015 International Conference on Educational Data Mining (EDM) (pp. 226–233). International Educational Data Mining Society.

  • Warren, J., Rixner, S., Greiner, J., & Wong, S. (2014). Facilitating human interaction in an online programming course. In 45th ACM technical symposium on Computer science education (pp. 665–670). ACM.

  • Wilson, S., Daniel, F., Jugel, U., & Soi, S. (2011). Orchestrated user interface mashups using w3c widgets. In International Conference on Web Engineering (pp. 49–61). Springer, Berlin, Heidelberg.

  • XBlocks Directory. (2020). Retrieved April 2020, from: https://openedx.atlassian.net/wiki/spaces/COMM/pages/43385346/XBlocks+Directory.

  • Xu, B., & Yang, D. (2016). Motivation classification and grade prediction for MOOCs learners. Computational Intelligence and Neuroscience, 2016, 1–7.

    Google Scholar 

  • Yousef, A. M. F., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2014). What drives a successful MOOC? An empirical examination of criteria to assure design quality of MOOCs. In 2014 IEEE 14th International Conference on Advanced Learning Technologies (pp. 44–48). IEEE.

Download references

Acknowledgements

The work received partial support from FEDER/Ministerio de Ciencia, Innovación y Universidades—Agencia Estatal de Investigación through project Smartlet (TIN2017-85179-C3-1-R), from the eMadrid Network, which is funded by the Madrid Regional Government (Comunidad de Madrid) with grant No. P2018/TCS-4307 and from the European Commission through Erasmus + projects LALA (586120-EPP-1-2017-1-ES-EPPKA2-CBHE-JP), InnovaT (598758-EPP-1-2018-1-AT-EPPKA2- CBHE-JP) and PROF-XXI (609767-EPP-1-2019-1- ES-EPPKA2-CBHE-JP). This publication reflects the views only of the authors and funders cannot be held responsible for any use which may be made of the information contained therein.

Author information

Authors and Affiliations

  1. Universidad Carlos III de Madrid, Avenida Universidad 30, 28911, Madrid, Spain

    Jesús Manuel Gallego-Romero, Carlos Alario-Hoyos, Iria Estévez-Ayres & Carlos Delgado Kloos

Authors
  1. Jesús Manuel Gallego-Romero
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carlos Alario-Hoyos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Iria Estévez-Ayres
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carlos Delgado Kloos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlos Alario-Hoyos.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Gallego-Romero, J.M., Alario-Hoyos, C., Estévez-Ayres, I. et al. Analyzing learners’ engagement and behavior in MOOCs on programming with the Codeboard IDE. Education Tech Research Dev 68, 2505–2528 (2020). https://doi.org/10.1007/s11423-020-09773-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-020-09773-6

Keywords

  • MOOCs
  • Programming
  • Learners’ engagement
  • Codeboard
  • Learning analytics
  • edX