User Modeling and User-Adapted Interaction

, Volume 19, Issue 4, pp 341–385 | Cite as

Log file analysis for disengagement detection in e-Learning environments

Original Paper

Abstract

Most e-Learning systems store data about the learner’s actions in log files, which give us detailed information about learner behaviour. Data mining and machine learning techniques can give meaning to these data and provide valuable information for learning improvement. One area that is of particular importance in the design of e-Learning systems is learner motivation as it is a key factor in the quality of learning and in the prevention of attrition. One aspect of motivation is engagement, a necessary condition for effective learning. Using data mining techniques for log file analysis, our research investigates the possibility of predicting users’ level of engagement, with a focus on disengaged learners. As demonstrated previously across two different e-Learning systems, HTML-Tutor and iHelp, disengagement can be predicted by monitoring the learners’ actions (e.g. reading pages and taking test/quizzes). In this paper we present the findings of three studies that refine this prediction approach. Results from the first study show that two additional reading speed attributes can increase the accuracy of prediction. The second study suggests that distinguishing between two different patterns of disengagement (spending a long time on a page/test and browsing quickly through pages/tests) may improve prediction in some cases. The third study demonstrates the influence of exploratory behaviour on prediction, as most users at the first login familiarize themselves with the system before starting to learn.

Keywords

e-Learning Disengagement Log files analysis Educational data mining Motivation User modelling 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Arroyo I., Woolf B.P.: Inferring learning and attitudes from a Bayesian Network of log file data. In: Looi, C.K., McCalla, G., Bredeweg, B., Breuker, J. (eds) Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology, pp. 33–40. IOS Press, Amsterdam (2005)Google Scholar
  2. Baker, R., Corbett, A., Koedinger, K.: Detecting student misuse of intelligent tutoring systems. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems, pp. 531–540 (2004)Google Scholar
  3. Bandura A.: Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, Prentice Hall (1986)Google Scholar
  4. Beal, C.R., Lee, H.: Creating a pedagogical model that uses student self reports of motivation and mood to adapt ITS instruction. In: Proceedings of the Workshop on Emotion and Motivation in Educational Software (EMES). IOS Press, Amsterdam. Retrieved on 10 June 2006 from http://k12.usc.edu/Pubs/ (2005)
  5. Beal, C.R., Qu, L., Lee, H.: Classifying learner engagement through integration of multiple data sources. In: Proceedings of the 21st National Conference on Artificial Intelligence. AAAI Press, Menlo Park. Retrieved on 10 June 2006 from http://k12.usc.edu/Pubs/ (2006)
  6. Beck J.: Engagement tracing: using response times to model student disengagement. In: Looi, C., McCalla, G., Bredeweg, B., Breuker, J. (eds) Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology, pp. 88–95. IOS Press, Amsterdam (2005)Google Scholar
  7. Becta, U.K.: The benefits of an interactive whiteboard. Retrieved on 11 June 2006 from http://schools.becta.org.uk/index.php?section=tl&catcode=ss_tl_use_02&rid=86 (2006)
  8. Breiman L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)MATHMathSciNetGoogle Scholar
  9. Chen, G.D., Shen, G.Y., Ou, K.L., Liu, B.: Promoting motivation and eliminating disorientation for web based courses by a multi-user game. Paper presented at The ED-MEDIA/ED-TELECOM 98 World Conference on Educational Multimedia and Hypermedia and World conference on Educational Telecommunications, June 20–25, Germany (1998)Google Scholar
  10. Cocea M.: Assessment of motivation in online learning environments. In: Wade, V., Ashman, H., Smith, B. (eds) Proceedings of the 4th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, pp. 414–418. Springer, Berlin (2006)CrossRefGoogle Scholar
  11. Cocea, M., Weibelzahl, S.: Can log files analysis estimate learners’ level of motivation? In: Proceedings of ABIS Workshop, ABIS 2006—14th Workshop on Adaptivity and User Modeling in Interactive Systems, pp. 32–35. Hildesheim (2006)Google Scholar
  12. Cocea M., Weibelzahl S.: Eliciting motivation knowledge from log files towards motivation diagnosis for Adaptive Systems. In: Conati, C., McCoy, K., Paliouras, G. (eds) Proceedings of the 11th International Conference on User Modelling, Lecture Notes in Artificial Intelligence (LNAI), vol. 4511, pp. 197–206. Springer, Berlin (2007a)Google Scholar
  13. Cocea M., Weibelzahl S.: Cross-system validation of engagement prediction from log files. In: Duval, E., Klamma, R., Wolpers, M. (eds) Creating New Learning Experiences on a Global Scale, 2nd European Conference on Technology Enhanced Learning, EC-TEL 2007, Lecture Notes in Computer Science (LNCS), vol. 4753, pp. 14–25. Springer, Berlin (2007b)Google Scholar
  14. Connolly T., Stansfield M.: Using games-based e-Learning technologies in overcoming difficulties in teaching information systems. J Inf Technol Educ 5, 459–476 (2006)Google Scholar
  15. Csikszentmihalyi M.: Finding Flow: The Psychology of Engagement with Everyday Life. BasicBooks, New York (1997)Google Scholar
  16. De Vicente A., Pain H.: Informing the detection of the students’ motivational state: an empirical study. In: Cerri, S.A., Gouarderes, G., Paraguau, F. (eds) Intelligent Tutoring Systems, 6th International Conference, pp. 933–943. Springer, Berlin (2002)Google Scholar
  17. Farzan, R., Brusilovsky, P.: Social navigation support in e-Learning: what are real footprints. In: Proceedings of IJCAI’05 Workshop on Intelligent Techniques for Web Personalization, pp. 49–56. Edinburgh (2005)Google Scholar
  18. Gussak D., Baylor A.L.: Constructing agents for self-learning: animated agents as expressive vehicles. In: Lassner, D., McNaught, C. (eds) Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications, pp. 477–478. AACE, Chesapeake (2003)Google Scholar
  19. Hurley T., Weibelzahl S.: Eliciting adaptation knowledge from on-line tutors to increase motivation. In: Conati, C., McCoy, K., Paliouras, G. (eds) Proceedings of The 11th International Conference on User Modelling, Lecture Notes in Artificial Intelligence (LNAI), vol. 4511, pp. 370–374. Springer, Berlin (2007)Google Scholar
  20. Ishii, T., Saitou, M., Hiramoto, S.: An instructional design for developing learning contents by subject matter experts. In: Proceedings of the 9th World Conference on Continuing Engineering Education, pp. 469–474 (2004)Google Scholar
  21. Johns, J., Woolf, B.: A dynamic mixture model to detect student motivation and proficiency. In: Proceedings of the 21st National Conference on Artificial Intelligence (AAAI-06). Boston, MA, pp. 163–168 (2006)Google Scholar
  22. Keller J.M.: Development and use of the ARCS model of instructional design. J Instr Dev 10(3), 2–10 (1987)CrossRefGoogle Scholar
  23. Lombard, M., Snyder-Duch, J., Campanella Bracken, C.: Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research. Retrieved on 11 June 2007 from http://www.temple.edu/mmc/reliability (2003)
  24. Machado, I., Martins, A., Paiva, A.: One for all and all for one: a learner modelling server in a multi-agent platform. In: Proceedings of the 7th International Conference on User Modelling, pp. 211–221. Springer, Wien (1999)Google Scholar
  25. Martyn M.: Clickers in the classroom: an active learning approach. EDUCAUSE Q 30(2), 71–74 (2007)Google Scholar
  26. Mitchell T.M.: Machine Learning. McGraw-Hill, New York (1997)MATHGoogle Scholar
  27. Pintrich P., Schunk D.: Motivation in Education: Theory, Research and Applications. 2nd edn. Prentice-Hall, Englewood Cliffs (2002)Google Scholar
  28. Quinlan R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Mateo (1993)Google Scholar
  29. Qu L., Wang N., Johnson W.L.: Detecting the learner’s motivational states in an interactive learning environment. In: Looi, C.-K., McCalla, G., Bredeweg, B., Breuker, J. (eds) Proceedings of the 12th International Conference on Artificial Intelligence in Education., pp. 547–554. IOS Press, Amsterdam (2005)Google Scholar
  30. Shneiderman B., Alavi M., Norman K., Borkowski E.: Windows of opportunity in electronic classrooms. Commun ACM 38(11), 19–24 (1995)CrossRefGoogle Scholar
  31. Speier C., Vessey I., Valacich J.S.: Effects of interruptions, task complexity, and information presentation on computer-supported decision-making performance. Decis Sci 34(4), 771–797 (2003)CrossRefGoogle Scholar
  32. Walonoski J., Heffernan N.T.: Detection and analysis of off-task gaming behavior in intelligent tutoring systems. In: Ikeda, Ashley, Chan (eds) Proceedings of the 8th International Conference in Intelligent Tutoring Systems, pp. 382–391. Springer, Berlin (2006a)Google Scholar
  33. Walonoski J., Heffernan N.T.: Prevention of off-task gaming behaviour within intelligent tutoring systems. In: Ikeda, Ashley, Chan (eds) Proceedings of the 8th International Conference in Intelligent Tutoring Systems, pp. 722–724. Springer, Berlin (2006b)Google Scholar
  34. Weber, G., Kuhl, H.-C., Weibelzahl, S.: Developing adaptive internet based courses with the authoring system NetCoach. In: Hypermedia: Openness, Structural Awareness, and Adaptivity (LNAI 2266), pp. 226–238. Springer, Berlin (2001)Google Scholar
  35. Witten I.H., Frank E.: Data mining. Practical Machine Learning Tools and Techniques. 2nd edn. Morgan Kaufmann Publishers/Elsevier, San Mateo (2005)MATHGoogle Scholar
  36. Zhang, G., Cheng, Z., He, A., Huang, T.: A WWW-based learner’s learning motivation detecting system. In: Proceedings of International Workshop on Research Directions and Challenge Problems in Advanced Information Systems Engineering, Honjo City, Japan, September 16–19, http://www.akita-pu.ac.jp/system/KEST2003/ (2003)

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  1. 1.School of InformaticsNational College of IrelandDublin 1Ireland
  2. 2.London Knowledge Lab, Birkbeck CollegeUniversity of LondonLondonUK

Personalised recommendations