Markov Decision Process for MOOC Users Behavioral Inference

  • Firas JarbouiEmail author
  • Célya Gruson-Daniel
  • Alain Durmus
  • Vincent Rocchisani
  • Sophie-Helene Goulet Ebongue
  • Anneliese Depoux
  • Wilfried Kirschenmann
  • Vianney Perchet
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11475)


Studies on massive open online courses (MOOCs) users discuss the existence of typical profiles and their impact on the learning process of the students. However defining the typical behaviors as well as classifying the users accordingly is a difficult task. In this paper we suggest two methods to model MOOC users behaviour given their log data. We mold their behavior into a Markov Decision Process framework. We associate the user’s intentions with the MDP reward and argue that this allows us to classify them.


User behaviour studies Learning analytics Markov Decision Process Inverse Reinforcement Learning 


  1. 1.
    Romero, C., Ventura, S.: Educational data science in massive open online courses. Wiley Interdisc. Rev. Data Min. Knowl. Discov. 7, e1187 (2017)CrossRefGoogle Scholar
  2. 2.
    Ramesh, A., Goldwasser, D., Huang, B., Daume III, H., Getoor, L.: Modeling learner engagement in MOOCs using probabilistic soft logic. In: NIPS Workshop on Data Driven Education, p. 62 (2013)Google Scholar
  3. 3.
    Corrin, L., de Barba, P., Corin, C., Kennedy, G.: Visualizing patterns of student engagement and performance in MOOCs (2014)Google Scholar
  4. 4.
    Ye, C., Biswas, G.: Early prediction of student dropout and performance in MOOCs using higher granularity temporal information. J. Learn. Anal. 1, 169–172 (2014)CrossRefGoogle Scholar
  5. 5.
    Thompson, C., Brooks, C., Teasley, S.: Towards a general method for building predictive models of learner success using educational time series data. In: Workshops of the International Conference on Learning Analytics and Knowledge (2014)Google Scholar
  6. 6.
    Geigle, C., Zhai, C.X.: Modeling MOOC student behavior with two-layer hidden Markov models. In: Learning at Scale (2017)Google Scholar
  7. 7.
    Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (2005)zbMATHGoogle Scholar
  8. 8.
    Andrew, N., Russell, S.J. : Algorithms for Inverse Reinforcement Learning. In: ICML 2000 Proceedings of the Seventeenth International Conference on Machine Learning (2000)Google Scholar
  9. 9.
    Wulfmeier, M., Ondruska, P., Posner, I.: Maximum Entropy Deep Inverse Reinforcement Learning (2016)Google Scholar
  10. 10.
    Levine, S., Popovie, Z., Koltun, V.: Nonlinear inverse reinforcement learning with Gaussian processes. In: Advances in Neural Information Processing Systems 24 (NIPS) (2011)Google Scholar
  11. 11.
    Surana, A., Srivastava, K.: Bayesian nonparametric inverse reinforcement learning for switched Markov decision processes. In: ICMLA 2014 13th International Conference on Machine Learning and Applications (2014)Google Scholar
  12. 12.
    Babe, M., Marivate, V.V., Subramanian, K., Littman, M.: Apprenticeship learning about multiple intentions. In: Proceedings of the International Conference on International Conference on Machine Learning (2011)Google Scholar
  13. 13.
    Ramachandran, D., Amir, E.: Bayesian inverse reinforcement learning. In: IJCAI 2007 Proceedings of the 20th International Joint Conference on Artifical Intelligence (2007)Google Scholar
  14. 14.
    Fox, E., Sudderth, E., Jordan, M., Willsky, A.: Nonparametric bayesian learning of switching linear dynamical systems. In: Neural Information Processing Systems (2009)Google Scholar
  15. 15.
    Michini, B., How, J.: Improving the efficiency of Bayesian inverse reinforcement learning. In: ICRA (2012)Google Scholar
  16. 16.
    Zhu, X., Ghahramani, Z.: Learning from labeled and unlabeled data with label propagation. Technical Report, Carnegie Mellon University (2002)Google Scholar
  17. 17.
    Viterbi, A.: Error bounds for convolutional codes and an asymptotically optimum decoding algorithm (1967)Google Scholar
  18. 18.
    Jarboui, F., Rocchisani, V., Kirchenmann, W.: Users behavioural inference with Markovian decision process and active learning. In: IAL@PKDD/ECML (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Firas Jarboui
    • 1
    • 2
    Email author
  • Célya Gruson-Daniel
    • 3
    • 5
  • Alain Durmus
    • 2
  • Vincent Rocchisani
    • 1
  • Sophie-Helene Goulet Ebongue
    • 3
  • Anneliese Depoux
    • 3
    • 4
  • Wilfried Kirschenmann
    • 1
  • Vianney Perchet
    • 2
  1. 1.ANEOBoulogne BillancourtFrance
  2. 2.CMLA, École normale supérieur Paris SaclayUniversité Paris SaclayParisFrance
  3. 3.Centre Virchow-Villermé for Public Health Paris-BerlinUniversité Sorbonne Paris-CitéParisFrance
  4. 4.GRIPIC - EA 1498Sorbonne UniversitéParisFrance
  5. 5.DRISS (Digital Research in Science & Society)ParisFrance

Personalised recommendations