Measuring the difficulty of activities for adaptive learning

  • Francisco J. Gallego-Durán
  • Rafael Molina-Carmona
  • Faraón Llorens-Largo
Long Paper


An effective adaptive learning system would theoretically maintain learners in a permanent state of flow. In this state, learners are completely focused on activities. To attain this state, the difficulty of learning activities must match learners’ skills. To perform this matching, it is essential to define, measure and deeply analyze difficulty. However, very few previous works deal with difficulty in depth. Most commonly, difficulty is defined as a one-dimensional value. This permits ordering activities, but limits the possibilities of deep analysis of activities and learners’ performance. This work proposes a new definition of difficulty and a way to measure it. The proposed definition depends on learners’ progress on activities over time. This expands the concept of difficulty over a two-dimensional space, also making it drawable. The difficulty graphs provide a rich interpretation with insights into the learning process. A practical case is presented: the PLMan learning system. This system is formed by a web application and a game to teach computational logic. The proposed definition is applied in this context. Measures are taken and analyzed using difficulty graphs. Some examples of these analyses are shown to illustrate the benefits of this proposal. Singularities and interesting spots are easily identified in graphs, providing insights in the activities. This new information lets experts adapt the learning system by improving activity classification and assignment. This first step lays solid foundations for automation, making the PLMan learning system fully adaptive.


Difficulty estimation Difficulty measure Learning activity Adaptive learning 


  1. 1.
    Aponte, M.V., Levieux, G., Natkin, S.: Scaling the level of difficulty in single player video games. In: Natkin, S., Dupire, J. (eds.) Entertainment Computing ICEC 2009, Lecture Notes in Computer Science, vol. 5709, pp. 24–35. Springer, Berlin (2009). doi: 10.1007/978-3-642-04052-8_3 CrossRefGoogle Scholar
  2. 2.
    Cheng, I., Shen, R., Basu, A.: An algorithm for automatic difficulty level estimation of multimedia mathematical test items. In: Eighth IEEE International Conference on Advanced Learning Technologies, 2008. ICALT ’08, pp. 175–179 (2008). doi: 10.1109/ICALT.2008.105
  3. 3.
    Csíkszentmihályi, M.: Flow: The Psychology of Optimal Experience. Perennial Modern Classics. Harper & Row, New York (1990)Google Scholar
  4. 4.
    Elo, A.E.: The Rating of Chessplayers, Past and Present. Arco Publishing Company, New York (1978)Google Scholar
  5. 5.
    Felder, R.M., Silverman, L.K.: Learning and teaching styles in engineering education. Eng. Educ. 78(7), 674–681 (1988).
  6. 6.
    Gallego-Durán, F.J.: Estimating difficulty of learning activities in design stages: a novel application of neuroevolution. PhD thesis, University of Alicante (2015).
  7. 7.
    Gallego-Durán, F.J., Molina-Carmona, R., Llorens-Largo, F.: An approach to measuring the difficulty of learning activities. In: Zaphiris, P., Ioannou, A. (eds.) Learning and Collaboration Technologies, LCT 2016, Lecture Notes in Computer Science, vol 9753, chapter 38, pp. 417–428. Springer Nature, Cham (2016). doi: 10.1007/978-3-319-39483-1_38
  8. 8.
    Getzels, J., Csíkszentmihályi, M.: The Creative Vision: A Longitudinal Study of Problem Finding in Art. Wiley, New York (1976)Google Scholar
  9. 9.
    Herbrich, R., Minka, T., Graepel, T.: \(\text{ Trueskill }^{{\rm TM}}\): A Bayesian skill rating system. In: Schölkopf, B., Platt, J., Hoffman, T. (eds.) Advances in Neural Information Processing Systems, vol. 19, pp. 569–576. MIT Press, Cambridge, MA (2007)Google Scholar
  10. 10.
    Hunicke, R.: The case for dynamic difficulty adjustment in games. In: Proceedings of the 2005 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, ACM, New York, NY, USA, ACE ’05, pp. 429–433 (2005). doi: 10.1145/1178477.1178573
  11. 11.
    Hunicke, R., Chapman, V.: Ai for dynamic difficulty adjustment in games. In: Proceedings of AIIDE 2004 (2004).
  12. 12.
    Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., Hall, C.: NMC Horizon Report: 2016 Higher Education Edition. New Media Consortium, EDUCAUSE Learning Initiative, Austin Texas, [S.l.] (2016)Google Scholar
  13. 13.
    Klanja-Milievi, A., Vesin, B., Ivanovi, M., Budimac, Z.: E-Learning personalization based on hybrid recommendation strategy and learning style identification. Comput. Educ. 56(3), 885–899 (2011). doi: 10.1016/j.compedu.2010.11.001 CrossRefGoogle Scholar
  14. 14.
    Missura, O., Gartner, T.: Predicting dynamic difficulty. In: Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., Weinberger, K. (eds.) Advances in Neural Information Processing Systems 24, Curran Associates, Inc., pp. 2007–2015 (2011).
  15. 15.
    Mladenov, M., Missura, O.: Offline learning for online difficulty prediction. In: Workshop on Machine Learning and Games at ICML 2010 (2010)Google Scholar
  16. 16.
    Mourato, F.J., dos Santos, M.P.: Measuring difficulty in platform videogames. In: 4a Conferencia Nacional em Interacao Pessoa-Mquina, Grupo Portugues de Computaao Grfica/Eurographics, pp. 173–180 (2010)Google Scholar
  17. 17.
    Nicholls, J.G., Miller, A.T.: The differentiation of the concepts of difficulty and ability. Child Dev. 54(4), 951 (1983). doi: 10.2307/1129899 CrossRefGoogle Scholar
  18. 18.
    Özyurt, O., Özyurt, H., Baki, A., Güven, B.: Integration into mathematics classrooms of an adaptive and intelligent individualized e-learning environment: Implementation and evaluation of UZWEBMAT. Comput. Human Behav. 29(3), 726–738 (2013). doi: 10.1016/j.chb.2012.11.013 CrossRefGoogle Scholar
  19. 19.
    Pedersen, C., Togelius, J., Yannakakis, G.N.: Modeling player experience in super mario bros. In: 2009 IEEE Symposium on Computational Intelligence and Games, Institute of Electrical and Electronics Engineers (IEEE), Piscataway, NJ, USA, CIG’09, pp. 132–139 (2009). doi: 10.1109/cig.2009.5286482
  20. 20.
    Radošević, D., Orehovački, T., Stapić, Z.: Automatic on-line generation of student’s exercises in teaching programming. In: Radošević, D., Orehovački, T., Stapić, Z. (eds.) “Automatic On-line Generation of Students Exercises in Teaching Programming”, Central European Conference on Information and Intelligent Systems, CECIIS (2010)Google Scholar
  21. 21.
    Ravi, G.A., Sosnovsky, S.: Exercise difficulty calibration based on student log mining. In: Mdritscher, F., Luengo, V., Lai-Chong Law, E., U, H. (eds.) Proceedings of DAILE’13: Workshop on Data Analysis and Interpretation for Learning Environments, Villard-de-Lans, France (2013).
  22. 22.
    Sadigh, D., Seshia, S.A., Gupta, M.: Automating exercise generation: a step towards meeting the MOOC challenge for embedded systems. In: Proceedings of Workshop on Embedded Systems Education (WESE) (2012)Google Scholar
  23. 23.
    Saldana, J., Marfia, G., Roccetti, M.: First person shooters on the road: Leveraging on aps and vanets for a quality gaming experience. In: 2012 IFIP Wireless Days. IEEE, pp. 1–6 (2012). doi: 10.1109/wd.2012.6402812
  24. 24.
    Sampayo-Vargas, S., Cope, C.J., He, Z., Byrne, G.J.: The effectiveness of adaptive difficulty adjustments on students’ motivation and learning in an educational computer game. Comput. Educ. 69, 452–462 (2013). doi: 10.1016/j.compedu.2013.07.004 CrossRefGoogle Scholar
  25. 25.
    Sangineto, E., Capuano, N., Gaeta, M., Micarelli, A.: Adaptive course generation through learning styles representation. Univ. Access Inf. Soc. 7(1–2), 1–23 (2007). doi: 10.1007/s10209-007-0101-0 Google Scholar
  26. 26.
    Schell, J.: The Art of Game Design: A Book of Lenses. CRC Press, San Francisco, CA, USA (2008)Google Scholar
  27. 27.
    Soflano, M., Connolly, T.M., Hainey, T.: Learning style analysis in adaptive GBL application to teach SQL. Comput. Educ. 86, 105–119 (2015). doi: 10.1016/j.compedu.2015.02.009 CrossRefGoogle Scholar
  28. 28.
    Villagrá-Arnedo, C., Gallego-Durán, F.J., Molina-Carmona, R., Llorens-Largo, F.: PLMan: towards a gamified learning system. In: Zaphiris, P., Ioannou, A. (eds.) Learning and Collaboration Technologies, LCT 2016, Lecture Notes in Computer Science, vol. 9753, chapter 8, pp. 82–93. Springer Nature, Cham (2016). doi: 10.1007/978-3-319-39483-1_8
  29. 29.
    Yang, T.C., Hwang, G.J., Yang, S.J.H.: Development of an adaptive learning system with multiple perspectives based on students' learning styles and cognitive styles. Educ. Technol. Soc. 16(4), 185–200 (2013).  

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  • Francisco J. Gallego-Durán
    • 1
  • Rafael Molina-Carmona
    • 1
  • Faraón Llorens-Largo
    • 1
  1. 1.Cátedra Santander-UA de Transformación DigitalUniversidad de AlicanteAlicanteSpain

Personalised recommendations