Using explanations for recommender systems in learning design settings to enhance teachers’ acceptance and perceived experience

  • Soultana KargaEmail author
  • Maya Satratzemi


The reuse of Learning Designs can bring significant advantages to the educational community such as the diffusion of best teaching practices and the improvement of teaching quality and learning outcomes. Although various tools, including Recommender Systems, have been developed to implement the notion of reusing Learning Designs, their adoption by teachers falls short of expectations. This paper investigates the results of providing explanations for Learning Design recommendations to teachers. To this end, we designed and implemented an explanatory mechanism incorporated into a Recommender System, which propose pre-existing Learning Designs to teachers. We then conducted a user-centric evaluation experiment. Overall, this study provides evidence that explanations should be incorporated into Recommender Systems that propose Learning Designs, as a way of improving the teacher-perceived experience and promoting their wider adoption by teachers. The more teachers accept and adopt Recommender Systems that propose Learning Designs, the more the educational community gains the benefits of reusing Learning Designs.


Explanations Recommender systems Learning design Technology enhanced learning 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Agostinho, S. (2009). Learning Design Representations to Document, Model, and Share Teaching Practice. In L. Lockyer, S. Bennett, S. Agostinho, & B. Harper (Eds.), Handbook of Research on Learning Design and Learning Objects: Issues, Applications, and Technologies (pp. 1–19). Hershey: IGI Global.
  2. Agostinho, S., Bennett, S., Lockyer, L., Jones, J., & Harper, B. (2013). Learning designs as a stimulus and support for teachers’ design practices. In H. Beetham & R. Sharpe (Eds.), Rethinking Pedagogy for a Digital Age: Designing for 21st century (pp. 119–132). New York: Routledge.Google Scholar
  3. Beetham, H., & Sharpe, R. (2007). Rethinking pedagogy for a digital age: designing and delivering e-learning.
  4. Bennett, S., Agostinho, S., & Lockyer, L. (2005). Reusable learning designs in university education. Proceedings of the IASTED International Conference on Education and Technology, ICET 2005, 2005, 102–106. Retrieved from Accessed 27 Nov 2018.
  5. Bennett, S., Thomas, L., Agostinho, S., Lockyer, L., Jones, J., & Harper, B. (2011). Understanding the design context for Australian university teachers: implications for the future of learning design. Learning, Media and Technology, 36(2), 151–167. Scholar
  6. Bennett, S., Agostinho, S., & Lockyer, L. (2017). The process of designing for learning: understanding university teachers’ design work. Educational Technology Research and Development, 65(1), 125–145. Scholar
  7. Cameron, L. (2009). How learning design can illuminate teaching practice. The Future of Learning Design Conference.Google Scholar
  8. Cameron, L., & Campbell, C. (2010). Sharing learning designs that work. In EdMedia: world Conference on Educational Media and Technology (pp. 1914–1919).Google Scholar
  9. Celik, D., & Magoulas, G. D. (2016). A review, timeline, and categorization of learning design tools. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 10013 LNCS, pp. 3–13).
  10. Chen, L., & Wang, F. (2017). Explaining recommendations based on feature sentiments in product reviews. In Proceedings of the 22Nd International Conference on Intelligent User Interfaces (pp. 17–28). New York: ACM.
  11. Conole, G. (2013). Designing for learning in an open world. New York: Springer New York. Retrieved from
  12. Conole, G., & Culver, J. (2010). The design of Cloudworks: applying social networking practice to foster the exchange of learning and teaching ideas and designs. Computers in Education, 54(3), 679–692. Scholar
  13. Conole, G., & Fill, K. (2005). A learning design toolkit to create pedagogically effective learning activities. Journal of Interactive Media in Education, 8, 1–16. Scholar
  14. Cramer, H., Evers, V., Ramlal, S., van Someren, M., Rutledge, L., Stash, N., Aroyo, L., & Wielinga, B. (2008). The effects of transparency on trust in and acceptance of a content-based art recommender. User Modeling and User-Adapted Interaction, 18(5), 455–496. Scholar
  15. Dalziel, J. (2003). Implementing learning design : the learning activity management system (LAMS). Retrieved from Accessed 28 Nov 2018.
  16. Dalziel, J., Conole, G., Wills, S., Walker, S., Bennett ǁ, S., Dobozy, E., et al. (2016). The Larnaca declaration on learning design. Journal of Interactive Media in Education, 2016(7), 1–24. Scholar
  17. Donald, C., Blake, A., Girault, I., Datt, A., & Ramsay, E. (2009). Approaches to learning design: past the head and the hands to the HEART of the matter. Distance Education, 30(2), 179–199. Scholar
  18. Erdt, M., Fernandez, A., & Rensing, C. (2015). Evaluating recommender systems for technology enhanced learning: a quantitative survey. IEEE Transactions on Learning Technologies, 1382(c), 1. Scholar
  19. Fazeli, S., Drachsler, H., Bitter-Rijpkema, M., Brouns, F., van der Vegt, W., & Sloep, P. B. (2017). User-centric evaluation of recommender systems in social learning platforms: accuracy is just the tip of the iceberg. IEEE Transactions on Learning Technologies, 11, 294–306. Scholar
  20. Gedikli, F., Jannach, D., & Ge, M. (2014). How should I explain? A comparison of different explanation types for recommender systems. International Journal of Human-Computer Studies, 72(4), 367–382. Scholar
  21. Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000). Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (pp. 241–250). New York: ACM.
  22. Hernández-Leo, D., Harrer, A., Dodero, J. M., Asensio-Pérez, J. I., & Burgos, D. (2007). A framework for the conceptualization of approaches to create-by-reuse of learning design solutions. Journal of Universal Computer Science, 13(7), 750–760.Google Scholar
  23. Hernández-Leo, D., Chacón, J., Prieto, L. P., Asensio-Pérez, J. I., & Derntl, M. (2013). Towards an integrated learning design environment. In D. Hernández-Leo, T. Ley, R. Klamma, & A. Harrer (Eds.), Scaling up learning for sustained impact (pp. 448–453). Berlin: Springer Berlin Heidelberg.Google Scholar
  24. Hernández-Leo, D., Asensio-Pérez, J. I., Derntl, M., Prieto, L. P., & Chacón, J. (2014). ILDE: community environment for conceptualizing, authoring and deploying learning activities. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 8719 LNCS, pp. 490–493).
  25. Hernández-Leo, D., Asensio-Pérez, J. I., Derntl, M., Pozzi, F., Chacón, J., Prieto, L. P., & Persico, D. (2018). An integrated environment for learning design. Frontiers in ICT, 5(May), 1–19. Scholar
  26. Hinkin, T. R., Tracey, J. B., & Enz, C. A. (1997). Scale construction: developing reliable and valid measurement instruments. Journal of Hospitality and Tourism Research, 21(1), 100–120. Scholar
  27. Jin, X., Zhou, Y., & Mobasher, B. (2005). A maximum entropy web recommendation system : combining collaborative and content features. Intelligence, 24(6), 612–617. Scholar
  28. Karga, S., & Satratzemi, M. (2018). A hybrid recommender system integrated into LAMS for learning designers. Education and Information Technologies, 23(3), 1297–1329.
  29. Koper, R., & Bennett, S. (2008). Learning design: Concepts. In H. Adelsberger, Kinshuk, J. Pawlowski, & D. Sampson (Eds.), (pp. 135–154). Springer Berlin Heidelberg. Retrieved from
  30. Kouki, P., Schaffer, J., Pujara, J., O’Donovan, J., & Getoor, L. (2017). User preferences for hybrid explanations. In Proceedings of the Eleventh ACM Conference on Recommender Systems (pp. 84–88). New York: ACM.
  31. Laurillard, D. (2012). Teaching as a design science. Building pedagogical patterns for learning and technology. Routledge: Taylor & Francis Group.Google Scholar
  32. Laurillard, D., Charlton, P., Craft, B., Dimakopoulos, D., Ljubojevic, D., Magoulas, G., Masterman, E., Pujadas, R., Whitley, E. A., & Whittlestone, K. (2013). A constructionist learning environment for teachers to model learning designs. Journal of Computer Assisted Learning, 29(1), 15–30. Scholar
  33. Lee, S. (2014). Rethinking pedagogy for a digital age: designing for twenty-first century learning. Open Learning: The Journal of Open, Distance and E-Learning, 29(2), 174–176. Scholar
  34. Lockyer, L., Bennett, S., Agostinho, S., & Harper, B. (2009). Handbook of Research on Learning Design and Learning Objects. In L. Lockyer, S. Bennett, S. Agostinho, & B. Harper (Eds.). IGI Global.
  35. Maina, M., Craft, B., & Mor, Y. (2015). The Art & Science of Learning Design. Rotterdam: Sense Publishers. Retrieved from Accessed 27 Nov 2018.
  36. McNee, S., Lam, S., Konstan, J., & Riedl, J. (2003). Interfaces for Eliciting New User Preferences in Recommender Systems. User Modeling 2003, 2702, 148.
  37. McNee, S. M., Riedl, J., & Konstan, J. a. (2006). Being accurate is not enough: How accuracy metrics have hurt recommender systems. CHI’06 Extended Abstracts on Human Factors in Computing Systems, 1101.
  38. Mor, Y., & Craft, B. (2012). Learning design: reflections on a snapshot of the current landscape. Research in Learning Technology, 20. Retrieved from Accessed 27 Nov 2018.
  39. Muhammad, K. I., Lawlor, A., & Smyth, B. (2016). A live-user study of opinionated explanations for recommender systems. In Proceedings of the 21st International Conference on Intelligent User Interfaces (pp. 256–260). New York: ACM.
  40. Muñoz, O. R., Penalba, F. A., & Sánchez, J. F. (2016). The skills, competences, and attitude toward information and communications technology recommender system: an online support program for teachers with personalized recommendations. New Review of Hypermedia and Multimedia, 22(1–2), 83–110. Scholar
  41. O’Donovan, J., Smyth, B., Gretarsson, B., Bostandjiev, S., & Höllerer, T. (2008). PeerChooser: visual interactive recommendation. In SIGCHI Conference on Human Factors in Computing Systems (pp. 1085–1088). Florence: ACM.
  42. Paquette, G., Teja, I. D. la, & K. Lundgren-Cayrol. (2006). Learning design classification definitions. Retrieved from Accessed 27 Nov 2018.
  43. Prieto, L., Dimitriadis, Y., Craft, B., Derntl, M., Émin, V., Katsamani, M., … Villasclaras, E. (2013). Learning design Rashomon II: Exploring one lesson through multiple tools. Research in Learning Technology, 21(0).
  44. Pu, P., & Chen, L. (2006). Trust building with explanation interfaces. In Proceedings of the 11th International Conference on Intelligent User Interfaces (pp. 93–100). New York: ACM.
  45. Pu, P., & Chen, L. (2011). A user - centric evaluation framework for recommender systems. Proceedings of the 5th ACM Conference on Recommender Systems - RecSys ‘11, 157–164.
  46. Pu, P., Faltings, B., Chen, L., Zhang, J., & Viappiani, P. (2011). Usability guidelines for product recommenders based on example critiquing research. In Recommender Systems Handbook (pp. 511–545). Scholar
  47. Quijano-Sanchez, L., Sauer, C., Recio-Garcia, J. A., & Diaz-Agudo, B. (2017). Make it personal: A social explanation system applied to group recommendations. Expert Systems with Applications, 76, 36–48.
  48. Resnick, P., & Varian, H. R. (1997). Recommender systems. Communications of the ACM, 40(3), 56–58. Scholar
  49. Salter, J., & Antonopoulos, N. (2006). CinemaScreen recommander agent: Combining collaborative and content-based filtering. IEEE Intelligent Systems, 21, 35–41. Scholar
  50. Santos, O. C., & Boticario, J. G. (2011). Requirements for semantic educational recommender systems in formal e-learning scenarios. Algorithms, 4(2), 131–154. Scholar
  51. Simon, B., Pulkkinen, M., Totschnig, M., & Kozlov, D. (2011). The ICOPER reference model for outcome-based higher education. Retrieved from Accessed 27 Nov 2018.
  52. Symeonidis, P., Nanopoulos, A., & Manolopoulos, Y. (2008). Providing justifications in recommender systems. Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans, 38(6), 1262–1272. Scholar
  53. Tintarev, N. 2010. Explaining Recommendations. PhD Thesis, University of Aberdeen.Google Scholar
  54. Tintarev, N., & Masthoff, J. (2011). Designing and evaluating explanations for recommender systems. In F. Ricci, L. Rokach, B. Shapira, & P. B. Kantor (Eds.), (pp. 479–510). Springer US. Retrieved from
  55. Tintarev, N., & Masthoff, J. (2012). Evaluating the effectiveness of explanations for recommender systems. User Modeling and User-Adapted Interaction, 22(4), 399–439. Scholar
  56. Tintarev, N., & Masthoff, J. (2015). Explaining recommendations: Design and evaluation. In Recommender Systems Handbook (2nd ed., pp. 353–382). Scholar
  57. Verbert, K., Ochoa, X., Derntl, M., Wolpers, M., Pardo, A., & Duval, E. (2012). Semi-automatic assembly of learning resources. Computers & Education, 59(4), 1257–1272. Scholar
  58. Weller, M. (2004). Learning objects and the e-learning cost dilemma. Journal of Open, Distance and E-Learning, 19(3), 37–41. Scholar
  59. Wills, S., & McDougall, A. (2009). Reusability of online role play: Learning objects or learning designs? In L. Lockyer, S. Bennett, S. Agostinho, & B. Harper (Eds.), Handbook of research on learning design and learning objects: Issues, applications, and technologies (p. ??). Information Science Reference. Retrieved from Accessed 27 Nov 2018.
  60. Wills, S., & Pegler, C. (2016). A deeper understanding of reuse: learning designs, activities, resources and their contexts. Journal of Interactive Media in Education, 2016(1), 1–11. Scholar
  61. Yoo, K. H., & Gretzel, U. (2011). Creating more credible and persuasive recommender systems: the influence of source characteristics on recommender system evaluations. Recommender Systems Handbook, 455–477.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Applied InformaticsUniversity of MacedoniaThessalonikiGreece

Personalised recommendations