Advertisement

Opportunities and Challenges in Using Learning Analytics in Learning Design

  • Marcel SchmitzEmail author
  • Evelien van LimbeekEmail author
  • Wolfgang GrellerEmail author
  • Peter SloepEmail author
  • Hendrik DrachslerEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10474)

Abstract

Educational institutions are designing, creating and evaluating courses to optimize learning outcomes for highly diverse student populations. Yet, most of the delivery is still monitored retrospectively with summative evaluation forms. Therefore, improvements to the course design are only implemented at the very end of a course, thus missing to benefit the current cohort. Teachers find it difficult to interpret and plan interventions just-in-time. In this context, Learning Analytics (LA) data streams gathered from ‘authentic’ student learning activities, may provide new opportunities to receive valuable information on the students’ learning behaviors and could be utilized to adjust the learning design already “on the fly” during runtime. We presume that Learning Analytics applied within Learning Design (LD) and presented in a learning dashboard provide opportunities that can lead to more personalized learning experiences, if implemented thoughtfully.

In this paper, we describe opportunities and challenges for using LA in LD. We identify three key opportunities for using LA in LD: (O1) using on demand indicators for evidence based decisions on learning design; (O2) intervening during the run-time of a course; and, (O3) increasing student learning outcomes and satisfaction. In order to benefit from these opportunities, several challenges have to be overcome. Following a thorough literature review, we mapped the identified opportunities and challenges in a conceptual model that considers the interaction of LA in LD.

Keywords

Learning design Learning analytics Learning dashboards Meta-cognitive competences Feedback Reflection 

Notes

Acknowledgements

We would like to thank the SURF Foundation & NRO for supporting the efforts of Marcel Schmitz, Evelien van Limbeek and Hendrik Drachsler under the REFLECTOR project grant.

References

  1. 1.
    Altbach, P., Reisberg, L., Rumbley, L.: Tracking a global academic revolution. Change 42(2), 30–39 (2010)CrossRefGoogle Scholar
  2. 2.
    Field, J.: Lifelong learning and the multigenerational workforce. In: Burke, R.J., Cooper, C.L., Antoniou, A.-S.G. (eds.) The multi-generational and Aging Workforce: Challenges and Opportunities, pp. 311–325. Edward Elgar Publishing, Cheltenham (2015)CrossRefGoogle Scholar
  3. 3.
    Volles, N.: Lifelong learning in the EU: changing conceptualisations, actors, and policies. Stud. High. Educ. 41(2), 343–363 (2014)CrossRefGoogle Scholar
  4. 4.
    Nonis, S., Hudson, G.: Academic performance of college students: influence of time spent studying and working. J. Edu. Bus. 81(3), 151–159 (2006)CrossRefGoogle Scholar
  5. 5.
    Tuononen, T., Parpala, A., Mattsson, M., Lindblom-Ylänne, S.: Work experience in relation to study pace and thesis grade: investigating the mediating role of student learning. High. Educ. 72(1), 41–58 (2016)CrossRefGoogle Scholar
  6. 6.
    Ministerie van Onderwijs, Cultuur & Wetenschap: De waarde(n) van weten: strategische agenda hoger onderwijs en onderzoek 2015–2025. Den Haag (2015)Google Scholar
  7. 7.
    Flavell, J.: Metacognition and cognitive monitoring: a new area of cognitive–developmental inquiry. Am. Psychol. 34(10), 906–911 (1979)CrossRefGoogle Scholar
  8. 8.
    Donald, C., Blake, A., Girault, I., Datt, A., Ramsay, E.: Approaches to learning design: past the head and the hands to the HEART of the matter. Distance Edu. 30(2), 179–199 (2009)CrossRefGoogle Scholar
  9. 9.
    Peterson, C.: Bringing ADDIE to life: instructional design at its best. J. Edu. Multimed. Hypermedia 12(3), 227–241 (2003)Google Scholar
  10. 10.
    Baker, R.: A framework for design and evaluation of internet-based distance learning courses: Phase one–Framework justification design and evaluation. Online J. Distance Learn. Admin. 6(2), 43–51 (2003)Google Scholar
  11. 11.
    Bloom, B., Engelhart, M., Furst, E., Hill, W., Krathwohl, D.: Taxonomy of Educational Objectives: The Classification Of Educational Goals. Handbook I: Cognitive Domain. David McKay Company, New York (1956)Google Scholar
  12. 12.
    Tyler, R.: Basic Principles of Curriculum and Instruction. University of Chicago Press, Chicago (1949)Google Scholar
  13. 13.
    Bundsgaard, J., Hansen, T.: Evaluation of learning materials: a holistic framework. J. Learn. Des. 4(4), 31–45 (2011)Google Scholar
  14. 14.
    Falconer, I., Beetham, H., Oliver, R., Littlejohn, A.: Mod4L final report: representing learning designs. Final report for the JISC-funded MOD4L project. Glasgow (2007)Google Scholar
  15. 15.
    Leacock, T., Nesbit, J.: A framework for evaluating the quality of multimedia learning resources. Edu. Technol. Soc. 10, 44–59 (2007)Google Scholar
  16. 16.
    Falconer, I.: Mediating between practitioner and developer communities: the learning activity design in education experience. Alt-J 15(2), 155–170 (2007)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Masie, E.: Big Learning Data. ASTD Press, Alexandria (2014)Google Scholar
  18. 18.
    Avella, J., Kebritchi, M., Nunn, S., Kanai, T.: Learning analytics methods, benefits, and challenges in higher education: a systematic literature review. Online Learn. J. 20(2), 13–29 (2016)Google Scholar
  19. 19.
    Siemens, G.: Learning analytics: the emergence of a discipline. Am. Behav. Sci. 57(10), 1380–1400 (2013)CrossRefGoogle Scholar
  20. 20.
    Siemens, G., Gasevic, D., Haythornthwaite, C., Dawson, S., Shum, S., Ferguson, R., Duval, E., Verbert, K., Baker, R.: Open learning analytics: an integrated and modularized platform. Educause Rev. 42(4), 53–54 (2007)Google Scholar
  21. 21.
    Cooper, A.: A framework of characteristics for analytics. CETIS Anal. Ser. 1(7), 1–17 (2012)MathSciNetGoogle Scholar
  22. 22.
    Scheffel, M., Drachsler, H., Stoyanov, S., Specht, M.: Quality indicators for learning analytics. Edu. Technol. Soc. 17(4), 124–140 (2014)Google Scholar
  23. 23.
    Greller, W., Drachsler, H.: Translating learning into numbers: a generic framework for learning analytics. Edu. Technol. Soc. 15(3), 42–57 (2012)Google Scholar
  24. 24.
    Ferguson, R.: Learning analytics: drivers, developments and challenges. Int. J. Technol. Enhanced Learn. 4(5/6), 304–317 (2012)CrossRefGoogle Scholar
  25. 25.
    Bichsel, J.: Analytics in Higher Education: Benefits, Barriers, Progress, and Recommendations. Educause Center for Applied Research, Louisville (2012)Google Scholar
  26. 26.
    Colvin, C., Rogers, T., Wade, A., Dawson, S., Gasevic, D., Shum, S., Nelson, K., Alexander, S., Lockyer, L., Kennedy, G., Corrin, L., Fisher, J.: Student retention and learning analytics: a snapshot of Australian practices and a framework for advancement. Australian Office for Learning and Teaching, Sydney (2015)Google Scholar
  27. 27.
    Rienties, B., Toetenel, L.: The impact of 151 learning designs on student satisfaction and performance: social learning (analytics) matters. In: Proceedings of LAK 2016 6th International Conference on Analytics and Knowledge, pp. 339–343 (2016)Google Scholar
  28. 28.
    Clow, D., Cross, S., Ferguson, R.: Evidence hub review. LACE Project, Milton Keynes (2014)Google Scholar
  29. 29.
    Papamitsiou, Z., Economides, A.: Learning analytics and educational data mining in practice: a systematic literature review of empirical evidence. Edu. Technol. Soc. 17(4), 49–64 (2014)Google Scholar
  30. 30.
    Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., Murphy, S.: Analytics4Action evaluation framework: a review of evidence-based learning analytics interventions at the Open University UK. J. Interact. Med. Edu. 1, 2 (2016)Google Scholar
  31. 31.
    Few, S.: Information dashboard design: the effective visual communication of data. O’Reilly Media, Sebastopol (2006)Google Scholar
  32. 32.
    Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.: Learning analytics dashboard applications. Am. Behav. Sci. 57(10), 1500–1509 (2013)CrossRefGoogle Scholar
  33. 33.
    Mor, Y., Ferguson, R., Wasson, B.: Editorial: learning design, teacher inquiry into student learning and learning analytics: a call for action. Br. J. Edu. Technol. 46(2), 221–229 (2015)CrossRefGoogle Scholar
  34. 34.
    Bodily, R., Verbert, K.: Trends and issues in student-facing learning analytics reporting systems research. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, pp. 309–318. ACM, New York (2017)Google Scholar
  35. 35.
    Schwendimann, B., Rodriguez-Triana, M., Vozniuk, A., Prieto, L., Boroujeni, M., Holzer, A., Gillet, D., Dillenbourg, P.: Perceiving learning at a glance: a systematic literature review of learning dashboard research. IEEE Trans. Learn. Technol. 10(1), 30–41 (2017)CrossRefGoogle Scholar
  36. 36.
    Butler, D., Winne, P.: Feedback and self-regulated learning: a theoretical synthesis. Rev. Edu. Res. 65(3), 245–281 (1995)CrossRefGoogle Scholar
  37. 37.
    Winne, P.: Inherent details in self-regulated learning. Edu. Psychol. 30(4), 173–187 (1995)CrossRefGoogle Scholar
  38. 38.
    Zimmerman, B.: Self-regulation involves more than metacognition: a social cognitive perspective. Edu. Psychol. 30(4), 217–221 (1995)CrossRefGoogle Scholar
  39. 39.
    Winne, P.: How software technologies can improve research on learning and bolster school reform. Edu. Psychol. 41(1), 5–17 (2006)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Lockyer, L., Heathcote, E., Dawson, S.: Informing pedagogical action: aligning learning analytics with learning design. Am. Behav. Sci. 57(10), 1439–1459 (2013)CrossRefGoogle Scholar
  41. 41.
    Bakharia, A., Dawson, S.: SNAPP: a bird’s-eye view of temporal participant interaction. In: Proceedings of LAK 2011 1st International Conference on Analytics and Knowledge, pp. 168–173 (2011)Google Scholar
  42. 42.
    Persico, D., Pozzi, F.: Informing learning design with learning analytics to improve teacher inquiry. Br. J. Edu. Technol. 46(2), 230–248 (2015)CrossRefGoogle Scholar
  43. 43.
    Wise, A., Shaffer, D.: Why theory matters more than ever in the age of big data. J. Learn. Anal. 2(2), 5–13 (2015)CrossRefGoogle Scholar
  44. 44.
    Rodríguez-Triana, M., Martínez-Monés, A., Asensio-Pérez, J., Dimitriadis, Y.: Scripting and monitoring meet each other: aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. Br. J. Edu. Technol. 46(2), 330–343 (2015)CrossRefGoogle Scholar
  45. 45.
    Bakharia, A., Corrin, L., Barba, P. De, Kennedy, G., Gasevic, D., Mulder, R., Williams, D., Dawson, S., Lockyer, L.: A conceptual framework linking learning design with learning analytics. In: Proceedings of LAK 2016 6th International Conference on Analytics and Knowledge, pp. 329–338 (2016)Google Scholar
  46. 46.
    Verbert, K., Govaerts, S., Duval, E., Santos, J., Van Assche, F., Parra, G., Klerkx, J.: Learning dashboards: an overview and future research opportunities. Pers. Ubiquit. Comput. 18(6), 1499–1514 (2014)Google Scholar
  47. 47.
    Park, Y., Jo, I.: Development of the learning analytics dashboard to support students’ learning performance. J. Univ. Comput. Sci. 21(1), 110–133 (2015)Google Scholar
  48. 48.
    FitzGerald, E., Kucirkova, N., Jones, A., Cross, S., Ferguson, R., Herodotou, C., Hillaire, G., Scanlon, E.: Dimensions of personalisation in technology-enhanced learning: a framework and implications for design. Br. J. Edu. Technol. (2017, in press)Google Scholar
  49. 49.
    Jivet, I., Scheffel, M., Drachsler, H., Specht, M.: Awareness is not enough: pitfalls of learning analytics dashboards in the educational practice. In: Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017) (in press)Google Scholar
  50. 50.
    Scheffel, M., Drachsler, H., Toisoul, C., Ternier, S., Specht, M.: The proof of the pudding: examining validity and reliability of the evaluation framework for learning analytics. In: Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017) (in press)Google Scholar
  51. 51.
    Koper, R., Olivier, B.: Representing the learning design of units of learning. Edu. Technol. Soc. 7(3), 97–111 (2004)Google Scholar
  52. 52.
    Prinsloo, P., Slade, S.: An elephant in the learning analytics room: the obligation to act. In: LAK 2017 Proceedings of the Seventh International Learning Analytics & Knowledge Conference, pp. 46–55 (2017)Google Scholar
  53. 53.
    Trigwell, K., Prosser, M.: Improving the quality of student learning: the influence of learning context and student approaches to learning on learning outcomes. High. Educ. 22(3), 251–266 (1991)CrossRefGoogle Scholar
  54. 54.
    De La Fuente, J., Sander, P., Martínez-Vicente, J.M., Vera, M., Garzón, A., Fadda, S.: Combined effect of levels in personal self-regulation and regulatory teaching on meta-cognitive, on meta-motivational, and on academic achievement variables in undergraduate students. Front. Psychol. 8, 232 (2017)CrossRefGoogle Scholar
  55. 55.
    Gašević, D., Jovanović, J., Pardo, A., Dawson, S.: Detecting learning strategies with analytics: links with self-reported measures and academic performance. J. Learn. Anal. 4(1), 113–128 (2017)Google Scholar
  56. 56.
    Pardo, A., Martinez-Maldonado, R., Buckingham Shum, S., Schulte, J., McIntyre, S., Gašević, D., Gao, J., Siemens, G.: Connecting data with student support actions in a course: a hands-on tutorial. In: LAK 2017 Proceedings of the Seventh International Learning Analytics & Knowledge Conference, pp. 522–523 (2017)Google Scholar
  57. 57.
    Pintrich, P., Smith, D., Garcia, T., Mckeachie, W.: Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educ. Psychol. Measur. 53(3), 801–813 (1993)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Zuyd University of Applied SciencesHeerlenThe Netherlands
  2. 2.Vienna University of EducationViennaAustria
  3. 3.Open UniversityHeerlenThe Netherlands
  4. 4.Goethe UniversityFrankfurtGermany
  5. 5.German Institute for International Educational Research (DIPF)FrankfurtGermany

Personalised recommendations