Abstract
While learning analytics (LA) is maturing from being a trend to being part of the institutional toolbox, the need for more empirical evidences about the effects for LA on the actual stakeholders, i.e. learners and teachers, is increasing. Within this paper we report about a further evaluation iteration of the Evaluation Framework for Learning Analytics (EFLA) that provides an efficient and effective measure to get insights into the application of LA in educational institutes. For this empirical study we have thus developed and implemented several LA widgets into a MOOC platform’s dashboard and evaluated these widgets using the EFLA as well as the framework itself using principal component and reliability analysis. The results show that the EFLA is able to measure differences between widget versions. Furthermore, they indicate that the framework is highly reliable after slightly adapting its dimensions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Arnold, K.E., Lonn, S., Pistilli, M.D.: An exercise in institutional reflection: the learning analytics readiness instrument (LARI). In: Proceedings of the 4th International Conference on Learning Analytics and Knowledge, LAK 2014, pp. 163–167. ACM, New York (2014)
Beheshitha, S., Hatala, M., Gašević, D., Joksimovic, S.: The role of achievement goal orientations when studying effect of learning analytics visualizations. In: Proceedings of the 6th International Conference on Learning Analytics and Knowledge, LAK 2016, pp. 54–63. ACM, New York (2016)
Brooke, J.: SUS: a quick and dirty usability scale. In: Jordan, P.W., Weerdmeester, B., Thomas, A., Mclelland, I.L. (eds.) Usability evaluation in industry. Taylor and Francis, London (1996)
Butler, D., Winne, P.: Feedback and self-regulated learning: a theoretical synthesis. Rev. Educ. Res. 65(3), 245–281 (1995)
Cobo, A., Rocha, R., Rodriguez-Hoyos, C.: Evaluation of the interactivity of students in virtual learning environments using a multicriteria approach and data mining. Behav. Inf. Technol. 33(10), 1000–1012 (2014)
Drachsler, H., Kalz, M.: The MOOC and learning analytics innovation cycle (MOLAC): a reflective summary of ongoing research and its challenges. J. Comput. Assist. Learn. 32(3), 281–290 (2016)
Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors 37(1), 32–64 (1995)
Ferguson, R., Clow, D.: Learning analytics community exchange: evidence hub. In: Proceedings of the 6th International Conference on Learning Analytics and Knowledge, LAK 2016, pp. 520–521. ACM, New York (2016)
Gašević, D., Dawson, S., Mirriahi, N., Long, P.: Learning analytics - a growing field and community engagement. J. Learn. Anal. 2(1), 1–6 (2015)
Gašević, D., Dawson, S., Siemens, G.: Let’s not forget: learning analytics are about learning. TechTrends 59(1), 64–71 (2015)
Greller, W., Drachsler, H.: Translating learning into numbers: a generic framework for learning analytics. Educ. Technol. Soc. 15(3), 42–57 (2012)
Khan, I., Pardo, A.: Data2U: scalable real time student feedback in active learning environments. In: Proceedings of the 6th International Conference on Learning Analytics and Knowledge, LAK 2016, pp. 249–253. ACM, New York (2016)
Kim, J., Jo, I.H., Park, Y.: Effects of learning analytics dashboard: analyzing the relations among dashboard utilization, satisfaction, and learning achievement. Asia Pac. Educ. Rev. 17(1), 13–24 (2016)
Long, P., Siemens, G.: Penetrating the fog: analytics in learning and education. EDUCAUSE Rev. 46(5), 31–40 (2011)
Lonn, S., Aguilar, S., Teasley, S.: Investigating student motivation in the context of a learning analytics intervention during a summer bridge program. Comput. Hum. Behav. 47, 90–97 (2015)
Persico, D., Pozzi, F.: Informing learning design with learning analytics to improve teacher inquiry. Br. J. Educ. Technol. 46(2), 230–248 (2014)
Scheffel, M., Drachlser, H., Specht, M.: Developing an evaluation framework of quality indicators for learning analytics. In: Proceedings of the 5th International Conference on Learning Analytics and Knowledge, LAK 2015, pp. 16–20. ACM, New York (2015)
Scheffel, M., Drachsler, H., Kreijns, K., de Kraker, J., Specht, M.: Widget, widget as you lead, i am performing well indeed!: using results from an exploratory offline study to inform an empirical online study about a learning analytics widget in a collaborative learning environment. In: Proceedings of the 7th International Conference on Learning Analytics and Knowledge, LAK 2017, pp. 289–298. ACM, New York (2017)
Scheffel, M., Drachsler, H., Stoyanov, S., Specht, M.: Quality indicators for learning analytics. Educ. Technol. Soc. 17(4), 117–132 (2014)
Schön, D.: The Reflective Practitioner: How Professionals Think in Action. Temple Smith, London (1983)
Siemens, G., Dawson, S., Lynch, G.: Improving the quality and productivity of the higher education sector - policy and strategy for system-level deployment of learning analytics. Discussion paper for the Australian Government, Society for Learning Analytics Research (SoLAR) (2013)
Verbert, K., Govaerts, S., Duval, E., Santos, J.L., Assche, F., Parra, G., Klerkx, J.: Learning dashboards: an overview and future research opportunities. Pers. Ubiquit. Comput. 18(6), 1499–1514 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Scheffel, M., Drachsler, H., Toisoul, C., Ternier, S., Specht, M. (2017). The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics. In: Lavoué, É., Drachsler, H., Verbert, K., Broisin, J., Pérez-Sanagustín, M. (eds) Data Driven Approaches in Digital Education. EC-TEL 2017. Lecture Notes in Computer Science(), vol 10474. Springer, Cham. https://doi.org/10.1007/978-3-319-66610-5_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-66610-5_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-66609-9
Online ISBN: 978-3-319-66610-5
eBook Packages: Computer ScienceComputer Science (R0)