Advertisement

Learning Analytics for Smart Learning Environments: A Meta-Analysis of Empirical Research Results from 2009 to 2015

  • Zacharoula Papamitsiou
  • Anastasios A. Economides
Living reference work entry

Abstract

Although several qualitative analyses appeared in the domain of Learning Analytics (LA), a systematic quantitative analysis of the effects of the empirical research findings toward the development of more reliable Smart Learning Environments (SLE) is still missing. This chapter aims at preserving and enhancing the chronicles of recent LA developments as well as covering the abovementioned gap. The core question is where these two research areas intersect and how the significant LA research findings could be beneficial for guiding the construction of SLEs. This meta-analysis study synthesizes research on the effectiveness of LA and targets at determining the influence of its dimensions on learning outcomes so far. Sixty-six experimental and quasi-experimental papers published from 2009 through September 2015 in the domain of LA were coded and analyzed. Overall, the weighted random effects mean effect size (g) was 0.433 (p = 0.001). The collection was heterogeneous (Qt(66) = 78.47). Here, the results of the statistical and classification processes applied during the meta-analysis process are presented and the most important issues raised are discussed.

Keywords

Learning analytics Smart-learning environments Meta-analysis review Systematic review Effectiveness Classification of research papers 

References

  1. Agudo-Peregrina, Á. F., Iglesias-Pradas, S., Conde-González, M. Á., & Hernández-García, Á. (2015). Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Computers in Human Behavior, 31, 542–550.CrossRefGoogle Scholar
  2. Ali, L., Asadi, M., Gašević, D., Jovanović, J., & Hatala, M. (2013). Factors influencing beliefs for adoption of a learning analytics tool: An empirical study. Computers & Education, 62, 130–148.CrossRefGoogle Scholar
  3. Ali, L., Hatala, M., Gašević, D., & Jovanović, J. (2012). A qualitative evaluation of evolution of a learning analytics tool. Computers & Education, 58(1), 470–489.CrossRefGoogle Scholar
  4. Aramo-Immonen, H., Jussila, J., & Huhtamäki, J. (2015). Exploring co-learning behavior of conference participants with visual network analysis of Twitter data. Computers in Human Behavior, 51, 1154–1162.CrossRefGoogle Scholar
  5. Baalsrud Hauge, J., Stanescu, I. A., Moreno-Ger, P., Arnab, S., Lim, T., Serrano-Laguna, A., … Degano, C. (2015). Learning analytics architecture to scaffold learning experience through technology-based methods. International Journal of Serious Games, 2(1), 29–44.Google Scholar
  6. Baker, R. S. J., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3–17.Google Scholar
  7. Barla, M., Bieliková, M., Ezzeddinne, A. B., Kramár, T., Šimko, M., & Vozár, O. (2010). On the impact of adaptive test question selection for learning efficiency. Computers & Education, 55(2), 846–857.CrossRefGoogle Scholar
  8. Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Interaction, 2013, 1–11.Google Scholar
  9. Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief (pp. 1–57). Washington, DC: US Department of Education, Office of Educational Technology.Google Scholar
  10. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2006). Comprehensive meta-analysis [computer software]. Englewood, NJ: Biostat. https://www.meta-analysis.com/
  11. Cambruzzi, W. L., Rigo, S. J., & Barbosa, J. L. V. (2015). Dropout prediction and reduction in distance education courses with the learning analytics multitrail approach. Journal of Universal Computer Science, 21(1), 23–47.Google Scholar
  12. Chen, C.-M., & Chen, M.-C. (2009). Mobile formative assessment tool based on data mining techniques for supporting web-based learning. Computers & Education, 52(1), 256–273.CrossRefGoogle Scholar
  13. Cheung Kong, S., & Song, Y. (2015). An experience of personalized learning hub initiative embedding BYOD for reflective engagement in higher education. Computers & Education, 88, 227–240.CrossRefGoogle Scholar
  14. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
  15. Cooper, H. (1989). Homework. White Plains, NY: Longman.CrossRefGoogle Scholar
  16. Cooper, H. (2010). Research synthesis and meta analysis: A step by step approach (Applied Social Research Methods Series 4th ed., Vol. 2). Thousand Oaks, CA: Sage.Google Scholar
  17. Credé, M., & Niehorster, S. (2012). Adjustment to college as measured by the student adaptation to college questionnaire: A quantitative review of its structure and relationships with correlates and consequences. Educational Psychology Review, 24(1), 133–165.CrossRefGoogle Scholar
  18. D’Mello, S. K. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105(4), 1082–1099.CrossRefGoogle Scholar
  19. Dejaeger, K., Goethals, F., Giangreco, A., Mola, L., & Baesens, B. (2012). Gaining insight into student satisfaction using comprehensible data mining techniques. European Journal of Operational Research, 218(2), 548–562.CrossRefGoogle Scholar
  20. Demmans Epp, C., Bull, S. (2015). Uncertainty representation in visualizations of learning analytics for learners: Current approaches and opportunities. TLT, 8(3), 242–260.Google Scholar
  21. Economides, A. A. (2009). Adaptive context-aware pervasive and ubiquitous learning. International Journal of Technology Enhanced Learning, 1(3), 169–192.CrossRefGoogle Scholar
  22. Feidakis, M., Daradoumis, T., Caballe, S., & Conesca, J. (2013). Measuring the impact of emotion awareness on elearning situations. Proceedings of the Seventh International Conference on Complex, Intelligent, and Software Intensive Systems.Google Scholar
  23. Ferguson, R. (2012). The state of learning analytics in 2012: A review and future challenges (Technical Report KMI-2012). Retrieved from http://kmi.open.ac.uk/publications/techreport/kmi-12-01
  24. Fidalgo-Blanco, Á., Sein-Echaluce, M. L., García-Peñalvo, F. J., & Conde, M. Á. (2015). Using learning analytics to improve teamwork assessment. Computers in Human Behavior, 47, 149–156.CrossRefGoogle Scholar
  25. Fulantelli, G., Taibi, D., & Arrigo, M. (2015). A framework to support educational decision making in mobile learning. Computers in Human Behavior, 47, 50–59.CrossRefGoogle Scholar
  26. Giesbers, B., Rienties, B., Tempelaar, D., & Gijselaers, W. (2013). Investigating the relations between motivation, tool use, participation, and performance in an e-learning course using web-videoconferencing. Computers in Human Behavior, 29(1), 285–292.CrossRefGoogle Scholar
  27. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.CrossRefGoogle Scholar
  28. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.Google Scholar
  29. Guo, W. W. (2010). Incorporating statistical and neural network approaches for student course satisfaction analysis and prediction. Expert Systems with Applications, 37(4), 3358–3365.CrossRefGoogle Scholar
  30. Guruler, H., Istanbullu, A., & Karahasan, M. (2010). A new student performance analysing system using knowledge discovery in higher educational databases. Computers & Education, 55(1), 247–254.CrossRefGoogle Scholar
  31. Hernández-García, A., González-González, I., Jiménez-Zarco, A. I., & Chaparro-Peláez, J. (2015). Applying social learning analytics to message boards in online distance learning: A case study. Computers in Human Behavior, 47, 68–80.CrossRefGoogle Scholar
  32. Hwang, G.-J. (2014). Definition, framework and research issues of smart learning environments – a context-aware ubiquitous learning perspective. Smart Learning Environments Open Journal, 1(4), 1–14.Google Scholar
  33. Joksimovic, S., Gasevic, D., Kovanovic, V., Adesope, O., & Hatala, M. (2014). Psychological characteristics in cognitive presence of communities of inquiry: A linguistic analysis of online discussions. The Internet and Higher Education, 22, 1–10.CrossRefGoogle Scholar
  34. Joksimović, S., Gašević, D., Loughin, T. M., Kovanović, V., & Hatala, M. (2015). Learning at distance: Effects of interaction traces on academic achievement. Computers & Education, 87, 204–217.CrossRefGoogle Scholar
  35. Kim, S., Song, S.-M., & Yoon, Y.-I. (2011). Smart learning services based on smart cloud computing. Sensors, 11(8), 7835–7850.CrossRefGoogle Scholar
  36. Koper, R. (2014). Conditions for effective smart learning environments. Smart Learning Environments Open Journal, 1(5), 1–17.Google Scholar
  37. Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89.CrossRefGoogle Scholar
  38. Lee, J., Zo, H., & Lee, H. (2014). Smart learning adoption in employees and HRD managers. British Journal of Educational Technology, 45(6), 1082–1096.CrossRefGoogle Scholar
  39. Leong, C.-K., Lee, Y.-H., & Mak, W.-K. (2012). Mining sentiments in SMS texts for teaching evaluation. Expert Systems with Applications, 39(3), 2584–2589.CrossRefGoogle Scholar
  40. Lin, C.-F., Yeh, Y., Hung, Y. H., & Chang, R. (2013). Data mining for providing a personalized learning path in creativity: An application of decision trees. Computers & Education, 68, 199–210.CrossRefGoogle Scholar
  41. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.Google Scholar
  42. Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review Online, 46(5), 31–40.Google Scholar
  43. Lykourentzou, I., Giannoukos, I., Nikolopoulos, V., Mpardis, G., & Loumos, V. (2009). Dropout prediction in e-learning courses through the combination of machine learning techniques. Computers & Education, 53(3), 950–965.CrossRefGoogle Scholar
  44. Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599.CrossRefGoogle Scholar
  45. Minović, M., Milovanović, M., Šošević, U., & Conde González, M. Á. (2015). Visualisation of student learning model in serious games. Computers in Human Behavior, 47, 98–107.CrossRefGoogle Scholar
  46. Moissa, B., Gasparini, I., & Kemczinski, A. (2015). A systematic mapping on the learning analytics field and its analysis in the massive open online courses context. International Journal of Distance Education Technologies, 13(3), 1–24.CrossRefGoogle Scholar
  47. Moridis, C. N., & Economides, A. A. (2009). Prediction of student’s mood during an online test using formula-based and neural network-based method. Computers & Education, 53(3), 644–652.CrossRefGoogle Scholar
  48. Moridis, C. N., & Economides, A. A. (2012). Affective learning: Empathetic agents with emotional facial and tone of voice expressions. IEEE Transactions on Affective Computing, 3, 260–272.CrossRefGoogle Scholar
  49. Muñoz-Merino, P. J., Ruipérez-Valiente, J. A., Alario-Hoyos, C., Pérez-Sanagustín, M., & Delgado Kloos, C. (2015). Precise effectiveness strategy for analyzing the effectiveness of students with educational resources and activities in MOOCs. Computers in Human Behavior, 47, 108–118.CrossRefGoogle Scholar
  50. Noh, K. S., Ju, S. H., & Jung, J. T. (2011). An exploratory study on concept and realization conditions of smart learning. Journal of Digital Policy & Management, 9(2), 79–88.Google Scholar
  51. Papamitsiou, Z., & Economides, A. (2014). Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence. Educational Technology & Society, 17(4), 49–64.Google Scholar
  52. Richardson, J. T. E. (2012). The attainment of White and ethnic minority students in distance education. Assessment & Evaluation in Higher Education, 37(4), 393–408.CrossRefGoogle Scholar
  53. Rienties, B. & Rivers, B.A. (2014). Measuring and understanding learner emotions: Evidence and prospects. Learning Analytics Review, 1, 1–30, ISSN: 2057–7494.Google Scholar
  54. Romero, C., & Ventura, S. (2007). Educational data mining: A survey from 1995 to 2005. Expert Systems with Applications, 33(1), 135–146.CrossRefGoogle Scholar
  55. Romero, C., & Ventura, S. (2013). Data mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 3(1), 12–27.Google Scholar
  56. Romero-Zaldivar, V.-A., Pardo, A., Burgos, D., & Kloos, C. D. (2012). Monitoring student progress using virtual appliances: A case study. Computers & Education, 58(4), 1058–1067.CrossRefGoogle Scholar
  57. Rosenthal, R. (1984). Meta-analytic procedures for social research. Newbury Park, CA: Sage.Google Scholar
  58. Scott, K., & Benlamri, R. (2010). Context-aware services for smart learning spaces. IEEE Transactions on Learning Technologies, 3(3), 214–227.CrossRefGoogle Scholar
  59. Serrano-Laguna, A., Torrente, J., Moreno-Ger, P., & Fernández-Manjón, B. (2012). Tracing a little for big improvements: Application of learning analytics and videogames for student assessment. Procedia Computer Science, 15, 203–209.CrossRefGoogle Scholar
  60. Shute, V. (2011). Stealth assessment in computer-based games to support learning. In Computer games and instruction (pp. 503–523), Charlotte, NC: Information Age.Google Scholar
  61. Spector, J. M. (2014). Conceptualizing the emerging field of smart learning environments. Smart Learning Environments, 1(1), 1–10.CrossRefGoogle Scholar
  62. Suthers, D. D., & Verbert, K. (2013). Learning analytics as a “middle space.” In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 1–4). New York, NY: ACM.Google Scholar
  63. Tabuenca, B., Kalz, M., Drachsler, H., & Specht, M. (2015). Time will tell: The role of mobile learning analytics in self-regulated learning. Computers & Education, 89, 53–74.CrossRefGoogle Scholar
  64. Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using signals for appropriate feedback: Perceptions and practices. Computers & Education, 57(4), 2414–2422.CrossRefGoogle Scholar
  65. Tatar, D., Roschelle, J., Vahey, P., & Penuel, W. R. (2003). Handhelds go to school: Lessons learned. IEEE Computer, 36(9), 30–37.CrossRefGoogle Scholar
  66. Tempelaar, D. T., Niculescu, A., Rienties, B., Giesbers, B., & Gijselaers, W. H. (2012). How achievement emotions impact students’ decisions for online learning, and what precedes those emotions. The Internet and Higher Education, 15(3), 161–169.CrossRefGoogle Scholar
  67. Tempelaar, D. T., Rienties, B., & Giesbers, B. (2014). In search for the most informative data for feedback generation: Learning analytics in a data-rich context. Computers in Human Behavior, 47, 157–167.CrossRefGoogle Scholar
  68. Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: Learning analytics in a data-rich context. Computers in Human Behavior, 47, 157–167.CrossRefGoogle Scholar
  69. Terzis, V., Moridis, C. N., & Economides, A. A. (2012). The effect of emotional feedback on behavioral intention to use computer based assessment. Computers & Education, 59, 710–721.CrossRefGoogle Scholar
  70. Traxler, J. (2007). Defining, discussing and evaluating mobile learning: The moving finger writes and having writ … The International Review of Research in Open and Distance Learning, 8(2), 1–12.Google Scholar
  71. van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2014). Supporting teachers in guiding collaborating students: Effects of learning analytics in CSCL. Computers & Education, 79, 28–39.CrossRefGoogle Scholar
  72. van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2015). Teacher regulation of cognitive activities during student collaboration: Effects of learning analytics. Computers & Education, 90, 80–94.CrossRefGoogle Scholar
  73. Veletsianos, G., Collier, A., & Schneider, E. (2015). Digging Deeper into Learners’ Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption. British Journal of Educational Technology, 46(3), 570–587.CrossRefGoogle Scholar
  74. Verbert, K., Manouselis, N., Drachsler, H., & Duval, E. (2012). Dataset-driven research to support learning and knowledge analytics. Journal of Educational Technology & Society, 15(3), 133–148.Google Scholar
  75. Westera, W., Nadolski, R., Hummel, H., & Wopereis, I. (2008). Serious games for higher education: A framework for reducing design complexity. Journal of Computer-Assisted Learning, 24(5), 420–432.CrossRefGoogle Scholar
  76. Westera, W., Nadolski, R., & Hummel, H. (2014). Serious gaming analytics: What students’ log files tell us about gaming and learning. International Journal of Serious Games (to appear). http://journal.seriousgamessociety.org/
  77. Xing, W., Wadholm, R., Petakovic, E., & Goggins, S. (2015). Group learning assessment: Developing a theory-informed analytics. Educational Technology & Society, 18(2), 110–128.Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Zacharoula Papamitsiou
    • 1
  • Anastasios A. Economides
    • 1
  1. 1.Interdepartmental Programme of Postgraduate Studies in Information SystemsUniversity of MacedoniaThessalonikiGreece

Personalised recommendations