Skip to main content
SpringerLink
Log in

Implementing a Learning Analytics Intervention and Evaluation Framework: What Works?

  • Chapter
  • First Online:
Big Data and Learning Analytics in Higher Education

Abstract

Substantial progress in learning analytics research has been made in recent years to predict which groups of learners are at risk. In this chapter, we argue that the largest challenge for learning analytics research and practice still lies ahead of us: using learning analytics modelling, which types of interventions have a positive impact on learners’ Attitudes, Behaviour and Cognition (ABC). Two embedded case-studies in social science and science are discussed, whereby notions of evidence-based research are illustrated by scenarios (quasi-experimental, A/B-testing, RCT) to evaluate the impact of interventions. Finally, we discuss how a Learning Analytics Intervention and Evaluation Framework (LA-IEF) is currently being implemented at the Open University UK using principles of design-based research and evidence-based research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Agudo-Peregrina, Á. F., Iglesias-Pradas, S., Conde-González, M. Á., & Hernández-García, Á. (2014). Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Computers in Human Behavior, 31(February), 542–550. doi:10.1016/j.chb.2013.05.031.

    Article  Google Scholar 

  • Aguiar, E., Chawla, N. V., Brockman, J., Ambrose, G. A., & Goodrich, V. (2014). Engagement vs performance: Using electronic portfolios to predict first semester engineering student retention. Paper presented at the Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, Indianapolis, IN.

    Google Scholar 

  • Arbaugh, J. B. (2005). Is there an optimal design for on-line MBA courses? Academy of Management Learning and Education, 4(2), 135–149. doi:10.5465/AMLE.2005.17268561.

    Article  Google Scholar 

  • Arbaugh, J. B. (2014). System, scholar, or students? Which most influences online MBA course effectiveness? Journal of Computer Assisted Learning, 30(4), 349–362. doi:10.1111/jcal.12048.

    Article  Google Scholar 

  • Baker, R. S. (2010). Data mining for education. International Encyclopedia of Education, 7, 112–118.

    Article  Google Scholar 

  • Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief (pp. 1–57). US Department of Education, Office of Educational Technology.

    Google Scholar 

  • Calvert, C. E. (2014). Developing a model and applications for probabilities of student success: A case study of predictive analytics. Open Learning: The Journal of Open, Distance and E-Learning, 29(2), 160–173. doi:10.1080/02680513.2014.931805.

    Article  Google Scholar 

  • Clow, D. (2014). Data wranglers: Human interpreters to help close the feedback loop. Paper presented at the Proceedings of the Fourth International Conference on Learning Analytics and Knowledge.

    Google Scholar 

  • Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. The Journal of the Learning Sciences, 13(1), 15–42.

    Article  Google Scholar 

  • Conde, M. Á., & Hernández-García, Á. (2015). Learning analytics for educational decision making. Computers in Human Behavior, 47, 1–3. doi:10.1016/j.chb.2014.12.034.

    Article  Google Scholar 

  • Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5), 304–317. doi:10.1504/ijtel.2012.051816.

    Article  Google Scholar 

  • Ferguson, R., & Buckingham Shum, S. (2012). Social learning analytics: Five approaches. Paper presented at the Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, Vancouver, BC.

    Google Scholar 

  • García-Peñalvo, F. J., Conde, M. Á., Alier, M., & Casany, M. J. (2011). Opening learning management systems to personal learning environments. Journal of Universal Computer Science, 17(9), 1222–1240. doi:10.3217/jucs-017-09-1222.

    Google Scholar 

  • Gasevic, D., Zouaq, A., & Janzen, R. (2013). “Choose your classmates, your GPA is at stake!”: The association of cross-class social ties and academic performance. American Behavioral Scientist, 57(10), 1460–1479. doi:10.1177/0002764213479362.

    Article  Google Scholar 

  • Giesbers, B., Rienties, B., Tempelaar, D. T., & Gijselaers, W. H. (2013). Investigating the relations between motivation, tool use, participation, and performance in an e-learning course using web-videoconferencing. Computers in Human Behavior, 29(1), 285–292. doi:10.1016/j.chb.2012.09.005.

    Article  Google Scholar 

  • González-Torres, A., García-Peñalvo, F. J., & Therón, R. (2013). Human–computer interaction in evolutionary visual software analytics. Computers in Human Behavior, 29(2), 486–495. doi:10.1016/j.chb.2012.01.013.

    Article  Google Scholar 

  • Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Journal of Educational Technology & Society, 15(3), 42–57.

    Google Scholar 

  • Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York: Routledge.

    Google Scholar 

  • Hess, F. M., & Saxberg, B. (2013). Breakthrough leadership in the digital age: Using learning science to reboot schooling. Thousand Oaks, CA: Corwin Press.

    Google Scholar 

  • Hickey, D. T., Kelley, T. A., & Shen, X. (2014). Small to big before massive: Scaling up participatory learning analytics. Paper presented at the Proceedings of the Fourth International Conference on Learning Analytics and Knowledge.

    Google Scholar 

  • Inkelaar, T., & Simpson, O. (2015). Challenging the ‘distance education deficit’ through ‘motivational emails’. Open Learning: The Journal of Open, Distance and E-Learning, 30(2), 152–163. doi:10.1080/02680513.2015.1055718.

    Article  Google Scholar 

  • Jindal-Snape, D., & Rienties, B. (Eds.). (2016). Multi-dimensional transitions of international students to higher education. London: Routledge.

    Google Scholar 

  • Knight, S., Buckingham Shum, S., & Littleton, K. (2013). Epistemology, pedagogy, assessment and learning analytics. Paper presented at the Proceedings of the Third International Conference on Learning Analytics and Knowledge.

    Google Scholar 

  • Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. doi:10.1016/j.compedu.2009.09.008.

    Article  Google Scholar 

  • MacNeill, S., Campbell, L. M., & Hawksey, M. (2014). Analytics for education. Journal of Interactive Media in Education, 2014(1), 7. doi:10.5334/2014-07.

    Article  Google Scholar 

  • Martin, A. J. (2007). Examining a multidimensional model of student motivation and engagement using a construct validation approach. British Journal of Educational Psychology, 77(2), 413–440. doi:10.1348/000709906X118036.

    Article  Google Scholar 

  • McMillan, J. H., & Schumacher, S. (2014). Research in education: Evidence-based inquiry. Harlow: Pearson Higher Ed.

    Google Scholar 

  • Nistor, N., Baltes, B., Dascălu, M., Mihăilă, D., Smeaton, G., & Trăuşan-Matu, Ş. (2014). Participation in virtual academic communities of practice under the influence of technology acceptance and community factors. A learning analytics application. Computers in Human Behavior, 34, 339–344. doi:10.1016/j.chb.2013.10.051.

    Article  Google Scholar 

  • Papamitsiou, Z., & Economides, A. (2014). Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence. Educational Technology & Society, 17(4), 49–64.

    Google Scholar 

  • Papamitsiou, Z., Terzis, V., & Economides, A. (2014). Temporal learning analytics for computer based testing. Paper presented at the Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, Indianapolis, IN.

    Google Scholar 

  • Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., & Perry, R. P. (2011). Measuring emotions in students’ learning and performance: The achievement emotions questionnaire (AEQ). Contemporary Educational Psychology, 36(1), 36–48. doi:10.1016/j.cedpsych.2010.10.002.

    Article  Google Scholar 

  • Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33–40.

    Article  Google Scholar 

  • Richardson, J. T. E. (2012a). The attainment of White and ethnic minority students in distance education. Assessment and Evaluation in Higher Education, 37(4), 393–408. doi:10.1080/02602938.2010.534767.

    Article  Google Scholar 

  • Richardson, J. T. E. (2012b). The role of response biases in the relationship between students’ perceptions of their courses and their approaches to studying in higher education. British Educational Research Journal, 38(3), 399–418. doi:10.1080/01411926.2010.548857.

    Article  Google Scholar 

  • Rienties, B., & Alden Rivers, B. (2014). Measuring and understanding learner emotions: Evidence and prospects (LACE review papers, Vol. 1). Milton Keynes: LACE.

    Google Scholar 

  • Rienties, B., Giesbers, S., Lygo-Baker, S., Ma, S., & Rees, R. (2014). Why some teachers easily learn to use a new virtual learning environment: A technology acceptance perspective. Interactive Learning Environments, 24(3), 539–552. doi:10.1080/10494820.2014.881394.

    Article  Google Scholar 

  • Rienties, B., Giesbers, B., Tempelaar, D. T., Lygo-Baker, S., Segers, M., & Gijselaers, W. H. (2012). The role of scaffolding and motivation in CSCL. Computers & Education, 59(3), 893–906. doi:10.1016/j.compedu.2012.04.010.

    Article  Google Scholar 

  • Rienties, B., Toetenel, L., & Bryan, A. (2015). “Scaling up” learning design: Impact of learning design activities on LMS behavior and performance. Paper presented at the 5th Learning Analytics Knowledge conference, New York.

    Google Scholar 

  • Rienties, B., & Townsend, D. (2012). Integrating ICT in business education: Using TPACK to reflect on two course redesigns. In P. Van den Bossche, W. H. Gijselaers, & R. G. Milter (Eds.), Learning at the crossroads of theory and practice (Vol. 4, pp. 141–156). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. doi:10.1177/0002764213498851.

    Article  Google Scholar 

  • Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the quality and productivity of the higher education sector policy and strategy for systems level deployment of learning analytics. Canberra: Office of Learning and Teaching, Australian Government. Retrieved from http://solaresearch.org/Policy_Strategy_Analytics.pdf.

  • Siroker, D., & Koomen, P. (2013). A/B testing: The most powerful way to turn clicks into customers. Hoboken, NJ: Wiley.

    Google Scholar 

  • Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15–21. doi:10.2307/3594400.

    Article  Google Scholar 

  • Slavin, R. E. (2008). Perspectives on evidence-based research in education—What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5–14. doi:10.3102/0013189X08314117.

    Article  Google Scholar 

  • Stiles, R. J. (2012). Understanding and managing the risks of analytics in higher education: A guide. (pp. 1–46). Educause.

    Google Scholar 

  • Tempelaar, D. T., Niculescu, A., Rienties, B., Giesbers, B., & Gijselaers, W. H. (2012). How achievement emotions impact students’ decisions for online learning, and what precedes those emotions. Internet and Higher Education, 15(3), 161–169. doi:10.1016/j.iheduc.2011.10.003.

    Article  Google Scholar 

  • Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: Learning analytics in a data-rich context. Computers in Human Behavior, 47, 157–167. doi:10.1016/j.chb.2014.05.038.

    Article  Google Scholar 

  • Tobarra, L., Robles-Gómez, A., Ros, S., Hernández, R., & Caminero, A. C. (2014). Analyzing the students’ behavior and relevant topics in virtual learning communities. Computers in Human Behavior, 31, 659–669. doi:10.1016/j.chb.2013.10.001.

    Article  Google Scholar 

  • Torgerson, D. J., & Torgerson, C. (2008). Designing randomised trials in health, education and the social sciences: An introduction. London: Palgrave Macmillan.

    Book  Google Scholar 

  • Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. Paper presented at the Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, Indianapolis, IN.

    Google Scholar 

  • Wolff, A., Zdrahal, Z., Herrmannova, D., Kuzilek, J., & Hlosta, M. (2014). Developing predictive models for early detection of at-risk students on distance learning modules. Workshop: Machine Learning and Learning Analytics. Paper presented at the Learning Analytics and Knowledge, Indianapolis, IN.

    Google Scholar 

  • Wolff, A., Zdrahal, Z., Nikolov, A., & Pantucek, M. (2013). Improving retention: Predicting at-risk students by analysing clicking behaviour in a virtual learning environment. Paper presented at the Proceedings of the Third International Conference on Learning Analytics and Knowledge.

    Google Scholar 

  • Yin, R. K. (2009). Case study research: Design and methods (Vol. 5). Thousand Oaks, CA: Sage.

    Google Scholar 

Download references

Acknowledgement

We would like to thank Prof Belinda Tynan, Kevin Mayles, and Avinash Boroowa from the Learning and Teaching Centre at the Open University UK for their continuous support, and critical feedback to the LA-IEF framework.

Author information

Authors and Affiliations

  1. The Open University, Milton Keynes, UK

    Bart Rienties, Simon Cross & Zdenek Zdrahal

Authors
  1. Bart Rienties
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simon Cross
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zdenek Zdrahal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bart Rienties .

Editor information

Editors and Affiliations

  1. University of Otago, Dunedin, New Zealand

    Ben Kei Daniel

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Rienties, B., Cross, S., Zdrahal, Z. (2017). Implementing a Learning Analytics Intervention and Evaluation Framework: What Works?. In: Kei Daniel, B. (eds) Big Data and Learning Analytics in Higher Education. Springer, Cham. https://doi.org/10.1007/978-3-319-06520-5_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-06520-5_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-06519-9

  • Online ISBN: 978-3-319-06520-5

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation