Facilitating Student Success in Introductory Chemistry with Feedback in an Online Platform
Instructional technologists and faculty in post-secondary institutions have increasingly adopted learning analytics interventions such as dashboards that provide real-time feedback to students to support student’ ability to regulate their learning. But analyses of the effectiveness of such interventions can be confounded by measures of students’ prior learning as well as their baseline level of self-regulated learning. For this research study, we sought to examine whether the frequency of accessing a dashboard was associated with learning outcomes after matching subjects on confounding variables. And because prior research has suggested that measures of prior learning are associated with students’ likelihood to use learning analytics interventions, we sought to adequately control for learners’ likelihood to access the feedback by using a propensity score matching with a non-binary treatment variable. We administered the Motivated Strategies for Learning Questionnaire and also collected demographic information for a propensity score matching process. Users’ frequency of accessing the intervention was categorized as High, Moderate, or Low/No usage. After matching users on characteristics associated with dashboard usage (gender, high school GPA, and the “Test Anxiety” and “Self Efficacy” factors) we found that both the “High” and “Moderate” users achieved significantly higher course grades than the “Low/No” users. The results suggest learners benefited from regularly accessing the feedback, but extreme amounts of usage were not necessary to achieve a positive effect. We discuss the implications for recommending how students use learning analytics interventions without excessively accessing feedback.
KeywordsLearning analytics Feedback Self-regulated learners Propensity score matching Motivation
The authors would like to acknowledge the University of Iowa’s Academic Technology Advisory Committee, which awarded a grant that assisted with the research and development of Elements of Success.
Compliance with Ethical Standards
Conflict of interest
The authors declare that they have no conflicts of interest.
- Agudo-Peregrina, Á. F., Iglesias-Pradas, S., Conde-González, M. Á., & Hernández-García, Á. (2014). Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2013.05.031.Google Scholar
- Bryer, J. M. (2013). TriMatch: An R package for propensity score matching of non-binary treatments. In The R user conference, useR! 2013 July 10–12, 2013 University of Castilla-La Mancha, Albacete, Spain (Vol. 10, No. 30, p. 34).Google Scholar
- Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE Review, 42(4), 40–57.Google Scholar
- Devolder, A., van Braak, J., & Tondeur, J. (2012). Supporting self-regulated learning in computer-based learning environments: Systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning. https://doi.org/10.1111/j.1365-2729.2011.00476.x.Google Scholar
- Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012). The student activity meter for awareness and self-reflection. In Extended abstracts on human factors in computing systems (pp. 869–884). ACM.Google Scholar
- Griffin, T. D., Wiley, J., & Salas, C. R. (2013). Supporting effective self-regulated learning: The critical role of monitoring. In International handbook of metacognition and learning technologies (pp. 19–34). New York: Springer.Google Scholar
- Pintrich, P. R., & Zusho, A. (2002). The development of academic self-regulation: The role of cognitive and motivational factors. In A. Wigfield & J. S. Eccles (Eds.), Development of achievement motivation: A volume in the educational psychology series (pp. 249–284). San Diego, CA: Academic Press.CrossRefGoogle Scholar
- Roll, I., & Winne, P. (2015). Understanding, evaluating, and supporting self-regulated learning using learning analytics. Journal of Learning Analytics, 2(1), 7–12. Retrieved November 8, 2016 from http://learning-analytics.info/journals/index.php/JLA/article/view/4491/4825.
- Trevors, G., Feyzi-Behnagh, R., Azevedo, R., & Bouchet, F. (2016). Self-regulated learning processes vary as a function of epistemic beliefs and contexts: Mixed method evidence from eye tracking and concurrent and retrospective reports. Learning and Instruction, 42, 31–46. https://doi.org/10.1016/j.learninstruc.2015.11.003.CrossRefGoogle Scholar
- Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. Metacognition in Educational Theory and Practice, 93, 27–30.Google Scholar
- Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. In Proceedings of the fourth international conference on learning analytics and knowledge (pp. 203–211). ACM.Google Scholar
- Wise, A. F. (2016). Data-informed learning environments. EDUCAUSE Review, Retrieved on November 10, 2016 from http://er.educause.edu/articles/2016/10/data-informed-learning-environments.