Advertisement

Technology, Knowledge and Learning

, Volume 23, Issue 1, pp 21–40 | Cite as

Facilitating Student Success in Introductory Chemistry with Feedback in an Online Platform

  • Sam Van Horne
  • Maura Curran
  • Anna Smith
  • John VanBuren
  • David Zahrieh
  • Russell Larsen
  • Ross Miller
Original research

Abstract

Instructional technologists and faculty in post-secondary institutions have increasingly adopted learning analytics interventions such as dashboards that provide real-time feedback to students to support student’ ability to regulate their learning. But analyses of the effectiveness of such interventions can be confounded by measures of students’ prior learning as well as their baseline level of self-regulated learning. For this research study, we sought to examine whether the frequency of accessing a dashboard was associated with learning outcomes after matching subjects on confounding variables. And because prior research has suggested that measures of prior learning are associated with students’ likelihood to use learning analytics interventions, we sought to adequately control for learners’ likelihood to access the feedback by using a propensity score matching with a non-binary treatment variable. We administered the Motivated Strategies for Learning Questionnaire and also collected demographic information for a propensity score matching process. Users’ frequency of accessing the intervention was categorized as High, Moderate, or Low/No usage. After matching users on characteristics associated with dashboard usage (gender, high school GPA, and the “Test Anxiety” and “Self Efficacy” factors) we found that both the “High” and “Moderate” users achieved significantly higher course grades than the “Low/No” users. The results suggest learners benefited from regularly accessing the feedback, but extreme amounts of usage were not necessary to achieve a positive effect. We discuss the implications for recommending how students use learning analytics interventions without excessively accessing feedback.

Keywords

Learning analytics Feedback Self-regulated learners Propensity score matching Motivation 

Notes

Acknowledgements

The authors would like to acknowledge the University of Iowa’s Academic Technology Advisory Committee, which awarded a grant that assisted with the research and development of Elements of Success.

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflicts of interest.

References

  1. Agudo-Peregrina, Á. F., Iglesias-Pradas, S., Conde-González, M. Á., & Hernández-García, Á. (2014). Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Computers in Human Behavior.  https://doi.org/10.1016/j.chb.2013.05.031.Google Scholar
  2. Austin, P. C. (2007). Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: a systematic review and suggestions for improvement. The Journal of Thoracic and Cardiovascular Surgery.  https://doi.org/10.1016/j.jtcvs.2007.07.021.Google Scholar
  3. Austin, P. C. (2011). Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies. Pharmaceutical Statistics.  https://doi.org/10.1002/pst.433.Google Scholar
  4. Baker, R. S. (2016). Stupid tutoring systems, intelligent humans. International Journal of Artificial Intelligence in Education.  https://doi.org/10.1007/s40593-016-0105-0.Google Scholar
  5. Bentler, P. M., & Bonett, D. (1980). Significance tests and goodness-of-fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–600.CrossRefGoogle Scholar
  6. Bernacki, M. L., Byrnes, J. P., & Cromley, J. G. (2012). The effects of achievement goals and self-regulated learning behaviors on reading comprehension in technology-enhanced learning environments. Contemporary Educational Psychology.  https://doi.org/10.1016/j.cedpsych.2011.12.001.Google Scholar
  7. Bryer, J. M. (2013). TriMatch: An R package for propensity score matching of non-binary treatments. In The R user conference, useR! 2013 July 1012, 2013 University of Castilla-La Mancha, Albacete, Spain (Vol. 10, No. 30, p. 34).Google Scholar
  8. Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245–281.CrossRefGoogle Scholar
  9. Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE Review42(4), 40–57.Google Scholar
  10. Charleer, S., Klerkx, J., & Duval, E. (2014). Learning dashboards. Journal of Learning Analytics, 1(3), 199–202.CrossRefGoogle Scholar
  11. Devolder, A., van Braak, J., & Tondeur, J. (2012). Supporting self-regulated learning in computer-based learning environments: Systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning.  https://doi.org/10.1111/j.1365-2729.2011.00476.x.Google Scholar
  12. Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012). The student activity meter for awareness and self-reflection. In Extended abstracts on human factors in computing systems (pp. 869–884). ACM.Google Scholar
  13. Griffin, T. D., Wiley, J., & Salas, C. R. (2013). Supporting effective self-regulated learning: The critical role of monitoring. In International handbook of metacognition and learning technologies (pp. 19–34). New York: Springer.Google Scholar
  14. Guo, S., Barth, R. P., & Gibbons, C. (2006). Propensity score matching strategies for evaluating substance abuse services for child welfare clients. Children and Youth Services Review.  https://doi.org/10.1016/j.childyouth.2005.04.012.Google Scholar
  15. Harvey, C., Eshleman, K., Koo, K., Smith, K. G., Paradise, C. J., & Campbell, A. M. (2016). Encouragement for faculty to implement vision and change. CBE-Life Sciences Education.  https://doi.org/10.1187/cbe.16-03-0127.Google Scholar
  16. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research.  https://doi.org/10.3102/003465430298487.Google Scholar
  17. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling.  https://doi.org/10.1080/10705519909540118.Google Scholar
  18. Huberth, M., Chen, P., Tritz, J., & McKay, T. (2015). Computer-tailored student support in introductory physics. PLoS ONE.  https://doi.org/10.1371/journal.pone.0137001.Google Scholar
  19. Ifenthaler, D., & Schumacher, C. (2016). Student perceptions of privacy principles for learning analytics. Educational Technology Research and Development.  https://doi.org/10.1007/s11423-016-9477-y.Google Scholar
  20. Ifenthaler, D., & Widanaprathirana, C. (2014). Development and validation of a learning analytics framework: Two case studies using support vector machines. Technology, Knowledge and Learning.  https://doi.org/10.1007/s10758-014-9226-4.Google Scholar
  21. Iglesias-Pradas, S., Ruiz-de-Azcárate, C., & Agudo-Peregrina, Á. F. (2015). Assessing the suitability of student interactions from Moodle data logs as predictors of cross-curricular competencies. Computers in Human Behavior.  https://doi.org/10.1016/j.chb.2014.09.065.Google Scholar
  22. Imai, K., & van Dyk, D. A. (2004). Causal inference with general treatment regimes. Journal of the American Statistical Association.  https://doi.org/10.1198/016214504000001187.Google Scholar
  23. Jairam, D., & Kiewra, K. A. (2010). Helping students soar to success on computers: An investigation of the SOAR study method for computer-based learning. Journal of Educational Psychology, 102(3), 601.  https://doi.org/10.1037/a0019137.CrossRefGoogle Scholar
  24. Janssen, J., Erkens, G., Kanselaar, G., & Jaspers, J. (2007). Visualization of participation: Does it contribute to successful computer-supported collaborative learning? Computers and Education.  https://doi.org/10.1016/j.compedu.2006.01.004.Google Scholar
  25. Junco, R., & Clem, C. (2015). Predicting course outcomes with digital textbook usage data. The Internet and Higher Education, 27, 54–63.CrossRefGoogle Scholar
  26. Larusson, J. A., & White, B. (2014). Learning analytics. Berlin: Springer.CrossRefGoogle Scholar
  27. Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist.  https://doi.org/10.1177/0002764213479367.Google Scholar
  28. Lonn, S., & Teasley, S. D. (2009). Saving time or innovating practice: Investigating perceptions and uses of Learning Management Systems. Computers & Education.  https://doi.org/10.1016/j.compedu.2009.04.008 Google Scholar
  29. Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers and Education.  https://doi.org/10.1016/j.compedu.2009.09.008.Google Scholar
  30. Morgan, P. L., Frisco, M. L., Farkas, G., & Hibel, J. (2008). A propensity score matching analysis of the effects of special education services. Journal of Special Education.  https://doi.org/10.1177/0022466908323007.Google Scholar
  31. Pardos, Z. A., Whyte, A., & Kao, K. (2016). moocRP: Enabling open learning analytics with an open source platform for data distribution, analysis, and visualization. Technology, Knowledge, and Learning.  https://doi.org/10.1007/s10758-015-9268-2.Google Scholar
  32. Pintrich, P. R. (1999). The role of motivation in promoting and sustaining self-regulated learning. Journal of Educational Research.  https://doi.org/10.1016/S0883-0355(99)00015-4.Google Scholar
  33. Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review.  https://doi.org/10.1007/s10648-004-0006-x.Google Scholar
  34. Pintrich, P. R., Smith, D. A., García, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813.CrossRefGoogle Scholar
  35. Pintrich, P. R., & Zusho, A. (2002). The development of academic self-regulation: The role of cognitive and motivational factors. In A. Wigfield & J. S. Eccles (Eds.), Development of achievement motivation: A volume in the educational psychology series (pp. 249–284). San Diego, CA: Academic Press.CrossRefGoogle Scholar
  36. Roll, I., & Winne, P. (2015). Understanding, evaluating, and supporting self-regulated learning using learning analytics. Journal of Learning Analytics, 2(1), 7–12. Retrieved November 8, 2016 from http://learning-analytics.info/journals/index.php/JLA/article/view/4491/4825.
  37. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.CrossRefGoogle Scholar
  38. Scholes, V. (2016). The ethics of using learning analytics to categorize students on risk. Educational Technology Research and Development.  https://doi.org/10.1007/s11423-016-9458-1.Google Scholar
  39. Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using Signals for appropriate feedback: Perceptions and practices. Computers and Education.  https://doi.org/10.1016/j.compedu.2011.05.016.Google Scholar
  40. Tempelaar, D., Rienties, B., & Giesbers, B. (2015). In search of the most informative data for feedback generation: Learning analytics in a data-rich context. Computers in Human Behavior.  https://doi.org/10.1016/j.chb.2014.05.038.Google Scholar
  41. Trevors, G., Feyzi-Behnagh, R., Azevedo, R., & Bouchet, F. (2016). Self-regulated learning processes vary as a function of epistemic beliefs and contexts: Mixed method evidence from eye tracking and concurrent and retrospective reports. Learning and Instruction, 42, 31–46.  https://doi.org/10.1016/j.learninstruc.2015.11.003.CrossRefGoogle Scholar
  42. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications. Behavioral Scientist.  https://doi.org/10.1177/0002764213479363.Google Scholar
  43. Winne, P. H. (2010). Improving measurements of self-regulated learning. Educational Psychologist.  https://doi.org/10.1080/00461520.2010.517150.Google Scholar
  44. Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. Metacognition in Educational Theory and Practice, 93, 27–30.Google Scholar
  45. Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidne (Eds.), Handbook of self-regulation (pp. 531–566). Orlando, FL: Academic Press.CrossRefGoogle Scholar
  46. Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. In Proceedings of the fourth international conference on learning analytics and knowledge (pp. 203–211). ACM.Google Scholar
  47. Wise, A. F. (2016). Data-informed learning environments. EDUCAUSE Review, Retrieved on November 10, 2016 from http://er.educause.edu/articles/2016/10/data-informed-learning-environments.
  48. Zimmerman, B. J. (1986). Becoming a self-regulated learner: Which are the key subprocesses? Contemporary Educational Psychology.  https://doi.org/10.1016/0361-476X(86)90027-5.Google Scholar
  49. Zimmerman, B. J. (2000). Self-efficacy: An essential motive to learn. Contemporary Educational Psychology.  https://doi.org/10.1006/ceps.1999.1016.Google Scholar

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2017

Authors and Affiliations

  1. 1.The Office of Teaching, Learning and Technology & The Office of AssessmentThe University of IowaIowa CityUSA
  2. 2.Department of Communication Sciences and DisordersThe University of IowaIowa CityUSA
  3. 3.The Office of Teaching, Learning and TechnologyThe University of IowaIowa CityUSA
  4. 4.Department of BiostatisticsThe University of IowaIowa CityUSA
  5. 5.Department of ChemistryThe University of IowaIowa CityUSA
  6. 6.ITS Administrative Information SystemsThe University of IowaIowa CityUSA
  7. 7.Office of Institutional Research and EffectivenessUniversity of DelawareNewarkUSA
  8. 8.Division of Pediatric Critical CareUniversity of Utah School of MedicineSalt Lake CityUSA
  9. 9.Division of Biomedical Statistics and InformaticsMayo ClinicRochesterUSA

Personalised recommendations