Abstract
This study examines how student and instructor participation in online discussions impacts students’ course performance. The context for the study is university introductory online mathematics/statistics courses, which typically have much higher failure rates than their face-to-face counterparts. Using text-mining techniques, we analyze online discussion data automatically collected by a Learning Management System across five years from 2869 students in 72 online courses, who collectively contributed 20,884 posts. These semi-automated techniques enable a broader and more scalable view of participation behaviors by investigating: (1) student posting and non-posting behaviors (called online speaking and listening, respectively), (2) the textual content of posts, and (3) instructors’ strategies for structuring discussions. Multilevel modeling results show that online listening behaviors significantly predict students’ course performance. Further, students’ posts that built on other contributions or applied new knowledge have the highest predictive value in terms of course performance. Finally, the instructors’ use of open-ended prompts is the only variable positively and significantly links to students’ course performance. Links to theory, instructional practice, and educational data mining are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bainbridge, J., Melitski, J., Zahradnik, A., Lauría, E., Jayaprakash, S. M., & Baron, J. (2015). Using learning analytics to predict at-risk students in online graduate public affairs and administration education. Journal of Public Affairs Education, 21(2), 247–262. https://doi.org/10.1080/15236803.2015.12001831
Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79(3), 1243–1289. https://doi.org/10.3102/0034654309333844
Bliss, C. A., & Lawrence, B. (2009). From posts to patterns: A metric to characterize discussion board activity in online courses. Journal of Asynchronous Learning Networks, 13(2), 15–32. https://doi.org/10.24059/olj.v13i2.1665
Bradley, M. E., Thom, L. R., Hayes, J., & Hay, C. (2008). Ask and you will receive: How question type influences quantity and quality of online discussions. British Journal of Educational Technology, 39(5), 888–900. https://doi.org/10.1111/j.1467-8535.2007.00804.x
Chen, G., Chiu, M. M., & Wang, Z. (2012). Social metacognition and the creation of correct, new ideas: A statistical discourse analysis of online mathematics discussions. Computers in Human Behavior, 28(3), 868–880. https://doi.org/10.1016/j.chb.2011.12.006
Darabi, A., Liang, X., Suryavanshi, R., & Yurekli, H. (2013). Effectiveness of online discussion strategies: A meta-analysis. American Journal of Distance Education, 27(4), 228–241. https://doi.org/10.1080/08923647.2013.837651
Dennen, V. P. (2008). Pedagogical lurking: Student engagement in non-posting discussion behavior. Computers in Human Behavior, 24(4), 1624–1633. https://doi.org/10.1016/j.chb.2007.06.003
De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers and Education, 46(1), 6–28. https://doi.org/10.1016/j.compedu.2005.04.005
Ertmer, P. A., Sadaf, A., & Ertmer, D. J. (2011). Student-content interactions in online courses: The role of question prompts in facilitating higher-level engagement with course content. Journal of Computing in Higher Education, 23(2–3), 157–186. https://doi.org/10.1007/s12528-011-9047-6
Farrow, E., Moore, J., & Gašević, D. (2019). Analysing discussion forum data: A replication study avoiding data contamination. In D. Azcona, & R. Chung (Eds.), Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 170–179). New York, NY https://doi.org/10.1145/3303772.3303779
Gao, F., Zhang, T., & Franklin, T. (2013). Designing asynchronous online discussion environments: Recent progress and possible future directions. British Journal of Educational Technology, 44(3), 469–483. https://doi.org/10.1111/j.1467-8535.2012.01330.x
Gilbert, P. K., & Dabbagh, N. (2005). How to structure online discussions for meaningful discourse: A case study. British Journal of Educational Technology, 36(1), 5–18. https://doi.org/10.1111/j.1467-8535.2005.00434.x
Groth, R. (2008). Analyzing online discourse to assess students’ thinking. The Mathematics Teacher, 101(6), 422–427. https://doi.org/10.5951/mt.101.6.0422
Groth, R., & Burgess, C. (2009). An exploration of two online approaches to mathematics teacher education. Journal of Computers in Mathematics and Science Teaching, 28(3), 221–246.
Hew, K. F., Cheung, W. S., & Ng, C. S. L. (2010). Student contribution in asynchronous online discussion: A review of the research and empirical exploration. Instructional Science, 38(6), 571–606. https://doi.org/10.1007/s11251-008-9087-0
Hung, J.-L., Rice, K., & Saba, A. (2012). An educational data mining model for online teaching and learning. Journal of Educational Technology Development and Exchange, 5(2), 77–94. https://doi.org/10.18785/jetde.0502.06
Jaggars, S. S., Edgecombe, N., & Stacey, G. W. (2013). What we know about online course outcomes: Research overview. Community College Research Center, Columbia University. Retrieved from http://eric.ed.gov/?id=ED542143
Ke, F., & Xie, K. (2009). Toward deep learning for adult students in online courses. Internet and Higher Education, 12(3–4), 136–145. https://doi.org/10.1016/j.iheduc.2009.08.001
Kovanovic, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards automated content analysis of discussion transcripts : A cognitive presence case. In S. Dawson, H. Drachsler, & H. Drachsler (Eds.), Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 15–24). New York, NY: ACM. https://doi.org/10.1145/2883851.2883950
Kumar, S., & Sree, R. R. (2014). Experiments towards determining best training sample size for automated evaluation of descriptive answers through sequential minimal optimization. Journal on Soft Computing, 4(2), 710–714. https://doi.org/10.21917/ijsc.2014.0102
Lee, J. E., & Recker, M. (2021). The effects of instructors’ use of online discussions strategies on student participation and performance in university online introductory mathematics courses. Computers & Education, 162, 104084. https://doi.org/10.1016/j.compedu.2020.104084
Lucas, M., Gunawardena, C., & Moreira, A. (2014). Assessing social construction of knowledge online: A critique of the interaction analysis model. Computers in Human Behavior, 30, 574–582. https://doi.org/10.1016/j.chb.2013.07.050
Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics failed to inform an institutional strategic plan. Journal of Educational Technology & Society, 15(3), 149–163.
Maurino, P. S. M. (2007). Looking for critical thinking in online threaded discussions. Journal of Educational Technology Systems, 35(3), 241–260. https://doi.org/10.2190/p4w3-8117-k32g-r34m
Mayfield, E., Adamson, D., & Rosé, C. (2013). LightSIDE: Researcher’s user manual. Retrieved from http://www.cs.cmu.edu/~cprose/LightSIDE_Researchers_Manual_Draft3.pdf
Mu, J., Stegmann, K., Mayfield, E., Rosé, C., & Fischer, F. (2012). The ACODEA framework: Developing segmentation and classification schemes for fully automatic analysis of online discussions. International Journal of Computer-Supported Collaborative Learning, 7(2), 285–305. https://doi.org/10.1007/s11412-012-9147-y
Pettijohn, I. I., Terry, F., & Pettijohn, T. F. (2007). Required discussion web pages in psychology courses and student outcomes. Journal of Instructional Psychology, 34(4), 256–263.
Poscente, K. R., & Fahy, P. J. (2003). Investigating triggers in CMC text transcripts. The International Review of Research in Open and Distance Learning, 4(2), 9–11. https://doi.org/10.19173/irrodl.v4i2.141
Rosé, C., Wang, Y. C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning, 3(3), 237–271. https://doi.org/10.1007/s11412-007-9034-0
Richardson, J. C., & Ice, P. (2010). Investigating students’ level of critical thinking across instructional strategies in online discussions. The Internet and Higher Education, 13(1–2), 52–59. https://doi.org/10.1016/j.iheduc.2009.10.009
Salter, N. P., & Conneely, M. R. (2015). Structured and unstructured discussion forums as tools for student engagement. Computers in Human Behavior, 46, 18–25. https://doi.org/10.1016/j.chb.2014.12.037
Saxe, K., & Braddy, L. (2015). A common vision for undergraduate mathematical sciences program in 2025. Retrieved from https://www.maa.org/sites/default/files/pdf/CommonVisionFinal.pdf
Stein, M. K., Engle, R. A., Smith, M. S., & Hughes, E. K. (2008). Orchestrating productive mathematical discussions: Five practices for helping teachers move beyond show and tell. Mathematical Thinking and Learning, 10(4), 313–340. https://doi.org/10.1080/10986060802229675
Thomas, D., Li, Q., Knott, L., & Li, Z. (2008). The structure of student dialogue in web-assisted mathematics courses. Journal of Educational Technology Systems, 36(4), 415–431. https://doi.org/10.2190/et.36.4.e
Tunstall, S. L., & Bossé, M. J. (2015). Promoting numeracy in an online college algebra course through projects and discussions. Numeracy, 8(2), 1–23. https://doi.org/10.5038/1936-4660.8.2.10
Uzun, K. (2018). Home-grown automated essay scoring in the literature classroom A solution for managing the crowd? Contemporary Educational Technology, 9(4), 423–436. https://doi.org/10.30935/cet.471024
Van Der Kleij, F. M., Feskens, R. C. W., & Eggen, T. J. H. M. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes: A meta-analysis. Review of Educational Research, 85(4), 475–511. https://doi.org/10.3102/0034654314564881
Vogel, F., Kollar, I., Ufer, S., Reichersdorfer, E., Reiss, K., & Fischer, F. (2016). Developing argumentation skills in mathematics through computer-supported collaborative learning: The role of transactivity. Instructional Science, 44(5), 477–500. https://doi.org/10.1007/s11251-016-9380-2
Wang, Q. (2008). Student-facilitators’ roles in moderating online discussions. British Journal of Educational Technology, 39(5), 859–874. https://doi.org/10.1111/j.1467-8535.2007.00781.x
Wang, X., Yang, D., Wen, M., Koedinger, K., & Rosé, C. P. (2015). Investigating how student’s cognitive behavior in MOOC discussion forums affect learning gains In O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, C. Mihaescu (Eds), Proceedings of the 8th International Conference on Educational Data Mining. Madrid, Spain
Warnock, S., Bingham, K., Driscoll, D., Fromal, J., & Rouse, N. (2012). Early participation in asynchronous writing environments and course success. Journal of Asynchronous Learning Network, 16(1), 35–48. https://doi.org/10.24059/olj.v16i1.178
Waters, J. (2012). Thought-leaders in asynchronous online learning environments. Journal of Asynchronous Learning Networks, 16(1), 19–34. https://doi.org/10.24059/olj.v16i1.219
Wen, M., Yang, D., & Rosé, C. (2014). Sentiment analysis in MOOC discussion forums: What does it tell us? In J. Stamper, Z. Pardos, M. Mavrikis, & B. M. McLaren (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (pp. 1–8). New York, NY: ACM.
Wise, A. F., Marbouti, F., Hsiao, Y.-T., & Hausknecht, S. (2012). A survey of factors contributing to learners’ “listening” behaviors in asynchronous online discussions. Journal of Educational Computing Research, 47(4), 461–480. https://doi.org/10.2190/ec.47.4.f
Wise, A. F., Speer, J., Marbouti, F., & Hsiao, Y. T. (2013). Broadening the notion of participation in online discussions: Examining patterns in learners’ online listening behaviors. Instructional Science, 41(2), 323–343. https://doi.org/10.1007/s11251-012-9230-9
Wise, A., Zhao, Y., & Hausknecht, S. (2014). Learning analytics for online discussions: Embedded and extracted approaches. Journal of Learning Analytics, 1(2), 48–71. https://doi.org/10.18608/jla.2014.12.4
Xie, K. (2013). What do the numbers say? The influence of motivation and peer feedback on students’ behaviour in online discussions. British Journal of Educational Technology, 44(2), 288–301. https://doi.org/10.1111/j.1467-8535.2012.01291.x
Xie, K., & Ke, F. (2011). The role of students’ motivation in peer-moderated asynchronous online discussions. British Journal of Educational Technology, 42(6), 916–930. https://doi.org/10.1111/j.1467-8535.2010.01140.x
Yang, D., Richardson, J. C., French, B. F., & Lehman, J. D. (2011). The development of a content analysis model for assessing students’ cognitive learning in asynchronous online discussions. Educational Technology Research and Development, 59(1), 43–70. https://doi.org/10.1007/s11423-010-9166-1
Acknowledgements
We thank Canvalytics research team and the AS staff members for their invaluable assistant.
Funding
The authors received no specific funding for this work.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in the study was approved by Institutional Review Board (IRB). All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent
For this type of study formal consent is not required.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
See Table
6.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lee, JE., Recker, M. Predicting student performance by modeling participation in asynchronous discussions in university online introductory mathematical courses. Education Tech Research Dev 70, 1993–2015 (2022). https://doi.org/10.1007/s11423-022-10153-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-022-10153-5