Abstract
Rigorous and interactive class discussions that support students to engage in high-level thinking and reasoning are essential to learning and are a central component of most teaching interventions. However, formally assessing discussion quality ‘at scale’ is expensive and infeasible for most researchers. In this work, we experimented with various modern natural language processing (NLP) techniques to automatically generate rubric scores for individual dimensions of classroom text discussion quality. Specifically, we worked on a dataset of 90 classroom discussion transcripts consisting of over 18000 turns annotated with fine-grained Analyzing Teaching Moves (ATM) codes and focused on four Instructional Quality Assessment (IQA) rubrics. Despite the limited amount of data, our work shows encouraging results in some of the rubrics while suggesting that there is room for improvement in the others. We also found that certain NLP approaches work better for certain rubrics.
Supported by a grant from the Learning Engineering Tools Competition.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We tried different ratios and 60% provides the best results.
References
Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer. arXiv:2004.05150 (2020)
Correnti, R., Matsumura, L.C., Walsh, M., Zook-Howell, D., Bickel, D.D., Yu, B.: Effects of online content-focused coaching on discussion quality and reading achievement: Building theory for how coaching develops teachers’ adaptive expertise. Read. Res. Q. 56(3), 519–558 (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Conference of the North American Chapter of ACL: Human Language Technologies, pp. 4171–4186, Minneapolis (2019)
Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. CoRR arXiv: abs/1508.01991 (2015)
Jacobs, J., Scornavacco, K., Harty, C., Suresh, A., Lai, V., Sumner, T.: Promoting rich discussions in mathematics classrooms: Using personalized, automated feedback to support reflection and instructional change. Teach. Teach. Educ. 112, 103631 (2022)
Matsumura, L.C., Garnier, H.E., Slater, S.C., Boston, M.D.: Toward measuring instructional interactions “at-scale’’. Educ. Assess. 13(4), 267–300 (2008)
Olshefski, C., Lugini, L., Singh, R., Litman, D., Godley, A.: The Discussion Tracker corpus of collaborative argumentation. In: Language Resources & Evaluation (2020)
Suresh, A., et al.: Using AI to promote equitable classroom discussions: The TalkMoves application. In: International Conference on Artificial Intelligence in Education, pp. 344–348 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tran, N., Pierce, B., Litman, D., Correnti, R., Matsumura, L.C. (2023). Utilizing Natural Language Processing for Automated Assessment of Classroom Discussion. In: Wang, N., Rebolledo-Mendez, G., Dimitrova, V., Matsuda, N., Santos, O.C. (eds) Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky. AIED 2023. Communications in Computer and Information Science, vol 1831. Springer, Cham. https://doi.org/10.1007/978-3-031-36336-8_76
Download citation
DOI: https://doi.org/10.1007/978-3-031-36336-8_76
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36335-1
Online ISBN: 978-3-031-36336-8
eBook Packages: Computer ScienceComputer Science (R0)