Skip to main content
SpringerLink
Log in

Automated feedback on discourse moves: teachers’ perceived utility of a professional learning tool

  • Research Article
  • Published:
Educational technology research and development Aims and scope Submit manuscript

Abstract

Technological tools that provide automated feedback on classroom teaching afford a unique opportunity for educators to engage in self-reflection and work towards improvement goals, in particular to ensure that their instructional environment is equitable and productive for students. More information is needed about how teachers experience automated professional learning tools, including what they perceive as relevant and impactful for their everyday teaching. This mixed-methods study explored the perceptions and engagement of 21 math teachers who used an AI-based tool that generates information about their discourse practices from classroom recordings. Findings indicate that teachers perceived the tool to have a high utility value, especially those who elected to use it over two school years. These teachers increased their use of talk moves over time, suggesting that they were making intentional changes due to their review and uptake of the personalized feedback. These results from this study speak to promising directions for developing AI-based professional learning tools that can support teacher learning and instructional improvement, particularly tools with robust perceived utility.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Alexander, R. (2020). A dialogic teaching companion. Routledge.

    Book  Google Scholar 

  • Amiel, T., & Reeves, T. C. (2008). Design-based research and educational technology: Rethinking technology and the research agenda. Journal of Educational Technology & Society, 11(4), 29–40. https://doi.org/10.2307/jeductechsoci.11.4.29

    Article  Google Scholar 

  • Appova, A., & Arbaugh, F. (2018). Teachers’ motivation to learn: Implications for supporting professional growth. Professional Development in Education, 44(1), 5–21. https://doi.org/10.1080/19415257.2017.1280524

    Article  Google Scholar 

  • Backfisch, I., Lachner, A., Stürmer, K., & Scheiter, K. (2021). Variability of teachers’ technology integration in the classroom: A matter of utility! Computers & Education, 166, 104159. https://doi.org/10.1016/j.compedu.2021.104159

    Article  Google Scholar 

  • Beauchamp, G., Burden, K., & Abbinett, E. (2015). Teachers learning to use the iPad in Scotland and Wales: A new model of professional development. Journal of Education for Teaching, 41(2), 161–179. https://doi.org/10.1080/02607476.2015.1013370

    Article  Google Scholar 

  • Bokhove, C., & Downey, C. (2018). Automated generation of ‘good enough’ transcripts as a first step to transcription of audio-recorded data. Methodological Innovations, 11(2), 205979911879074. https://doi.org/10.1177/2059799118790743

    Article  Google Scholar 

  • Candela, A. G., Boston, M. D., & Dixon, J. K. (2020). Discourse actions to promote student access. Mathematics Teacher: Learning and Teaching PK-12, 113(4), 266–277.

    Article  Google Scholar 

  • Chapin, S. H., O'Connor, M. C., & Anderson, N. C. (2009). Classroom discussions: Using math talk to help students learn, Grades K-6. Math Solutions.

  • Chapin, S., & O’Connor, C. (2012). Project Challenge: Using challenging curriculum and mathematical discourse to help all students learn. In C. Dudley-Marling & S. Michaels (Eds.), High-expectation curricula: Helping all students succeed with powerful learning (pp. 113–127). Teachers College.

    Google Scholar 

  • Chen, G. (2020). A visual learning analytics (VLA) approach to video-based teacher professional development: Impact on teachers’ beliefs, self-efficacy, and classroom talk practice. Computers & Education, 144, 1–15. https://doi.org/10.1016/j.compedu.2019.103670

    Article  Google Scholar 

  • Chen, G., Clarke, S. N., & Resnick, L. B. (2015). Classroom Discourse Analyzer (CDA): A discourse analytic tool for teachers. Technology, Instruction, Cognition & Learning, 10(2), 85–105. https://doi.org/10.1080/10508406.2020.1783269

    Article  Google Scholar 

  • Chen, G., Chan, C. K., Chan, K. K., Clarke, S. N., & Resnick, L. B. (2020). Efficacy of video-based teacher professional development for increasing classroom discourse and student learning. Journal of the Learning Sciences, 29(4–5), 642–680. https://doi.org/10.1080/10508406.2020.1783269

    Article  Google Scholar 

  • Clarke, D., & Hollingsworth, H. (2002). Elaborating a model of teacher professional growth. Teaching and Teacher Education, 18(8), 947–967.

    Article  Google Scholar 

  • Correnti, R., Stein, M. K., Smith, M. S., Scherrer, J., McKeown, M. G., Greeno, J. G., & Ashley, K. (2015). Improving teaching at scale: Design for the scientific measurement and learning of discourse practice. In L. B. Resnick, C. S. C. Asterhan, & S. N. Clarke (Eds.), Socializing intelligence through academic talk and dialogue (pp. 315–334). American Educational Research Association.

    Chapter  Google Scholar 

  • Correnti, R., Matsumura, L. C., Walsh, M., Zook-Howell, D., Bickel, D. D., & Yu, B. (2021). Effects of online content-focused coaching on discussion quality and reading achievement: Building theory for how coaching develops teachers’ adaptive expertise. Reading Research Quarterly, 56(3), 519–558. https://doi.org/10.1002/rrq.317

    Article  Google Scholar 

  • Creswell, J. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). Sage.

    Google Scholar 

  • Davis, F.D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 319–340.

  • Demszky, D., Liu, J., Hill, H.C., Jurafsky, D., & Piech, C. (2021). Can automated feedback improve teachers’ uptake of student ideas? Evidence from a randomized controlled trial In a large-scale online course. (EdWorkingPaper: 21–483). Retrieved from Annenberg Institute at Brown University. https://doi.org/10.26300/thn9-wh86

  • Donnelly, P.J., Blanchard, N., Samei, B., Olney, A.M., Sun, X., Ward, B., ... & D’Mello, S.K. (2016). Automatic teacher modeling from live classroom audio. Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization (pp. 45–53).

  • Donnelly, P.J., Blanchard, N., Olney, A.M., Kelly, S., Nystrand, M., & D'Mello, S.K. (2017). Words matter: automatic detection of teacher questions in live classroom discourse using linguistics, acoustics, and context. Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 218–227).

  • Ertmer, P. A., Ottenbreit-Leftwich, A. T., Sadik, O., Sendurur, E., & Sendurur, P. (2012). Teacher beliefs and technology integration practices: A critical relationship. Computers & Education, 59(2), 423–435. https://doi.org/10.1016/j.compedu.2012.02.001

    Article  Google Scholar 

  • Fishman, B., Penuel, W. R., Hegedus, S., & Roschelle, J. (2011). What happens when the research ends? Factors related to the sustainability of a technology-infused mathematics curriculum. Journal of Computers in Mathematics and Science Teaching, 30(4), 329–353.

    Google Scholar 

  • Ford, K., & Welling-Riley, K. (2021). Student talk in science class. The Learning Professional, 42(3), 58–61.

    Google Scholar 

  • Frank, K. A., Zhao, Y., Penuel, W. R., Ellefson, N., & Porter, S. (2011). Focus, fiddle, and friends: Experiences that transform knowledge for the implementation of innovations. Sociology of Education, 84(2), 137–156. https://doi.org/10.1177/0038040711401812

    Article  Google Scholar 

  • Franklin, R. K., Mitchell, J. O., Walters, K. S., Livingston, B., Lineberger, M. B., Putman, C., Yarborough, R., & Karges-Bone, L. (2018). Using Swivl robotic technology in teacher education preparation: A pilot study. TechTrends, 62(2), 184–189. https://doi.org/10.1007/s11528-017-0246-5

    Article  Google Scholar 

  • Fütterer, T., Scherer, R., Scheiter, K., Stürmer, K., & Lachner, A. (2023). Will, skills, or conscientiousness: What predicts teachers’ intentions to participate in technology-related professional development? Computers & Education. https://doi.org/10.1016/j.compedu.2023.104756

    Article  Google Scholar 

  • Gaudin, C., & Chaliès, S. (2015). Video viewing in teacher education and professional development: A literature review. Educational Research Review, 16, 41–67. https://doi.org/10.1016/j.edurev.2015.06.001

    Article  Google Scholar 

  • Given, L. M. (Ed.). (2008). The Sage encyclopedia of qualitative research methods. Sage Publications.

    Google Scholar 

  • Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487

    Article  Google Scholar 

  • Herbel-Eisenmann, B. A., Steele, M. D., & Cirillo, M. (2013). (Developing) teacher discourse moves: A framework for professional development. Mathematics Teacher Educator, 1(2), 181–196.

    Article  Google Scholar 

  • Heyd-Metzuyanim, E., Smith, M., Bill, V., & Resnick, L. B. (2019). From ritual to explorative participation in discourse-rich instructional practices: A case study of teacher learning through professional development. Educational Studies in Mathematics, 101(2), 273–289. https://doi.org/10.1007/s10649-018-9849-9

    Article  Google Scholar 

  • Hulleman, C. S., Godes, O., Hendricks, B. L., & Harackiewicz, J. M. (2010). Enhancing interest and performance with a utility value intervention. Journal of Educational Psychology, 102(4), 880. https://doi.org/10.1037/a0019506

    Article  Google Scholar 

  • Hulleman, C. S., Kosovich, J. J., Barron, K. E., & Daniel, D. B. (2017). Making connections: Replicating and extending the utility value intervention in the classroom. Journal of Educational Psychology, 109(3), 387. https://doi.org/10.1037/edu0000146

    Article  Google Scholar 

  • Jacobs, J., Boardman, A., Potvin, A., & Wang, C. (2018). Understanding teacher resistance to instructional coaching. Professional Development in Education, 44(5), 690–703. https://doi.org/10.1080/19415257.2017.1388270

    Article  Google Scholar 

  • Jacobs, J., Scornavacco, K., Harty, C., Suresh, A., Lai, V., & Sumner, T. (2022). Promoting rich discussions in mathematics classrooms: Using personalized, automated feedback to support reflection and instructional change. Teaching and Teacher Education. https://doi.org/10.1016/j.tate.2022.103631

    Article  Google Scholar 

  • Jensen, B., Valdés, G., & Gallimore, R. (2021). Teachers learning to implement equitable classroom talk. Educational Researcher, 50(8), 546–556. https://doi.org/10.3102/0013189X211014859

    Article  Google Scholar 

  • Jensen, E., Dale, M., Donnelly, P.J., Stone, C., Kelly, S., Godley, A., & D'Mello, S.K. (2020). Toward automated feedback on teacher discourse to enhance teacher learning. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–13).

  • Kale, U. (2018). Technology valued? Observation and review activities to enhance future teachers’ utility value toward technology integration. Computers & Education, 117, 160–174.

    Article  Google Scholar 

  • Kale, U., & Akcaoglu, M. (2018). The role of relevance in future teachers’ utility value and interest toward technology. Educational Technology Research and Development, 66(2), 283–311. https://doi.org/10.1007/s11423-017-9547-9

    Article  Google Scholar 

  • Kelly, S., Olney, A. M., Donnelly, P., Nystrand, M., & D’Mello, S. K. (2018). Automatically measuring question authenticity in real-world classrooms. Educational Researcher, 47(7), 451–464. https://doi.org/10.3102/0013189X18785613

    Article  Google Scholar 

  • Kennedy, M. M. (2016). How does professional development improve teaching? Review of Educational Research, 86(4), 945–980. https://doi.org/10.3102/0034654315626800

    Article  Google Scholar 

  • Kershner, R., Hennessy, S., Dowdall, K., Owen, H., & Calcagni, M. E. (2020). Teachers as ‘natural experimenters’: Using T-SEDA to develop classroom dialogue. In L. Rolls & E. Hargreaves (Eds.), Reimagining professional development in schools (Chapter 8). Routledge.

    Google Scholar 

  • Kosovich, J. J., Hulleman, C. S., Phelps, J., & Lee, M. (2019). Improving algebra success with a utility-value intervention. Journal of Developmental Education, 42(2), 2–10.

    Google Scholar 

  • Loucks-Horsley, S., Stiles, K. E., Mundry, S., Love, N., & Hewson, P. W. (2009). Designing professional development for teachers of science and mathematics. Corwin Press.

    Google Scholar 

  • Major, L., & Watson, S. (2018). Using video to support in-service teacher professional development: The state of the field, limitations and possibilities. Technology, Pedagogy and Education, 27(1), 49–68. https://doi.org/10.1080/1475939X.2017.1361469

    Article  Google Scholar 

  • Mathison, S. (1988). Why triangulate? Educational Researcher, 17(2), 13–17.

    Article  Google Scholar 

  • Michaels, S., O’Connor, M. C., Hall, M. W., & Resnick, L. B. (2010). Accountable talk sourcebook: For classroom conversation that works. University of Pittsburgh Institute for Learning.

    Google Scholar 

  • Mumtaz, S. (2000). Factors affecting teachers’ use of information and communications technology: A review of the literature. Journal of Information Technology for Teacher Education, 9(3), 319–342. https://doi.org/10.1080/14759390000200096

    Article  Google Scholar 

  • O’Connor, C., & Michaels, S. (2019). Supporting teachers in taking up productive talk moves: The long road to professional learning at scale. International Journal of Educational Research, 97, 166–175. https://doi.org/10.1016/j.ijer.2017.11.003

    Article  Google Scholar 

  • O’Connor, C., Michaels, S., & Chapin, S. (2015). Scaling down to explore the role of talk in learning: From district intervention to controlled classroom study. In L. B. Resnick, C. S. C. Asterhan, & S. N. Clarke (Eds.), Socializing intelligence through academic talk and dialogue (pp. 111–126). AERA.

    Chapter  Google Scholar 

  • Okumuş, S., Lewis, L., Wiebe, E., & Hollebrands, K. (2016). Utility and usability as factors influencing teacher decisions about software integration. Educational Technology Research and Development, 64(6), 1227–1249. https://doi.org/10.1007/s11423-016-9455-4

    Article  Google Scholar 

  • Prince, T. (2016). “Panopto” for lecture capture: A first time user’s perspective. Society for Information Technology & Teacher Education International Conference (pp. 2550–2565). Association for the Advancement of Computing in Education (AACE).

  • Resnick, L. B., Asterhan, C. S. C., Clarke, S. N., & Schantz, F. (2018). Next generation research in dialogic learning. In G. E. Hall, L. F. Quinn, & D. M. Gollnick (Eds.), Wiley handbook of teaching and learning (pp. 323–338). Wiley-Blackwell.

    Chapter  Google Scholar 

  • Rogers, E. (1995). Diffusion of innovations. The Free Press.

    Google Scholar 

  • Rosenzweig, E. Q., Wigfield, A., & Hulleman, C. S. (2020). More useful or not so bad? Examining the effects of utility value and cost reduction interventions in college physics. Journal of Educational Psychology, 112(1), 166. https://doi.org/10.1037/edu0000370

    Article  Google Scholar 

  • Scherer, R., Siddiq, F., & Tondeur, J. (2019). The technology acceptance model (TAM): A meta-analytic structural equation modeling approach to explaining teachers’ adoption of digital technology in education. Computers & Education, 128, 13–35. https://doi.org/10.1016/j.compedu.2018.09.009

    Article  Google Scholar 

  • Schildkamp, K. (2019). Data-based decision-making for school improvement: Research insights and gaps. Educational Research, 61(3), 257–273. https://doi.org/10.1080/00131881.2019.1625716

    Article  Google Scholar 

  • Scott, P., & Mouza, C. (2007). The impact of professional development on teacher learning, practice and leadership skills: A study on the integration of technology in the teaching of writing. Journal of Educational Computing Research, 37(3), 229–266.

    Article  Google Scholar 

  • Sedova, K. (2017). A case study of a transition to dialogic teaching as a process of gradual change. Teaching and Teacher Education, 67, 278–290. https://doi.org/10.1016/j.tate.2017.06.018

    Article  Google Scholar 

  • Song, Y., Lei, S., Hao, T., Lan, Z., & Ding, Y. (2021). Automatic classification of semantic content of classroom dialogue. Journal of Educational Computing Research, 59(3), 496–521. https://doi.org/10.1177/0735633120968554

    Article  Google Scholar 

  • Suresh, A., Penuel, W., Jacobs, J., Martin, M. & Sumner, T. (in press). Using AI tools to provide teachers with real-time feedback on their classroom discourse patterns. In X. Zhai & J. Krajcik (eds), Uses of Artificial Intelligence in STEM Education. Oxford University Press

  • Suresh, A., Jacobs, J., Clevenger, C., Lai, V., Tan, C. Martin, J.H. & Sumner, T. (2021). Using AI to promote equitable classroom discussions: The TalkMoves application. Proceedings of the International Conference on Artificial Intelligence in Education (pp. 344–348). Springer. https://doi.org/10.1007/978-3-030-78270-2_61

  • Suresh, A., Jacobs, J., Clevenger, C. Perkoff, M., Martin, J. & Sumner, T. (2022a). The TalkMoves dataset: K-12 mathematics lesson transcripts annotated for teacher and student discursive moves. Proceedings of the 13th Conference Language Resources and Evaluation (pp. 4654–4662). Marseille, France: European Language Resources Association. https://arxiv.org/abs/2204.09652

  • Suresh, A., Jacobs, J., Perkoff, M., Martin, J. & Sumner, T. (2022b). Fine-tuning transformers with additional context to classify discursive moves in mathematics classrooms. Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (pp. 71–81). Seattle: WA: Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.bea-1.11

  • Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5–23.

    Article  Google Scholar 

  • Webb, N. M., Franke, M. L., Ing, M., Turrou, A. C., Johnson, N. C., & Zimmerman, J. (2019). Teacher practices that promote productive dialogue and learning in mathematics classrooms. International Journal of Educational Research, 97, 176–186. https://doi.org/10.1016/j.ijer.2017.07.009

    Article  Google Scholar 

  • Zuboff, S. (1985). Automate/informate: The two faces of intelligent technology. Organizational Dynamics, 14(2), 5–18.

    Article  Google Scholar 

Download references

Funding

This project was funded by the National Science Foundation, grant 1837986. Any opinions, findings and conclusions or recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. Institute of Cognitive Science, University of Colorado Boulder, 594 UCB, Boulder, CO, 80309, USA

    Jennifer Jacobs, Charis Clevenger, Abhijit Suresh & Tamara Sumner

  2. School of Education, University of Colorado Boulder, 249 UCB, Boulder, CO, 80309, USA

    Karla Scornavacco

Authors
  1. Jennifer Jacobs
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Karla Scornavacco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charis Clevenger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Abhijit Suresh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tamara Sumner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jennifer Jacobs.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jacobs, J., Scornavacco, K., Clevenger, C. et al. Automated feedback on discourse moves: teachers’ perceived utility of a professional learning tool. Education Tech Research Dev (2024). https://doi.org/10.1007/s11423-023-10338-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11423-023-10338-6

Keywords

Search

Navigation