Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms

  • Kenneth HolsteinEmail author
  • Bruce M. McLaren
  • Vincent Aleven
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10947)


When used in K-12 classrooms, intelligent tutoring systems (ITSs) can be highly effective in helping students learn. However, they might be even more effective if designed to work together with human teachers, to amplify their abilities and leverage their complementary strengths. In the present work, we designed a wearable, real-time teacher awareness tool: mixed-reality smart glasses that tune teachers in to the rich analytics generated by ITSs, alerting them to situations the ITS may be ill-suited to handle. A 3-condition experiment with 286 middle school students, across 18 classrooms and 8 teachers, found that presenting teachers with real-time analytics about student learning, metacognition, and behavior had a positive impact on student learning, compared with both business-as-usual and classroom monitoring support without advanced analytics. Our findings suggest that real-time teacher analytics can help to narrow the gap in learning outcomes across students of varying prior ability. This is the first experimental study showing that real-time teacher analytics can enhance student learning. This research illustrates the promise of AIED systems that integrate human and machine intelligence to support student learning.


Real-time analytics Classroom orchestration Teacher-in-the-loop Human-AI hybrid Intelligent tutors Dashboards Classroom evaluation 



This work was supported by NSF Award #1530726, and IES Grant R305B150008 to CMU. The opinions expressed are those of the authors and do not represent the views of NSF, IES or the U.S. ED. Special thanks to Gena Hong, Octav Popescu, Jonathan Sewall, Mera Tegene, Cindy Tipper, and all participating students and teachers.


  1. 1.
    Alavi, H.S., Dillenbourg, P.: An ambient awareness tool for supporting supervised collaborative problem solving. IEEE TLT 5, 264–274 (2012)Google Scholar
  2. 2.
    Aleven, V.: Help seeking and intelligent tutoring systems: theoretical perspectives and a step towards theoretical integration. In: Azevedo, R., Aleven, V. (eds.) International Handbook of Metacognition and Learning Technologies, pp. 311–335. Springer, New York (2013). Scholar
  3. 3.
    Aleven, V., McLaren, B.M., Sewall, J., Koedinger, K.R.: A new paradigm for intelligent tutoring systems: example-tracing tutors. IJAIED 19(2), 105–154 (2009)Google Scholar
  4. 4.
    Aleven, V., Roll, I., McLaren, B.M., Koedinger, K.R.: Help helps, but only so much: research on help seeking with intelligent tutoring systems. IJAIED 26, 205–223 (2016)Google Scholar
  5. 5.
    Alphen, E.V., Bakker, S.: Lernanto: using an ambient display during differentiated instruction. In: CHI EA (2016)Google Scholar
  6. 6.
    Baker, R.S.: Modeling and understanding students off-task behavior in intelligent tutoring systems. In: CHI, pp. 1059–1068 (2007)Google Scholar
  7. 7.
    Baker, R.S.: Stupid tutoring systems, intelligent humans. IJAIED 26(2), 600–614 (2016)Google Scholar
  8. 8.
    Baker, R.S., Corbett, A.T., Roll, I., Koedinger, K.R.: Developing a generalizable detector of when students game the system. UMUAI 18(3), 287–314 (2008)Google Scholar
  9. 9.
    Beck, J.E., Gong, Y.: Wheel-spinning: students who fail to master a skill. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 431–440. Springer, Heidelberg (2013). Scholar
  10. 10.
    Bull, S., Kay, J.: Open learner models. In: Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.) Advances in Intelligent Tutoring Systems, pp. 301–322. Springer, Heidelberg (2010). Scholar
  11. 11.
    Chromebook Management Software for Schools.
  12. 12.
    Cohen, J.: A power primer. Psychol. Bull. 112(1), 155–159 (1992)CrossRefGoogle Scholar
  13. 13.
    Corbett, A.T., Anderson, J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. UMUAI 4(4), 253–278 (1995)Google Scholar
  14. 14.
    Desmarais, M.C., Baker, R.S.: A review of recent advances in learner and skill modeling in intelligent learning environments. UMUAI 22(1–2), 9–38 (2012)Google Scholar
  15. 15.
    Dillenbourg, P., Jermann, P.: Technology for classroom orchestration. In: Khine, M., Saleh, I. (eds.) New Science of Learning, pp. 525–552. Springer, New York (2010). Scholar
  16. 16.
    Dillenbourg, P.: Trends in classroom orchestration. STELLAR 1, 5 (2011)Google Scholar
  17. 17.
    Hanington, B., Martin, B.: Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions. Rockport (2012)Google Scholar
  18. 18.
    Hapara | Making Learning Visible.
  19. 19.
    Holstein, K., Hong, G., Tegene, M., McLaren, B. M., Aleven, V.: The classroom as a dashboard: co-designing wearable cognitive augmentation for K-12 teachers. In: LAK, pp. 79–88. ACM (2018)Google Scholar
  20. 20.
    Holstein, K., McLaren, B.M., Aleven, V.: Informing the design of teacher awareness tools through causal alignment analysis. In: ICLS (in press)Google Scholar
  21. 21.
    Holstein, K., McLaren, B.M., Aleven, V.: Intelligent tutors as teachers’ aides: exploring teacher needs for real-time analytics in blended classrooms. In: LAK, pp. 257–266. ACM (2017)Google Scholar
  22. 22.
    Holstein, K., McLaren, B.M., Aleven, V.: SPACLE: investigating learning across virtual and physical spaces using spatial replays. In: LAK, pp. 358–367. ACM (2017)Google Scholar
  23. 23.
    Kai, S., Almeda, V.A., Baker, R.S., Shechtman, N., Heffernan, C., Heffernan, N.: Modeling wheel-spinning and productive persistence in skill builders. In: JEDM (in press)Google Scholar
  24. 24.
    Kelly, K., Heffernan, N., Heffernan, C., Goldman, S., Pellegrino, J., Soffer Goldstein, D.: Estimating the effect of web-based homework. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 824–827. Springer, Heidelberg (2013). Scholar
  25. 25.
    Kulik, J.A., Fletcher, J.D.: Effectiveness of intelligent tutoring systems: a meta-analytic review. RER 86(1), 42–78 (2016)Google Scholar
  26. 26.
    LanSchool Classroom Management Software.
  27. 27.
    Long, Y., Aleven, V.: Supporting students’ self-regulated learning with an open learner model in a linear equation tutor. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 219–228. Springer, Heidelberg (2013). Scholar
  28. 28.
    Long, Y., Aleven, V.: Gamification of joint student/system control over problem selection in a linear equation tutor. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 378–387. Springer, Cham (2014). Scholar
  29. 29.
  30. 30.
    Martinez-Maldonado, R., Clayphan, A., Yacef, K., Kay, J.: MTFeedback: providing notifications to enhance teacher awareness of small group work in the classroom. IEEE TLT 8(2), 187–200 (2015)Google Scholar
  31. 31.
    Mavrikis, M., Gutierrez-Santos, S., Poulovassilis, A.: Design and evaluation of teacher assistance tools for exploratory learning environments. In: LAK, pp. 168–172. ACM (2016)Google Scholar
  32. 32.
    Molenaar, I., Knoop-van Campen, C.: Teacher dashboards in practice: usage and impact. In: Lavoué, É., Drachsler, H., Verbert, K., Broisin, J., Pérez-Sanagustín, M. (eds.) EC-TEL 2017. LNCS, vol. 10474, pp. 125–138. Springer, Cham (2017). Scholar
  33. 33.
    Nye, B.D.: Barriers to ITS adoption: a systematic mapping study. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 583–590. Springer, Cham (2014). Scholar
  34. 34.
    Pelánek, R., Řihák, J.: Experimental analysis of mastery learning criteria. In: UMAP, pp. 156–163. ACM (2017)Google Scholar
  35. 35.
    Prieto, L.P.: Supporting orchestration of blended CSCL scenarios in distributed learning environments. Unpublished doctoral thesis (2012)Google Scholar
  36. 36.
    Prieto, L.P., Sharma, K., Dillenbourg, P., Jesús, M.: Teaching analytics: towards automatic extraction of orchestration graphs using wearable sensors. In: LAK, pp. 148–157. ACM (2016)Google Scholar
  37. 37.
    Ritter, S., Yudelson, M., Fancsali, S.E., Berman, S.R.: How mastery learning works at scale. In: L@S, pp. 71–79. ACM (2016)Google Scholar
  38. 38.
    Ritter, S., Yudelson, M., Fancsali, S., Berman, S.R.: Towards integrating human and automated tutoring systems. In: EDM, pp. 626–627 (2016)Google Scholar
  39. 39.
    Rodríguez-Triana, M.J., Prieto, L.P., Vozniuk, A., Boroujeni, M.S., Schwendimann, B.A., Holzer, A., Gillet, D.: Monitoring, awareness and reflection in blended technology enhanced learning: a systematic review. IJTEL 9(23), 126–150 (2017)CrossRefGoogle Scholar
  40. 40.
    Segal, A., Hindi, S., Prusak, N., Swidan, O., Livni, A., Palatnic, A., Schwarz, B., Gal, Y.: Keeping the teacher in the loop: technologies for monitoring group learning in real-time. In: André, E., Baker, R., Hu, X., Rodrigo, M.M.T., du Boulay, B. (eds.) AIED 2017. LNCS (LNAI), vol. 10331, pp. 64–76. Springer, Cham (2017). Scholar
  41. 41.
    Schofield, J.W.: Computers and Classroom Culture. University Press, Cambridge (1995)CrossRefGoogle Scholar
  42. 42.
    Stang, J.B., Roll, I.: Interactions between teaching assistants and students boost engagement in physics labs. Phys. Rev. Phys. Educ. Res. 10(2), 020117 (2014)CrossRefGoogle Scholar
  43. 43.
    Tissenbaum, M., Matuk, C.: Real-time visualization of student activities to support classroom orchestration. In: ICLS, pp. 1120–1127 (2016)Google Scholar
  44. 44.
    Xhakaj, F., Aleven, V., McLaren, B.M.: Effects of a teacher dashboard for an intelligent tutoring system on teacher knowledge, lesson planning, lessons and student learning. In: Lavoué, É., Drachsler, H., Verbert, K., Broisin, J., Pérez-Sanagustín, M. (eds.) EC-TEL 2017. LNCS, vol. 10474, pp. 315–329. Springer, Cham (2017). Scholar
  45. 45.
    Yacef, K.: Intelligent teaching assistant systems. In: ICCE, pp. 136–140. IEEE (2002)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Kenneth Holstein
    • 1
    Email author
  • Bruce M. McLaren
    • 1
  • Vincent Aleven
    • 1
  1. 1.Carnegie Mellon UniversityPittsburghUSA

Personalised recommendations