Supporting classroom orchestration with real-time feedback: A role for teacher dashboards and real-time agents

  • Mike TissenbaumEmail author
  • Jim Slotta


This paper investigates the role of the physical classroom environment, coupled with a technology environment that includes real-time agents and data analytics, to support the orchestration of complex collaborative inquiry designs in a high school physics classroom. This design-based research contributes to the wider domain of scripting and orchestration (e.g., Dillenbourg 2012; Dimitriadis 2012; Fischer et al. 2013). Guided by a theoretical perspective of learning in knowledge communities (Slotta et al., 2018), we partnered with a physics teacher to co-design curricular activities and assessments that engaged students in collectively solving, tagging and evaluating physics problems, creating a knowledge base of student-contributed examples, and using those examples as a resource in collaborative inquiry challenges. To support the teacher in orchestrating such a complex curricular design, we developed a tablet application that allowed the teacher see the state of the class in real-time, control the flow of activities and helped him know when and where he was needed within the flow of class activities. The tablet leveraged a set of specially designed real-time software agents to process student interactions in real time, allowing dynamic orchestration of student groups, material allocation, and teacher notifications. The paper begins with a review of recent literature on scripting and orchestration, drawing connection to the theoretical perspective of knowledge communities. We then describe our theoretical model, the design-based method, and details of our curriculum and technology environment. The paper concludes with a summary of how the teacher tablet and the real-time software agents helped support the teacher’s real-time facilitation and orchestration.


Learning Collaboration Orchestration Teacher dashboards 



  1. Abowd, G. D., & Mynatt, E. D. (2004). Designing for the human experience in smart environments. Smart environments: technologies, protocols, and applications, 2, 167–207.Google Scholar
  2. Alavi, H. S., Dillenbourg, P., & Kaplan, F. (2009). Distributed awareness for class orchestration. In . Sage, Distributed Awareness for Class Orchestration.Google Scholar
  3. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The journal of the learning sciences, 4(2), 167-207.Google Scholar
  4. Aronson, E. (1978). The jigsaw classroom. Sage.Google Scholar
  5. Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004, April). Off-task behavior in the cognitive tutor classroom: When students game the system. In Proceedings of the SIGCHI conference on Human factors in computing systems.Google Scholar
  6. Barab, S. & Squire, K. (2004). Design-based research: Putting a stake in the ground. The journal of the learning sciences, (October 2013), 37-41.Google Scholar
  7. Bielaczyc, K., Collins, A., O'Donnell, A. M., Hmelo-Silver, C. E., & Erkens, G. (2006). Fostering knowledge-creating communities. Collaborative learning, reasoning, and technology, 37–60.Google Scholar
  8. Brown, A. L., Ellery, S., & Campione, J. C. (1998). Creating zones of proximal development electronically. In J. G. Greeno & S. V. Goldman (Eds.), Thinking practices in mathematics and science learning (pp. 341–368). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  9. Brusilovsky, P. (2001). Adaptive hypermedia. User modeling and user-adapted interaction, 11(1-2), 87-110.Google Scholar
  10. Charles, E. S., & Whittaker, C. (2015). Active learning spaces: Blending technology and orchestration. In Exploring the material conditions of learning: The CSCL conference (Vol. 1, pp. 225-226).Google Scholar
  11. Clark-Wilson, A. (2010). Emergent pedagogies and the changing role of the teacher in the TI-Nspire navigator-networked mathematics classroom. ZDM, 42(7), 747–761.CrossRefGoogle Scholar
  12. Cole, M. (1996). Cultural psychology. A once and future discipline. Cambridge, MA: The Belknap Press of Harvard University Press.Google Scholar
  13. Cook, D. J., & Das, S. K. (2007). How smart are our environments? An updated look at the state of the art. Pervasive and Mobile Computing, 3(2), 53–73.CrossRefGoogle Scholar
  14. Crook, C. (1998). Children as computer users: The case of collaborative learning. Computers & Education, 30(3 & 4), 237–247.CrossRefGoogle Scholar
  15. De Winter, J. C. (2013). Using the Student's t-test with extremely small sample sizes. Practical Assessment, Research & Evaluation, 18(10).Google Scholar
  16. Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 5–8.Google Scholar
  17. Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with instructional design. Three worlds of CSCL. Can we support CSCL?, 61-91.Google Scholar
  18. Dillenbourg, P. (2012). Design for classroom orchestration, position paper. In P. Dillenbourg, Y. Dimitriadis, M. Nussbaum, J. Roschelle, C. K. Looi & J. Asensio (Eds.), Design for classroom orchestration. Computers & Education.Google Scholar
  19. Dillenbourg, P., Jarvela, S., & Fischer, F. (2009). The evolution of research on computer- supported collaborative learning. In N. Balacheff, S. Ludvigsen, T. Jong, A. Lazonder, & S. Barnes (Eds.), Technology-enhanced learning (pp. 3–19). Dordrecht: Springer Netherlands.CrossRefGoogle Scholar
  20. Dillenbourg, P., Zufferey, G., Alavi, H. S., Jermann, P., Do, L. H. S., Bonnard, Q., ... & Kaplan, F. (2011). Classroom orchestration: The third circle of usability. In Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceedings. Volume I—Long Papers (Vol. 1, No. CONF, pp. 510-517). International Society of the Learning Sciences.Google Scholar
  21. Dillenbourg, P., & Jermann, P. (2007). Designing integrative scripts. In scripting computer-supported collaborative learning (pp. 275-301). Springer US.Google Scholar
  22. Dimitriadis, Y. (2012). Supporting teachers in orchestrating CSCL classrooms. Research on E- learning and ICT in education, (September), 33-40.Google Scholar
  23. Dori, Y. J., & Belcher, J. (2005). How does technology-enabled active learning affect undergraduate students' understanding of electromagnetism concepts? The Journal of the Learning Sciences, 14(2), 243–279.CrossRefGoogle Scholar
  24. Dunleavy, M., Dede, C., & Mitchell, R. (2009). Affordances and limitations of immersive participatory augmented reality simulations for teaching and learning. Journal of Science Education and Technology, 18(1), 7–22.CrossRefGoogle Scholar
  25. Fischer, F., Slotta, J., Dillenbourg, P., Tchounikine, P., Kollar, I., & Wecker, C. (2013). Scripting and orchestration: Recent theoretical advances. In Proceedings of the international conference of computer-supported collaborative learning (CSCL2013) (pp. 564–571). WI: Madison.Google Scholar
  26. Furberg, A. (2016). Teacher support in computer-supported lab work: Bridging the gap between lab experiments and students’ conceptual understanding. International Journal of Computer-Supported Collaborative Learning, 11(1), 89–113.CrossRefGoogle Scholar
  27. Greene, J. C. (2006). Toward a methodology of mixed methods social inquiry. Research in the Schools, 13(1), 93–98.Google Scholar
  28. Hod, Y., Charles, E. S., Acosta, A., Ben-Zvi, D., Chen, M. H., Choi, K., et al. (2016). Future learning spaces for learning communities: New directions and conceptual frameworks. Singapore: International Society of the Learning Sciences.Google Scholar
  29. Hollan, J., Hutchins, E., & Kirsh, D. (2000). Distributed cognition: Toward a new foundation for human-computer interaction research. ACM Transactions on Computer-Human Interaction (TOCHI), 7(2), 174–196.CrossRefGoogle Scholar
  30. Hmelo-Silver, C. E. (2000). Knowledge recycling: Crisscrossing the landscape of educational psychology in a problem-based learning course for preservice teachers. Journal on Excellence in College Teaching, 11(2), 41–56.Google Scholar
  31. Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266.CrossRefGoogle Scholar
  32. Ingulfsen, L., Furberg, A., & Strømme, T. A. (2018). Students’ engagement with real-time graphs in CSCL settings: Scrutinizing the role of teacher support. International Journal of Computer-Supported Collaborative Learning, 13(4), 365–390.CrossRefGoogle Scholar
  33. Ishii, H., Wisneski, C., Brave, S., Dahley, A., Gorbet, M., Ullmer, B., et al. (1998). ambientROOM : Integrating with architectural ambient space media. Chi, 1998(April), 173–174.Google Scholar
  34. Jennings, N. R., & Wooldridge, M. (1998). Applications of intelligent agents. InAgent technology (pp. 3-28). Springer Berlin Heidelberg.Google Scholar
  35. Jennings, N. (2001). An agent-based approach for building complex software systems. Communications of the ACM, 44(4), 35–41.CrossRefGoogle Scholar
  36. Jermann, P., Soller, A., & Muehlenbrock, M. (2001). From mirroring to guiding: A review of the state of art technology for supporting collaborative learning. In European conference on computer-supported collaborative learning EuroCSCL-2001 (pp. 324-331).Google Scholar
  37. Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of mixed methods research, 1(2), 112-133.Google Scholar
  38. Kirschner, P., Strijbos, J., Kreijns, K., & Beers, P. (2004). Designing electronic collaborative learning environments. Educational Technology Research & Development, 52(3), 47–66.CrossRefGoogle Scholar
  39. Kling, R., & Courtright, C. (2003). Group behavior and learning in electronic forums: A sociotechnical approach. The Information Society, 19(3), 221–235.CrossRefGoogle Scholar
  40. Koh, E., & Hong, H. (2017). Developing professional competency in a CSCL environment for teamwork: Two TPACK case studies of teachers as co-designers.Google Scholar
  41. Kollar, I., Hämäläinen, R., Evans, M., De Wever, B., & Perrotta, C. (2011). Orchestrating CSCL–more than a metaphor. In connecting computer-supported collaborative learning to policy and practice: CSCL2011 conference proceedings (Vol. 2, pp. 946-947).Google Scholar
  42. Lemke, J. L. (2000). Across the scales of time: Artifacts, activities, and meanings in ecosocial systems. Mind, Culture and Activity, 7(4), 273–290.CrossRefGoogle Scholar
  43. Lui, M., & Slotta, J. D. (2014). Immersive simulations for smart classrooms: Exploring evolutionary concepts in secondary science. Technology, Pedagogy and Education, 23(1), 57–80.CrossRefGoogle Scholar
  44. Mason, J. (2006). Mixing methods in a qualitatively driven way. Qualitative Research, 6(1), 9–25.CrossRefGoogle Scholar
  45. Moher, T., Slotta, J. D., Acosta, A., Cober, R., Dasgupta, C., Fong, C., ... & Peppler, K. (2015). Knowledge construction in the instrumented classroom: Supporting student investigations of their physical learning environment. International Society of the Learning Sciences, Inc.[ISLS].Google Scholar
  46. Nussbaum, M., Alvarez, C., Mcfarlane, A., Gomez, F., Claro, S., & Radovic, D. (2009). Technology as small group face-to-face collaborative scaffolding. Computers & Education, 52(1), 147–153.CrossRefGoogle Scholar
  47. Ontario Ministry of Education (2008). The Ontario Curriculum Grades 11 and 12.Google Scholar
  48. O'Donnell, A. M., & Dansereau, D. F. (1992). Scripted cooperation in student dyads: A method for analyzing and enhancing academic learning and performance. In R. Hertz-Lazarowitz & N. miller (Eds.), interaction in cooperative groups the theoretical anatomy of group learning (pp. 120-141) Cambridge University press.Google Scholar
  49. O'Driscoll, C., Mithileash, M., Mtenzi, F., & Wu, B. (2008). Deploying a context aware smart classroom. Education and Development Conference.Google Scholar
  50. Palincsar, A. S., & Herrenkohl, L. R. (2002). Designing collaborative learning contexts. Theory Into Practice, 41(1), 26–32.CrossRefGoogle Scholar
  51. Papazoglou, M. (2001). Agent-oriented technology in support of e-business. Communications of the ACM, 44(4), 71–77.CrossRefGoogle Scholar
  52. Pea, R. D., & Maldonado, H. (2006). WILD for learning: Interacting through new computing devices anytime, anywhere. The Cambridge handbook of the learning sciences, 852–886.Google Scholar
  53. Penuel, W. R., Roschelle, J., & Shechtman, N. (2007). Designing formative assessment software with teachers: An analysis of the co-design process. 10.1007/s11412-019-09306-1 Research and practice in technology enhanced learning, 2(01), 51-74.Google Scholar
  54. Purba, S. W. D., & Hwang, W. Y. (2017). Investigation of learning behaviors and achievement of vocational high school students using an ubiquitous physics tablet PC app. Journal of Science Education and Technology, 26(3), 322–331.CrossRefGoogle Scholar
  55. Roschelle, J., Dimitriadis, Y., & Hoppe, U. (2013). Classroom orchestration: Synthesis. Computers & Education, 69, 523–526.CrossRefGoogle Scholar
  56. Roschelle, J., & Pea, R. (2002). A walk on the WILD side: How wireless handhelds may change computer-supported collaborative learning. International Journal of Cognition and Technology, 1(1), 145–168.CrossRefGoogle Scholar
  57. Roschelle, J., Penuel, W. R., & Shechtman, N. (2006). Co-design of innovations with teachers: Definition and dynamics. In Proceedings of the 7th international conference on learning sciences (pp. 606–612).Google Scholar
  58. Roschelle, J., Rafanan, K., & Estrella, G. (2010). From handheld collaborative tool to effective classroom module: Embedding CSCL in a broader design framework. Computers & Education, 55(3), 1018–1026.CrossRefGoogle Scholar
  59. Rubio-Fernández, A., Muñoz-Merino, P. J., & Kloos, C. D. (2019, June). Analyzing the group formation process in intelligent tutoring systems. In International Conference on Intelligent Tutoring Systems (pp. 34-39). Springer, Cham.Google Scholar
  60. Serenko, A., & Detlor, B. (2002). Agent toolkits: A general overview of the market and an assessment of instructor satisfaction with utilizing toolkits in the classroom.Google Scholar
  61. Sharples, M. (2013). Shared orchestration within and beyond the classroom. Computers and Education, 69, 504–506.CrossRefGoogle Scholar
  62. Slotta, J. D., & Linn, M. C. (2009). WISE science: Web-based inquiry in the classroom Teachers College press.Google Scholar
  63. Slotta, J. D., & Najafi, H. (2013). Supporting collaborative knowledge construction with web 2.0 technologies (In Emerging technologies for the classroom (pp. 93–112)). New York, NY: Springer.CrossRefGoogle Scholar
  64. Slotta, J., & Peters, V. (2008, June). A blended model for knowledge communities: Embedding scaffolded inquiry. In Proceedings of the 8th international conference on International conference for the learning sciences-Volume 2 (pp. 343-350). International society of the learning sciences.Google Scholar
  65. Slotta, J. D., Quintana, R. M., & Moher, T. (2018). Collective inquiry in communities of learners. In International handbook of the learning sciences (pp. 308-317). Routledge.Google Scholar
  66. Soller, A. (2001). Supporting social interaction in an intelligent collaborative learning system. International Journal of Artificial Intelligence in Education (IJAIED), 12, 40–62.Google Scholar
  67. Stanley, K. O., Bryant, B. D., & Miikkulainen, R. (2005). Real-time neuroevolution in the NERO video game. Evolutionary Computation, IEEE Transactions on, 9(6), 653–668.CrossRefGoogle Scholar
  68. Schwarz, B. B., Prusak, N., Swidan, O., Livny, A., Gal, K., & Segal, A. (2018). Orchestrating the emergence of conceptual learning: A case study in a geometry class. International Journal of Computer-Supported Collaborative Learning, 13(2), 189–211.CrossRefGoogle Scholar
  69. Szewkis, E., Nussbaum, M., Rosen, T., Abalos, J., Denardin, F., Caballero, D., Tagle, A., & Alcoholado, C. (2011). Collaboration within large groups in the classroom. International Journal of Computer-Supported Collaborative Learning, 6(4), 561–575.CrossRefGoogle Scholar
  70. Tchounikine, P. (2013). Clarifying design for orchestration: Orchestration and orchestrable technology, scripting and conducting. Computers & Education, 69, 500–503.CrossRefGoogle Scholar
  71. Tchounikine, P. (2016). Contribution to a theory of CSCL scripts: Taking into account the appropriation of scripts by learners. International Journal of Computer-Supported Collaborative Learning, 11(3), 349–369.CrossRefGoogle Scholar
  72. Tissenbaum, M., Lui, M., & Slotta, J. D. (2012). Co-designing collaborative smart classroom curriculum for secondary school science. Journal of Universal Computer Science., 18(3), 327–352.Google Scholar
  73. Tissenbaum & Slotta (2015), Scripting and orchestration of learning across contexts: A role for intelligent agents and data mining. In Milrad, Wong & Specht (eds.) Seamless learning in the age of connectivity. Springer.Google Scholar
  74. Tissenbaum, M., & Slotta, J. D. (2019). Developing a smart classroom infrastructure to support real-time student collaboration and inquiry: A 4-year design study. Instructional Science, 1–40.Google Scholar
  75. van Aalst, J., & Chan, C. K. (2007). Student-directed assessment of knowledge building using electronic portfolios. The Journal of the Learning Sciences, 16(2), 175–220.CrossRefGoogle Scholar
  76. Weiser, M., & Brown, J. S. (1996). Designing calm technology. PowerGrid Journal, 1(1), 75–85.Google Scholar
  77. White, T. (2018). Connecting levels of activity with classroom network technology. International Journal of Computer-Supported Collaborative Learning, 13(1), 93–122.CrossRefGoogle Scholar
  78. Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(02), 115–152.CrossRefGoogle Scholar
  79. Yau, S. S., Gupta, S. K., Karim, F., Ahamed, S. I., Wang, Y., & Wang, B. (2003). Smart classroom: Enhancing collaborative learning using pervasive computing technology. In ASEE 2003 Annual Conference and Exposition (pp. 13633–13642).Google Scholar

Copyright information

© International Society of the Learning Sciences, Inc. 2019

Authors and Affiliations

  1. 1.College of EducationUniversity of Illinois Urbana-ChampaignChampaignUSA
  2. 2.Ontario Institute for Studies in EducationUniversity of TorontoTorontoCanada

Personalised recommendations