The contributions to this issue represent diverse theoretical perspectives, technological platforms, educational contexts, and design approaches to real-time data capture, aggregation, and visualization of collaborative learning. Collectively, they address themes related to the design, theory, and impact of real-time dashboards in authentic CSCL environments, and help advance the conversation by offering critical perspectives and empirical evidence for the value of real-time dashboards in orchestrating collaborative learning in classrooms. We summarize each paper and discuss some common themes below.
The study by van Leeuwen, Rummel and van Gog examines how real-time data can support teacher noticing, a practice that involves identifying which students and behaviors require attention, and determining how to respond (Van Es and Sherin 2002, 2008). Given that teachers must quickly and accurately ascertain when and how they are needed during class time, the authors ask how information can be best presented on a dashboard to promote teachers’ speed in detecting, and accuracy in interpreting issues with students’ collaborative work.
To explore this question, van Leeuwen et al. created a dashboard prototype for MathTutor, an environment that supports dyads in the conceptual reasoning and procedural practice in the domain of mathematics. The researchers first engaged teachers in a co-design process to determine what specific information to display. Then, in a laboratory-based, between-subjects experiment, the authors compared teachers’ response times to, and interpretations of dashboards showing fictitious classroom scenarios of students working on fractions problems. These scenarios were designed to reflect different combinations of social and cognitive issues. For example, one member of the dyad might be monopolizing the activity, students might be taking turns instead of collaborating, a dyad might be approaching the task through trial-and-error rather than through discussion, the dyad might be stuck on a particular kind of problem, and so forth. The authors compared teachers’ abilities to identify the social and/or cognitive issue of these situations given three different levels of real-time information: a mirroring dashboard showed the status of the classroom’s activities; an alerting dashboard showed the classroom status and also highlighted students in need of attention; and an advising dashboard showed the classroom status, highlighted students in need of attention, and also offered advice on how to address those potential issues.
The authors found no significant differences between conditions in teachers’ speed and confidence in interpreting the classroom situations. However, the teachers tended to interpret social and affective reasons for the situations that went beyond the initially identified cognitive or social issue. Van Leeuwen et al. also found that with the advising dashboards, teachers spent longer considering the information displayed, provided richer interpretations of the situations, and even questioned and disagreed with the advice that these dashboards displayed. These findings resonate with other research, which highlights the importance of trust in recommendation systems for ensuring their adoption. They also resonate with research that suggests the benefit of guiding teachers to interpret data. Altogether, van Leeuwen et al.’s findings offer preliminary evidence for the value of advising dashboards, and suggest that such information can complement teachers’ own observations of their students, enrich their interpretations of the situations, and inform their decisions on how to act on that information.
In the second paper Gerard, Kidron & Linn ask how real-time guidance can help teachers support their students’ collaborative revision of science explanations. To do this, they use natural language processing (NLP) to automatically assess middle school students’ written science explanations in the Web-based Inquiry Science Environment (WISE). The system then generates written guidance, based on a rubric informed by the Knowledge Integration perspective (Linn and Eylon 2011). This guidance encourages students to consider missing or inaccurate ideas, and revisit a relevant visualization in the unit in order to verify that idea, and can be customized by the teacher before being sent to students.
The authors conducted a classroom-based implementation of a plate tectonics unit with one teacher of 6th grade students. Based on audio and video recordings of teacher-student interactions, as well as students’ responses to a pretest, posttest, and embedded assessments, the authors identified different ways that the teacher used the real-time feedback to personalize the guidance that they ultimately gave to their students. For example, she directed students with partial understanding to revisit visualizations in order to gather more evidence, and prompted more advanced students to evaluate and identify missing ideas. The authors also found that students made more substantial revisions on the posttest than on the pretest, thus demonstrating that real-time data can support teachers in guiding their students to collaboratively revise their science explanations.
Resonant with the study by van Leeuwen et al., this study highlights the knowledge that teachers bring to their interpretations of classroom situations, and the need for a system to take that knowledge into account. It shows how, by integrating automated assessment and feedback into teachers’ instructional practices, a real-time system can augment teachers’ abilities to guide their students. In this case, pairing system-generated assessment with teachers’ personal knowledge of their students ensured that students received both timely and personalized guidance that contributed to their improved revision practices and learning outcomes.
In the third paper Tissenbaum and Slotta developed and studied the role of real-time software agents in orchestrating collaborative inquiry in a high school physics classroom. Software agents can be programmed to respond to particular conditions in an environment, essentially mining data in real-time, including artifacts, emergent metadata, and other traces of individual and collaborative learning.
Guided by the Collective Inquiry and Learning Communities framework (Slotta et al. 2018), the authors used a design-based research approach to implement a curriculum within a smart classroom environment. They integrated software agents to support various aspects of students’ collaborative activity, including coordinating their changing locations around the room, displaying their community-constructed knowledge base, and showing the time remaining on different tasks. This information passed into the teacher’s tablet, which informed him of student groups’ progress through activities; allowed him to dynamically regroup students based on their previous interactions in the room; and facilitated the distribution of content from the students’ collectively developed knowledge base, according to their real-time needs.
The authors found that by offloading managerial duties, the system allowed the teacher to act as a wandering facilitator of student learning in his classroom. They also found that the teachers’ access to real-time alerts about group work, provided at key moments during the activity, had a significant impact on students’ physics problem-solving approaches. Overall, this study shows how real time data can support students and teachers during complex inquiry, and particularly within environments designed to give leverage to both the physical and digital dimensions of collaborations.
In the fourth paper Olsen, Rummel and Aleven investigated the value of collaborative and individual work on elementary students’ learning about fractions. Their study focused on a collaborative intelligent tutoring system (CITS), which tracks students’ real-time activity, and uses this to provide students with real-time cognitive and social support during their work. For example, the system might stop and redirect students who have proceeded too long in the wrong direction, provide a common focus for partners’ discussion, or offer correctional feedback on their responses. The CITS additionally incorporated group awareness and group accountability features to promote effective collaboration. Thus, student partners sit side-by-side, but view different versions of the same activity on their screens, and through a collaborative script, may be assigned different responsibilities on the same problem.
Olsen et al. conducted a quasi-experimental classroom-based study with 4th and 5th grade students. In their study, they compared the relative benefits for elementary students learning fractions with a CITS when working individually, collaboratively, or through activities that combined individual and collaborative work. The authors found various positive effects of collaboration. For example, students in the combined condition requested fewer hints, and made fewer errors than students in the collaboration-only and the individual-only conditions. These students also finished with higher learning gains than students who only worked collaboratively or who only worked individually, and also reported higher situational interest in the activity.
In contrast to the other studies in this issue, which focused on how real-time data can support teachers, Olsen et al. show how real-time data can serve students directly. By informing students of their partner’s state of knowledge, and by incorporating structures for accountability, this study shows how student-facing real-time data can play a role in enhancing students learning from, and interest in collaborative problem solving.
In the fifth paper Martinez-Maldonado’s study sought to document university instructors’ perspectives on using a mobile orchestration tool in their information science classrooms. Through a two-year participatory design and evaluation process with the instructors, the author designed and developed a mobile dashboard to support them in orchestrational and assessing collaboration and progress in a multi-week interactive tabletop activity. This tablet provided visualizations that gave the teacher insight into individual students’ participation and overall group progress in activities. The tablet also allowed students tabletops to be remote controlled, such as to be paused for a whole class announcement, or advanced to the next activity. Additionally, the tablet provided real-time alerts to the teacher to notify them when time allocated to a task had run out or when a known misconception was detected.
Martinez-Maldonado conducted a longitudinal study of four instructors using the mobile dashboard with 150 students over 72 classroom sessions during a 10-week period. A qualitative analysis of observations and interviews with instructors showed evidence for the potential of the technology for helping instructors to assess group collaboration, monitor class task progression, and highlight groups in need of the instructor’s assistance.
Notably, Martinez-Maldonado’s findings also point to the trade-offs of real-time data and the format in which data are delivered. For example, the instructors commented on the orchestrational load introduced by various data streams and visualizations, raising the question of when, and in what format, more data becomes less, rather than more helpful. As well, having the dashboard on a mobile device was both convenient for allowing instructors to circulate the classroom, and frustrating in that it kept one hand constantly occupied. The findings also flagged the potential issue of instructors’ over-reliance on real-time data, as such data give an inherently incomplete picture of the classroom, and that its immediacy sometimes encourages reaction rather than reflection. Overall, Martinez-Maldonado’s study shows the value of seeking instructors’ perspectives following their long-term use of real-time tools, as these can provide more balanced views of their affordances and trade-offs.