Designing for the co-Orchestration of Social Transitions between Individual, Small-Group and Whole-Class Learning in the Classroom

Abstract

Educational technologies are often developed such that students work on specific social levels (e.g., individual, small group, whole class) at specific times. However, in the reality of the classroom, learning activities are not so cleanly divided, with transitions occurring between social levels for students at different times. To support these social transitions in a way that can promote student learning, we need to lower the teacher’s orchestration load around managing fluid social transitions. Co-orchestration, in which the orchestration decisions are shared between different parties, can help to lower the orchestration load when it is designed according to the teacher’s values and classroom culture. In this paper, we present a taxonomy of social transitions and investigate how the responsibilities of orchestration can be divided between primary school teachers and a co-orchestration system in order to support the extension from rigid social transitions to fluid transitions in technology-enhanced classrooms. Across two studies, we used a design process involving co-design and prototyping with teachers. We uncovered and refined co-orchestration design desires that balance teachers’ orchestration loads while providing them with a sense of control. We present six design desires for maintaining a balance between teacher and system responsibilities regarding the orchestration of social transitions that can be implemented, such as in our mid-fidelity prototype, to support the range of social transitions. The list of desires contributes to co-orchestration research and more broadly technology design for classrooms by highlighting the changing balance of teacher control depending on what is the focus of the orchestration support.

Introduction

Consider a classroom in which the teacher plans an activity for the students to work together in pairs in order to solve a set of problems and then share their answers with another pair. This is a common lesson design that, despite being routine for the teacher, can still be difficult to run in the classroom. For example, as the teacher is getting the students started on the activity, one student tells the teacher that he/she does not have a partner. The teacher has to break off from giving instructions to the class in order to identify any other students without a partner and pair them up. Then, as the students begin working in their pairs, the teacher walks around the room to help the pairs. As the teacher is helping one pair, another pair interrupts to say that they have completed the activity and to ask what to do next. The teacher informs them that they should wait until another pair is ready and then form a group to discuss their answers. As the teacher goes back to helping the original pair, yet another pair interrupts to say that they too have finished. The teacher must find the first finished pair and tell them to work together. The teacher finds that as more and more students finish the activity, he/she is no longer able to help the students who are still struggling, as all of his/her time is being spent on helping the students who are finishing to transition to the next activity. Thus, in this example, the teacher takes longer to get students started at the beginning of class due to dealing with absent students, and the majority of the class time is used to help students to transition between activities.

These actions demonstrate the classroom orchestration in which teachers must continuously engage to not only support the separate learning activities but also with respect to the transitions that occur between the activities. Classroom orchestration describes the planning and real-time support and adaptation of a lesson to events that unfold in the classroom (Dellatola and Daradoumis 2014; Kollar and Fischer 2013; Prieto et al. 2011a). As the example above demonstrates, social transitions can occupy a significant amount of the teacher’s orchestration time and effort in the classroom, particularly when these social transitions are fluid. In the classroom, social transitions can be planned, such as moving from an individual to a whole-class activity, or can arise due to extrinsic constraints, such as a student being absent, which changes the social structure of groups. Fluid social transitions are those that occur asynchronously between students – not all at the same time for everyone in the class. As the transitions become more fluid, teachers may struggle to enact the lesson in their classroom without support. Moreover, classroom orchestration can become even more difficult for the teacher when the students are working with educational technology and the teacher not only has to instruct the students what to do but also has to provide directions regarding the software. In this paper, we are specifically interested in how we can develop classroom orchestration technology to support fluid social transitions in a technology-enhanced classroom, where individual and collaborative learning is interleaved, in a way that aligns with teachers’ values.

Currently, much of the technology developed to support learning in the classroom is used to support students working in a single social level, defined as the grouping (e.g., individual, small group, whole class) within which students are working at a specific time. For example, numerous studies have demonstrated that intelligent tutoring systems (ITSs) support individual student problem solving through the use of individualized feedback and problem selection (Ma et al. 2014). Small-group learning has similarly found support through the use of technology, which has been explored in the field of computer-supported collaborative learning. For example, technology can be used to create student groups more effectively by taking into account students’ previous work (Marcos-García et al. 2009), and students can be assigned different roles within the collaboration (Roscoe et al. 2018). However, in the reality of the classroom, lessons do not merely comprise of a single social level; rather, teachers usually combine multiple social levels in order to engage their students (Dillenbourg 2004; Dillenbourg and Tchounikine 2007). Moreover, at times, students may work on different levels to those intended by the teacher (Ogan et al. 2012). Previous research has shown that these combinations of social levels can be beneficial for learning (Mullins et al. 2011; Olsen et al. 2017; Rodriguez et al. 2017). Thus, it is important to not only consider how we develop support for activities in one social level, but also how we develop support across social levels and for transitioning between them.

However, technologies that allow for students to work across multiple social levels are often still designed with little flexibility in terms of social transitions. When planned transitions are administered synchronously across the class and all students move to the next activity at the same time, we lose some flexibility in supporting individual student learning. Students are forced to proceed at the same pace regardless of their skill level, and have to work on the prescribed social level even if another level may be more beneficial. For unplanned transitions, workarounds are often put in place to prevent the need for the transition rather than providing a way for the transition to be easily addressed within the technology. Such workarounds include having students start at a fixed point at the beginning of each class, enabling students to be paired up more easily (Walker et al. 2009), pairing students at the beginning of each class to avoid having to shuffle groups due to students being absent (Walker et al. 2011), working with student groups comprising of more than two people, or only implementing the intervention over one session (Celepkolu et al. 2017; Martinez-Maldonado et al. 2011; Rodriguez et al. 2017; Walker et al. 2014). Only more recently have technologies begun to support more flexible scripts to support students arriving late or leaving early (Manathunga and Hernández-Leo 2019).

Instead of relying on less-than-ideal workarounds for unplanned social transitions and having rigid planned social transitions, orchestration systems can be used to extend the use of technology in classrooms in order to support more fluid transitions that can be beneficial to student learning. Orchestration systems are technologies that support the teacher in implementing, running, and monitoring a classroom activity, lowering their orchestration load and enabling them to more effectively support students (Prieto et al. 2011a; Verbert et al. 2014). In recent years, there has been an increased focus on the use of orchestration systems to provide teacher support for classroom activities. Specifically, these systems have shifted from primarily supporting teachers in taking orchestration actions to sharing this responsibility and decision-making process across the system, teachers, and students, which is known at co-orchestration (Holstein et al. 2018a; Prieto 2012). Co-orchestration describes a situation in which the responsibility for making orchestration decisions is shared across multiple entities and can be used to lower the orchestration load of teachers. Previous research has implicitly considered that roles can be shared among multiple entities (Sharples 2013; Prieto 2012; Prieto et al. 2015) but only recently have researchers investigated what it means to have an effective shared control for different types of support (Holstein et al. 2020). Much of the current research focuses on the design of co-orchestration systems to support teachers in supporting student learning during a single activity (Holstein et al. 2019; van Leeuwen et al. 2019). Although the needs and values of the teacher may be the same whether they are providing cognitive, social, or social transition support, the role that they want to play in each of these processes may differ. It is then imperative to not treat all of these situations as similar but instead to understand in what areas the similarities and differences lie. We focus on the teacher’s perspective in this paper as a first step to understanding the division of orchestration responsibilities around social transitions. We decided to focus on the teacher perspective to scope the design space; involving students is left for future work. As classroom orchestration is teacher-centric (Dillenbourg and Jermann 2010), working with teachers is a natural starting point.

In this paper, we aim to investigate how the responsibilities in a co-orchestration system can be divided to support a range of social transitions. To address this question, we engaged in a design process to design and investigate orchestration support for fluid transitions in a primary school technology-enhanced classroom through a series of two studies that culminated in a mid-fidelity prototype tested with teachers. This paper makes contributions to co-orchestration research, and the design of intelligent learning technologies more broadly, by addressing the new topic of how we can support fluid social transitions in the classroom with technology in a more deliberate way than is currently the case.

The rest of the paper is structured as follows: The next section provides an overview of classroom orchestration systems, a transition taxonomy that outlines the space for social transitions, and our research question. In third and fourth section respectively, we present our design sessions and findings (Study 1) and our follow-up system scenario sessions (Study 2). Finally, we conclude with an overall discussion of our design desires in fifth section and a conclusion in sixth section.

Related Work

Classroom Orchestration

For orchestration systems to support fluid social transitions, there are several aspects that the system must encompass. These aspects include a range of supported social levels, the transitional support, the flexibility for students to work at their own pace, and the sharing of the orchestration responsibility. In the classroom, students often work across multiple social levels within a single lesson. However, many of the orchestration systems that have been developed focus on a single type of learning scenario or may only focus on the orchestration within a single social level. Some systems (exclusively) support individual learning (Holstein et al. 2018b), collaborative learning (Alavi and Dillenbourg 2012; Cuendet et al. 2011; Mercier 2016; van Leeuwen et al. 2017), or whole-class learning (Raca and Dillenbourg 2013). Other orchestration systems provide support for a subset of social levels that are relevant to a specific learning activity (Manathunga et al. 2015; Martinez-Maldonado et al. 2013, 2015a). By contrast, some systems have a more open design to support a range of social levels (Håklev et al. 2017; Looi and Song 2013; Muñoz-Cristóbal et al. 2015; Phiri et al. 2016; Prieto et al. 2014; Van Lehn et al. 2016; Wang et al. 2015, 2018). By supporting students’ learning across multiple social levels, as the more open designs do, a system is often able to fully support classroom learning rather than merely being used for a single activity.

Additionally, only when a system can support multiple social levels can it begin to provide support beyond the monitoring of an activity and help the teacher to transition between activities. When students remain on the same social level and do not undergo any social transitions, such as when working on multiple problem sets in an ITS, there is less coordination between activities for the teacher to manage. The majority of orchestration systems that provide support for multiple social levels have limited support for smooth transitions (Niramitranon et al. 2010; Looi and Song 2013; Wang et al. 2015, 2018). Other systems that do provide support around transitions often focus on the timing of the individual activities, by displaying timers or lesson timelines (Håklev et al. 2017; Martinez-Maldonado et al. 2013, 2015a). These time indicators still leave the majority of the orchestration load for the teachers and do not provide any support for the social transitions.

Moreover, to avoid the complexity of social transitions, the majority of orchestration systems restrict students to all move at the same pace between activities. Within activities, many systems will provide progress indicators to help teachers understand where individual students are in the process (Alavi and Dillenbourg 2012; Wang et al. 2015, 2018). However, at the end of each activity, all students are expected to move to the next activity, meaning that those who finish early are left waiting while others may not get time to complete the task. Several systems, like FACT (Van Lehn et al. 2016) and PyramidApp (Manathunga and Hernández-Leo 2019), allow for some flexibility for students to work out of sync, but these mostly address the case of students who start the activity late rather than encompassing asynchronous pacing as a supported lesson design feature.

Finally, to support more complex learning designs, such as fluid social transitions, the orchestration load needs to be shared across different parties (e.g., teachers, students, the system). Some orchestration systems accomplish this role distribution on an ad hoc basis for the specific system (Muñoz-Cristóbal et al. 2013, 2015). However, more recent work has suggested that a more proactive approach should be taken to design co-orchestration systems for both individual and collaborative learning (Holstein et al. 2018a; Prieto 2012). Existing orchestration recommendations provide guidelines for designing an orchestration system (Dillenbourg 2009; Dillenbourg and Jermann 2010; Dillenbourg et al. 2011; Prieto et al. 2011a) but they do not address co-orchestration designs or consider which role distributions might be most beneficial. More recent work has considered the division of roles for providing cognitive and social support to students within a single activity (Holstein et al. 2019; van Leeuwen et al. 2019) building on these existing recommendations. In this paper, we aim to extend this knowledge to desires for co-orchestration systems that support fluid social transitions across multiple social levels.

Transition Taxonomy

To develop technology that supports social transitions within a lesson, we first must understand what are the range of possible social transitions. In this section, we present a transition taxonomy that outlines a range of social transitions that can occur in a classroom distilled from prior literature (see Table 1). In classroom orchestration, two different types of social transitions can occur, namely, those that are based on pedagogical theory and those that are based on pragmatic limitations in the classroom (Dillenbourg 2013). Both types of transitions can add extra load on the teacher, whether designed or not.

Table 1 Taxonomy of different transition types and their alignment with the scenarios used in Study 1 and Study 2

Social transitions are pragmatic transitions (see rows 1–6 of Table 1) when they are not based in learning theory but still are practical considerations that add to the teacher’s load in the classroom. The classroom can be thought of as a system that is continually changing and adapting. Students leave and enter the classroom throughout the day either due to sickness or other commitments, which leads to the teacher needing to make changes to planned groupings (see rows 1–4 of Table 1). These social transitions are often unplanned and can lead to changes for the whole activity, such as when a student is out sick, or to parts of the activity, for example, due to a student having a conflicting appointment for part of the lesson. The transitions can result either in having fewer students in the activity than expected, or in students joining the activity after it has begun. Because these transitions only impact a subset of the class, they occur asynchronously.

A second type of pragmatic transition arises from limited time and space in the classroom limiting the number of students that can be engaged in an activity at a time (see Table 1, rows 5–6), In this case, there may be multiple “stations” available in the classroom that students will rotate through. Within each station, students may be working in a different group size, meaning not all students would be working in the same group size at the same time. As students rotate through the stations, they will engage in different activities, making these transitions between activities. Often to keep the balance of number of students at each station equal, the transitions between stations would happen all at the same time, or synchronously. However, as students work at a different pace, they may be allowed to move through the stations at their own pace, supporting asynchronous transitions.

In contrast to pragmatic transitions, pedagogical transitions are social transitions that are part of the learning design, often inspired by learning theory. For transitions that occur between activities, we can look at research on collaboration scripts to understand the range of transitions (see Table 1, rows 7–12). Collaboration scripts often span multiple social levels to support student learning (Dillenbourg and Tchounikine 2007). Within these scripts, students may transition from working in a smaller group to a larger group in which they can share information, such as in the Pyramid Script or Think-Pair-Share (Lyman 1987; Manathunga and Hernández-Leo 2019). Students may also work within larger groups and later transition to smaller groups. Furthermore, social transitions may occur when the members of the groups change to support different types of idea exchanges, such as in a Jigsaw Script when students move from expert groups to mixed groups (Aronson 1978). Finally, as with the pragmatic between-activity transitions, the pedagogical between-activity transitions can occur either synchronously or asynchronously (Dillenbourg et al. 2016). In other words, students can all move to the next activity of the script either as a whole class, or as students complete their current activity, they can transition to the next activity at their own pace (Manathunga and Hernández-Leo 2019).

Finally, students may transition within an activity when their current social level is not supporting their learning needs (rows 13 and 14 in Table 1). To understand this type of transition, we can look to adaptive learning in both individual and collaborative tasks as well as what occurs naturally in the classroom. With regard to both individual and collaborative learning, learning support is becoming increasingly adaptive. Thanks to this adaptivity, educational interventions do not have to be one-size-fits-all but are able to account for students’ knowledge, actions, dialogues, or affective states (Aleven et al. 2017; Magnisalis et al. 2011). This, in turn, can increase the likelihood that a learning activity is at the correct level for the students. For example, when students are engaged in individual work, these changes can be on a lower level, such as providing different feedback on problem-solving steps (Mitrovic et al. 2013) or prompting students to take time to consider a hint if they have weak help-seeking behaviors (Roll et al. 2012). In a collaborative setting, the cognitive and social interventions can take place, for example, through the use of prompts, tutor dialogue, or roles (Kumar et al. 2007; Walker et al. 2014) when the technology detects that the students are not having a productive collaboration. The majority of the adaptive learning support that occurs is restricted to either supporting students working individually or those working collaboratively. Little research has examined adaptive learning support across social levels, the implementation of which can require a higher orchestration load. However, we do observe these transitions occurring naturally in the classroom such as when students ask a classmate for support on a problem (Ogan et al. 2012). Students may also decrease in group size such as when they divide a task to work on it individually. These types of social transitions occur within an activity. Because these transitions are based on the individually needs of the students, they naturally occur asynchronously during a class.

Research Question

When aiming to design a system that meets classroom needs, researchers often observe current classroom practices (Looi and Song 2013; Martinez-Maldonado et al. 2015b) and then use their expertise to design the system independently. However, it can be important to work with teachers outside of an observable classroom context in order to gain an understanding of the values and experiences that need to be accounted for in not-yet-realized systems. Moreover, the researcher can address how these values and experiences can be integrated with the researcher’s own expertise when designing the orchestration system. Teachers have existing cultures within their classrooms, and for a new technology to be adopted in the long term, it needs to align with their values, as expressed through their beliefs and current teaching practices (Blumenfeld et al. 2000; Windschitl and Sahl 2002). By understanding teachers’ classroom practices, co-orchestration and the distribution of responsibilities can be designed to support teachers within their classrooms rather than restricting their control.

To investigate the teachers’ perspectives and experiences regarding classroom orchestration of social transitions, we aim to include teachers throughout the design process (Spinuzzi 2005). Including participants in the design process is becoming more common within education. Then using thematic analysis (e.g., affinity diagramming), we can design educational technology that will be useful in the classroom rather than technology which merely aligns with verbally expressed requirements. Thematic analysis supports this alignment by analyzing the subjective data in a rigorous bottom-up approach. The themes emerge from the grouping of the data as they relate to the research question (Aronson 1995; Fereday and Muir-Cochrane 2006) rather than imposing preconceived themes.

The aim of this study is to investigate how a co-orchestration system can be designed to lower teachers’ orchestration loads while still providing them with a sense of autonomy for social transitions. Specifically, we raise the following research question: How should orchestration responsibilities be divided between the system and teachers to productively support teachers in orchestrating fluid social transitions in a technology-enhanced classroom? To address this question, we used two design studies. We first engaged in co-design design sessions with teachers to develop a better understanding of teachers’ values in the classroom and of how orchestration roles can be distributed between a teacher and a system to lower teachers’ orchestration loads around social transitions. If roles are not well distributed, the technology can lead to greater load rather than a reduction. Using our findings, we propose a set of design desires for shared control around the co-orchestration of social transitions. Second, we assessed the relevance of these desires with teachers by asking teachers to review a series of technology-supported classroom orchestration scenarios, allowing teachers to reflect on the shared control.

Study 1

We began our design process with a series of co-design design sessions in which we engaged the teachers in joint design sessions with a researcher. We conducted these sessions in order to better understand how social transitions can be integrated into the classroom. Through this method we can support each teacher in reflecting on his/her own teaching practices (Penuel et al. 2007; Sanders and Simons 2009) while creating a tighter coupling between these teaching practices, curriculum, and technology (Penuel et al. 2004). In turn, this practice can lead to the development of technology that is more closely aligned to classroom needs.

In these sessions, teachers were able to provide a researcher with insights into their experiences of managing their classroom with respect to activity transitions. The outcome of these sessions was a set of design desires for how to divide co-orchestration responsibilities for social transitions, specifically shared with an orchestration system, in the classroom while still providing a sense of teacher autonomy and control over the social transitions. The desires were based on how the teachers designed the learning experience around certain scenarios and their discussions of past experiences that guided these designs.

Methods

Participants

A total of seven teachers (six females, one male, on par with the US average distribution of gender among teachers; U.S. Department of Education, 2012) participated in the sessions. The teachers represented a convenience sample although we aimed to recruit teachers from a wide range of school districts to represent a range of experiences. The teachers ranged from 2nd to 7th grade teachers and came from five different schools. The teachers who taught older grades taught math and technology subjects, while the teachers in lower grades taught all of the primary school core subjects. Across the schools, there were different levels of technology integrated into the classroom, ranging from only whole-class technology (e.g., SmartBoards) with infrequent access to individual use of technology, to daily individual access to computers/tablets. All of the teachers had experience in designing lessons that spanned multiple social levels. The teachers’ number of years of teaching ranged from 1.5 years to 15.5 years (M = 11.2), allowing us to work with teachers with a range of experiences. Four of the teachers had worked with our research team in some capacity on previous experiments pertaining to technological interventions, while the other three had not previously worked with us on other projects.

Materials

To support the design process, we provided a set of materials that could be used to express the designs. The materials included paper, scissors, precut shapes, pens, and markers. At the beginning of the session, all participants were informed about the materials and the researcher explained how they could be used throughout the process. However, the majority of the teachers did not use the materials; instead, the ideas emerged through dialogue between the teachers.

Design and Procedures

We conducted three sessions, each lasting for two hours, with each teacher attending one session. Two sessions included two teachers and one session included three teachers, along with a researcher who led the sessions. The sessions were designed to solicit teacher input about how to support social transitions in the classroom so that the benefits of fluid transitions could be explored. Each session was semi-structured, and centered around descriptions of specific scenarios involving social transitions within the classroom. These scenarios provided the teachers with a context in which to discuss their classrooms but did not confine the discussion. Accordingly, the teachers were able to lead the discussions but within the framework provided by the researcher-developed scenarios. We engaged the teachers in five main scenarios that aligned with the social transitions outlined in Table 1. These scenarios covered all of the types of social transitions within our taxonomy except for transitions that occurred between activities that were at the same social level. We did not include this scenario as we were mainly focused on the pragmatic social transitions and those between social levels. During the five main scenarios, we probed with further follow-up questions to create specific sub-scenarios for the teachers to consider that may influence how they would have students transition within their classroom. These questions probed at the transitions when there were pre-assigned groups, groups were being made in real-time, and when students were struggling to complete a prerequisite activity.

For each of the scenarios, we asked the teachers to design the flow of materials and information as well as distribute responsibility between the students, teachers, and an orchestration system that is not restricted by what they believe current technology can support. The teachers were prompted to design the physical layout of the classroom – as having students work on different activities can impact the movement in the classroom, what they see as their role in the scenario, what the student responsibilities and series of events would be, and what the system would be doing within this process. For the distribution of roles in the classroom, the teachers were asked to think about who would be making decisions, when these decisions would be made in the process, and how these decisions would be communicated within the class. Through the design discussions, we intended to elicit past experiences and pain points that the teachers had within their classroom based on where they wanted to spend their time and where they were willing to share the responsibility of orchestrating.

Data Analysis

Each of the two-hour sessions was video-recorded. To analyze the results, we used thematic analysis (Aronson 1995; Fereday and Muir-Cochrane 2006). This allowed the relevant themes tied to our research questions (i.e., how and when to distribute the roles in a co-orchestration system supporting social transitions) to emerge naturally through a bottom-up process rather than imposing any predefined ideas upon the data. We conducted this analysis by means of an iterative process using affinity diagramming (Miles et al. 2013), which involved the grouping and re-grouping of individual pieces of data to find common themes across sessions.

The data that were used for grouping in the thematic analysis were extracted by a researcher, who reviewed the videos and transcribed on-topic dialogue into separate notes. Each note represented a complete thought expressed by a single participant during the sessions. An example of a note is as follows: “You want to be able to have some say but then with the computer, see what happens, and then monitor”. In this example, the teacher is expressing his/her frustration that the computer often acts as a black box in terms of decision making. Although the materials were provided during the session, the teachers did not use them as they discussed the scenarios. In this case, there were no artifacts to analyze and the group dialogues were the primary source of analysis. Across the three sessions, we obtained a total of 147 notes. The primary researchers grouped these notes into overarching themes, which included both expressed goals and inhibitors. These note groupings were discussed and verified by three members of the research group and reassessed if required. The initial work by a single researcher may have introduced bias into the notes that the review and discussion between multiple researchers aimed to address. By engaging in the thematic analysis, we were able to evaluate the data with respect to shared control around co-orchestration rather than reproducing features of orchestration that have already been reported within the literature, such as the need for awareness and flexibility. Instead, we focus on who is responsible for the different aspects of the awareness and flexibility in the orchestration cycle.

Findings

During the sessions, as they designed for future scenarios, the teachers used past experiences to express their values and frustrations within the classroom. The themes that emerged from the sessions are common orchestration themes, i.e., the importance of planning, classroom monitoring and flexibility. However, in terms of the role that the teacher plays in these areas of orchestration, we found more nuanced patterns.

Data analysis revealed that the main concern for teachers was their ability to support student learning. During each of the sessions, the teachers discussed impacts on student learning, such as not having enough support and finding it “very frustrating to have [students] leave your room knowing that they could have done more” (T6), students working while you are there “and then when you leave, they just stop” (T2), or understanding “what happened today that everyone was seeking help” (T4). In the classroom, teachers “are trying to stay on this timeline” (T1) while providing effective support for their students. Whenever they consider taking an action in the classroom, they weigh the time that this will take against the benefit it will bring to the students. Our results grouped naturally into four different aspects: how teachers wanted to spend their time, how teachers were willing to invest their time, how teachers did not want to spend their time, and for what they did not feel they had sufficient time. An understanding of these different uses of time can help to foster our understanding of where the support from a co-orchestration system would be most beneficial and what role it can play in the orchestration process of social transitions.

How Teachers Want to Spend their Time

Throughout the sessions, a main theme that continuously emerged was the importance of having the classroom activities planned before class. In all three sessions, the teachers discussed the preparation work that they did before class. If an activity was not planned in advance, it added too much orchestration load and it became hard for the teachers “to walk around and juggle” the different tasks (T7). Additionally, when a lesson is not prepared ahead of time, there is more downtime and “for that quick and quiet transition, [one has to have] the materials ready ahead of time” (T5). Lesson planning provides the first opportunity for teachers to determine how to support students in their learning. When planning, teachers must make a judgment call, as they “are looking at the formative assessment to understand if there needs [to be] a little more [instruction], but there are still those [curriculum requirements] that [they] need to meet” (T3).

In the classroom, teachers continue to adjust the lesson plan to support student learning based on what is currently needed in the classroom. These adjustments may be based on cognitive indicators but also on behavioral struggles. Currently in the classroom, teachers may adjust the time spent on an activity depending on how students are performing. When the “lower-ability [students] need more time, [teachers] bump it up 2 minutes and when the higher ability fly through, [teachers] bump it down 2 minutes” (T5). Additionally, teachers believed “one of the biggest obstacles to collaboration is behavior” (T1), such as when students are socializing instead of working. Even when collaboration has been planned, teachers sometimes realize “this is just not the day for it” (T7). The teachers believed that there were “times you are going to need to have that override button” (T7) in order to adjust to what is currently occurring in the classroom and have a productive learning session.

Additionally, when students are working individually or in small groups, teachers “want to not just be monitoring, but if [students] are stuck, being able to help” (T6). Across all three sessions, the teachers expressed that they “get so much more done, so much faster one-to-one while everyone else is going” (T7) and felt that working one-on-one with students who seem to be stuck is one of the best uses of their time for supporting student learning. All of the teachers expressed that they wanted to be able to “rotate and be right in the mix of it” (T2) and to have “time where [they] can support someone who has fallen behind or needs a little extra time” (T3). When the orchestration load is too high in the classroom, teachers are unable to spend this one-on-one time with the students, as all of their time is used for managing the activity.

To be able to support in-classroom orchestration actions, such as one-on-one help and adjusting to classroom needs, the teacher must be aware of what is happening in the classroom. This awareness might take place on the individual/group level, where the teacher might “walk around and in a few swipes, be able to see what the group is doing” (T6) or on the whole-class level by being “able to see everything that is happening” (T3). This awareness would not just be about the students. Rather, with a system, the teachers would “always want to know why [the system is making a decision]” (T7). Only by being aware of what is happening in the classroom can the teacher make appropriate lesson plans, change activities in a flexible manner, and provide students with help when it is needed.

How Teachers Are Willing to Invest their Time

In addition to daily orchestration actions that teachers take in the classroom to increase the efficiency of learning time, teachers also engage their students in a significant amount of up-front learning in order to set useful routines (Prieto et al. 2011b). These routines involve both collaboration and general classroom routines. Collaboration is a skill that students must be taught in order for it to be an effective learning tool in class. If “collaboration is something that everyone [at the school] comes on board with” and “students are used to it and they can do it from day one” (T2), it is much easier to integrate into the classroom. Otherwise, “it is a ton of frontloading to get them to collaborate, but once it is up and running it is truly beautiful” (T4). The teachers felt that collaboration became more viable in the classroom through “prepping about what good collaborating looks like” (T6) and “modeling how you work with a buddy” (T5).

Besides teaching collaboration skills, all of the teachers felt that classroom routines were necessary for the class to run smoothly. Once students internalize these class rhythms, they are able to take some of the orchestration load off of the teacher. For example, “if [students] have been taught originally what to do when they see computers grouped together and then they know exactly what their job is” (T6) or “after the signal, whatever we have taught, that they move into exactly what they need to do” (T3). Teachers would “ideally like [students] to be able to walk in and know where they are going,” but “some [students] are not [good with the structure] and need reminders” (T2). Although it is important to instill these routines in the students early on, their implementation often still requires periodic support.

How Teachers Do Not Want to Spend their Time

In the classroom, the teacher is responsible for many tasks outside of teaching the students. A main component of the teachers’ responsibilities lies in classroom management, where they are responsible for the smooth running of the class without behavioral interruptions. Although the teachers stated that they used routines to instill class rhythms, the teachers in all three sessions felt that they still spend a significant amount of time on daily management. They felt that “so much of a good teacher at this time is just their management” and “if [they] could just focus on the teaching, that would be incredibly helpful” (T4). Management not only takes up a great deal of their time, but it is often not at the forefront of their mind when they are helping students. During class, “once [they] get into the thick of it, [they’re] not going to have time” and often “forget…because [they] are doing 500 other things” (T1). The teachers all wanted to focus on the content learning and felt that “if there is anything that can be done in the classroom for the management piece that would be amazing” (T3).

Part of teachers’ management responsibility lies in communicating information to the students. When the students begin an activity, “it takes a lot of time telling them here’s who you are going to be with, here is where you are going to work, and here is what you are going to do” (T1). This takes the teacher’s time away from helping the students who are behind. The teachers in all of the sessions believed that “it might be nice to have a system [prompt the students]” (T6) so that they could focus on the teaching. The desire for direct communication to the students applied not only to the directions on the activity but also the reasoning behind the activity. Thus, to help motivate students’ learning, the teachers believed that the system could communicate the reasoning behind its choices because “if [the students] understood that there was a reason, [the teachers thought] that they would function better that way” (T7).

The time spent communicating instructions to students was not only restricted to the beginning of an activity, but rather continued throughout the activity. The teachers felt that “whenever it is collaborative, [they] are constantly [getting interrupted]” during their one-on-one time (T7). The students often come over to ask what they are supposed to do next or for additional instructions. The teachers try to “think of every problem and misconception that [the students] will have so that [the students] do not come and interrupt [their] group time” (T5), enabling them to address these issues in advance. The teachers in every session felt that “the time is precious that we get with our small-group kids” and they expressed a desire for the students to obtain the required information from a different source before asking the teacher (T5). The teachers emphasized that the wish to avoid interruption applied not only to the students but also to the system, as they “don’t want to have to go back and touch something or press something. That’s not going to work” (T2). Thus, with the fluid social transitions, it appears that teachers do not want to spend their time approving every transition before it can occur.

Finally, when a lesson includes collaborative learning activities, the teachers also have to coordinate student groups. In all three of the sessions, the teachers discussed their desire for flexibility in the groupings, giving the students a chance to work with different partners. However, they felt that they “end up having the same groups every time” (T1) because it is often easier. Moreover, they found that it is “sometimes, you know, it is draining to constantly be thinking about how to pair two students up” (T2). To be able to focus on content rather than groupings, they felt that “if the technology…helps group them into [groups]…that would be a huge time saver” (T3), although they would still wish to check the groups since “it might be necessary to…change something. Like [some students] just cannot be near each other” (T1).

For What Teachers Do Not Have Sufficient Time

Although the teachers want to be aware of what is happening in the classroom, they do not feel like they can always monitor everything that occurs. When the teachers are working with individual students, “there really isn’t a way to monitor [the other students]” (T1). Often, the teachers will “walk and check in” (T4), but “when [they] are in the middle of it, [they] do not always notice and [the students] have now just been sitting there for three minutes” (T4). Across all sessions, the teachers stated that they found it difficult to monitor the students and to “to be aware of everything that is happening around [them] and being on point” (T2).

Furthermore, even if the teacher is able to visit every student/group, there is information that is not visible to them or that they cannot glean quickly. When the teacher visits a student or group of students, he/she only sees a snapshot of how the students are currently working. However, “if the system can be tracking behaviors” (T1), it can help to foster the teachers’ awareness of their classroom and of how to support students. Moreover, in all three of the sessions, the teachers discussed their limitations and believed that “a system would be able to do the stuff, the fine details that we miss, that we do not know that we have missed” (T2) and “could eliminate human/teacher error” (T4). To increase the teachers’ classroom awareness, they expressed the need for more support in monitoring than they can currently achieve on their own.

Study 1 Discussion and Design Desires

As we are researching classroom orchestration, many of the topics in our results are around established themes of classroom orchestration including planning, monitoring, and flexibility (Dillenbourg and Jermann 2010; Prieto et al. 2011a). However, rather than focusing on these overall themes, we focused on the role that the teacher would play in these areas of classroom orchestration, and whether they do and do not want to spend their time as it pertains to social transitions. As with other research on co-orchestration, we found that teachers wanted to focus their time on teaching and on supporting the students with their learning (Holstein et al. 2017). However, what this means for an orchestration system that supports fluid social transitions, which teachers considered to be a form of classroom management, compared to the orchestration of help-giving, is very different. Rather than co-orchestrating actions within a learning activity, we focus on co-orchestrating fluid social transitions, which need a different division of shared control. We discuss where teachers do not need complete control over the system for social transitions. Our initial design desires are summarized in Table 2. Specifically, we found that teachers were willing to relinquish control when the system is still within the bounds of what the teacher has planned, gives the teacher more time to work with students, or when the teacher feels that the system can do the task better. If more of the real-time orchestration decisions are shared with the orchestration system, teachers can have greater awareness of and react faster to student actions, and allow students to follow their own paths through the activity, within the scope of the planned lesson. At the same time, this shared decision making can ease the bottleneck that depends on the teacher’s time in the classroom. From the findings, we formulated six design desires for co-orchestrating fluid social transitions. These desires begin to explore the area of orchestrating fluid transitions that extend the current literature (Dillenbourg 2013; Dillenbourg and Jermann 2010; Prieto et al. 2011a) by focusing on balancing the roles played by the teacher and the system in the orchestration process.

Table 2 Summary of the design desires related to shared control for the classroom orchestration of social transitions. Desires 4 and 5 were updated after Study 2 to reflect the findings of Study 2, which are in italics

The first two desires share the orchestration control based on planning and execution of a lesson. Even before the class begins, teachers are able to influence their students’ learning based on how they design the lesson. To support teachers in terms of the value they place on teaching the students, we recommend that in the orchestration process for social transitions, 1) teachers are the instigators of the lesson-designing process. Through lesson design, teachers get to play a significant role in their students’ content learning, which emerged as an important theme in the sessions. When planning a lesson, teachers are able to take the time to make informed decisions regarding the session and social transitions in a way that aligns with their class routines and learning goals. By specifying when and how students will transition between social levels during the activity in terms of the activity impact, group size direction, and fluidity of the transitions, the teacher can then share the orchestration load of carrying out this plan with the system while still retaining control over the parameters within which it is working. This brings us to our next design desire, that 2) the system is responsible for carrying out real-time orchestration decisions around social transitions within the confines of the teacher’s lesson design. These orchestration decisions can include anything from pairing students, which can occur either during the class or during the planning depending on the lesson specifications, to managing the timing of the social transitions. This desire is in contrast to many orchestration system designs in which the support remains at the mirroring, alerting, or advising level (Holstein et al. 2018b; Molenaar and Knoop-van Campen 2018; Soller et al. 2005; van Leeuwen et al. 2019) leaving it up to the teacher to decide on, and enact, orchestration actions. By automating the orchestration of real-time social transition decisions that adhere to guidelines within the planned lesson, teachers can spend more time supporting their students’ content learning. This means that the teacher’s one-on-one time with students/groups is not interrupted every time a decision needs to be made and the teacher can be confident that the other students are progressing. When students are transitioning asynchronously, the system can share the orchestration load, and teachers, therefore, do not have to spend all of their time supporting this activity, which is not meaningful to them.

However, during class, there are often times when something unexpected arises and the lesson needs to be changed to adjust to this external constraint. The teachers stated that they took such actions every day, and emphasized that the ability to make decisions regarding these deviations from the lesson design allowed them to maintain control. Accordingly, as our third design desire, 3) teachers should have the ability to make any real-time changes to the current learning design, as the ability to flexibly adapt the lesson during the class gives the teacher a sense of autonomy and control in the classroom to support student learning. Although this desire is not new to orchestration literature – in fact, a main component of classroom orchestration is supporting flexibility within the classroom – past systems have not had as much control over the adaptation in real time as in our second desire. Therefore, we find it important to re-iterate that teachers must have flexibility with social transitions, even if their control is reduced in other areas. These changes can refer to changing their own lesson design or overriding an action that the system has taken. To be able to make these decisions, the teacher must have an awareness of the current state of the classroom. However, the teachers in the sessions stated that they cannot always have such awareness when working one-on-one with students. Therefore, the system can provide support for the orchestration process by 4) providing monitoring of the students, which 5) the teachers can use for classroom awareness. These two design desires are often the common division in orchestration systems between the system and teacher. When the students are interacting with technology, it is often able to track the students on a more fine-grained level than teachers feel they can achieve in real time. By monitoring student actions, the system can provide the teacher with information to support his/her awareness. However, in a co-orchestration system, the system not only has to provide the teacher with monitoring of the student’s actions, but needs to provide the teacher with awareness of the system’s actions and why it has made a particular orchestration move. Thus, system support gives the teachers more context about what they are observing in the classroom while they are mainly focusing on one-on-one interactions.

Finally, we recommend that 6) the system is responsible for the communication of information to the students. When teachers are the primary source of information, they are constantly interrupted during class and cannot focus their time on supporting student content learning. The system could directly communicate to the students the directions and explanations for social transitions. This direct communication from the system would allow for more fluid transitions to occur, as students would not always be waiting for teacher instructions. The students would then have more responsibility to take charge of their own learning, as they are not as reliant on the teacher.

Study 2

To further develop and assess our design desires, we conducted a series of out-of-class sessions with teachers to understand how the co-orchestration design principles for social transitions aligned with their values and classroom practices two months after the initial study. By realizing the design desires from the design sessions in a physical artifact, namely a mid-fidelity prototype of an orchestration system for specific social transition scenarios, teachers can further reflect on how our findings align with their values in the classroom. The prototype also leaves less ambiguity by providing a sense of interacting with the system. This process allows any further discrepancies in understanding between teachers and researchers to be addressed with respect to how responsibilities should be divided in a co-orchestration system (Martinez-Maldonado et al. 2015b).

Even with our design principles guiding the design of the system, it may not align with teachers’ values if the system does not provide sufficient support to avoid teachers feeling overwhelmed. On the other hand, the system may not provide the teachers with sufficient freedom, leading them to feel frustrated about a loss of control over their classrooms. Of course, the design does not need to be ‘either or’. As the goal is co-orchestration, there may be some areas where the system does not provide sufficient support and others where it does not provide sufficient freedom. To determine where this balance is well aligned and where it is not, in the following section, we present the findings from the scenario review sessions with the teachers, as they reflect on the support around social transitions through a series of classroom social transition scenarios.

Methods

Participants

In the second part of our design process, we continued to work with the majority of the teachers who participated in Study 1. A total of seven teachers (six females, one male) participated in this process. Six of the teachers had previously participated in Study 1, while one teacher had not.

Materials

Based on the findings from Study 1, we developed an orchestration system prototype that focused on demonstrating the co-orchestration of a range of fluid social transition scenarios when students are working on personal computing devices. The prototype supported a range of scenarios with limited interactivity and no backend. This design allowed us to have the teacher still interact with the system but for a predefined set of interactions. Through discussions with teachers during Study 1 around how they would interact with the system and their current practices, we assumed that all teachers would also have access to a personal device, such as a tablet or laptop that they would have with them in the classroom. As the students are all working on individual computers, it is assumed that the orchestration system has access to all data about student-system interactions and system state, aligning with standard log data (Koedinger et al. 2010), although not to interactions among students (e.g., analytics extracted from dialogue), or actions of students outside the system.

The scenarios in this system were developed to demonstrate the co-orchestration principles that were derived from Study 1 across a range of social transition types. Although each scenario was narrow in scope, each represented a broad type of transition that may occur in the classroom. The main elements of the prototype focused on the activity planning and the real-time orchestration of a technology-enhanced classroom that spans multiple social planes. The prototype tasks included one for activity planning and four scenarios regarding real-time planned and unplanned social transitions that teachers may encounter in their class, including fixed transitions with absent students (S1) or late/early dismissed students (S2), fluid social transitions between activities (S3), and fluid social transitions within activities (S4). These scenarios were designed to cover the range of transitions as outlined in Table 1. As with Study 1, we did not cover rotating groups as they were out of scope for our research question. Additionally, we did not include rotating activities or asynchronous between activity transitions as neither provided a new type of transition that was not supported in another scenario. Finally, we did not include a scenario with a student returning from an absence as the teachers expressed during Study 1 that they would want to work one-on-one with those students to bring them up to speed rather than having them jump into the on-going lesson. Below we describe each of the prototype activities and how they were developed to align with our six desires.

In the prototype, we first support the planning of the lesson. From Study 1, the first design desire was to support the teacher in designing the lesson (desire 1) while the system provides support with student pairing (desire 2). In designing a session for his/her students, a teacher creates a macro-script that comprises the activities on which students work, along with the social level of the activity, when and how groups are made, and the transitions between activities (transitions to next activity). Figure 1 shows an example of these choices. The different elements can be combined to create unique designs that allow for a range of social transitions. In the planning task, the system provides a template. Teachers can fill the above elements into this template in order to plan their lesson, including synchronous, time-based transitions as well as fluid (i.e., asynchronous) transitions between and within activities. For these within-activity transitions to occur, the system would monitor the students’ actions (desire 4) and recommend to the students when it may be beneficial for them to transition social levels while still working on the same activity. Additionally, through automatic grouping from the system (with teacher checks; desire 3), students can be quickly paired, which allows students to work with a variety of partners but still gives the teacher a chance to review and approve the group assignments.

Fig. 1

Example of a teacher’s lesson on paper with no defined timing of transitions and subsequent lesson design with the system and choosing adaptive transitions

In addition to supporting the planning of lessons with social transitions, the system provides real-time support for when these plans are put into action in the classroom. Scenarios S1 and S2 involve timed transitions, in which all students switch activities at the same time (see Fig. 2). Although timed transitions are not fluid, they are used most commonly in the classroom, and teachers could benefit from support in making these transitions well-timed and in making adjustments when they are not well-timed (Campbell and Skinner 2004). In the scenarios, the prototype system shows the teacher the amount of time that has passed, the students’ progress, and the activity in which students are currently engaged. Thus, it addresses the need for system monitoring (desire 4), which provides the teacher with awareness of the state of the classroom (desire 5). The system also provides orchestration tools that allow the teacher to flexibly adjust the activity (e.g., adding/decreasing time, pausing an activity, moving students between activities and groups; desire 3). Without teacher intervention, when the time remaining for an activity reaches zero, the system automatically moves students to the next activity (desire 2), directly informs the students about this change (desire 6), and makes the teacher aware of the change through the interface (desire 4/5). This information can all be seen in the top bars of the scenarios shown in Fig. 2.

Fig. 2

Examples of S1 (left) and S2 (right) where the teacher can monitor unplanned transitions and planned synchronous transitions. On the left students are able to mark their partner as absent and they are automatically given a new pair. On the right, as students who arrive late come to class, they are automatically put in a group and the teacher can see how long someone has been waiting

The student absenteeism scenario (S1) illustrates the occurrence of pragmatic transitions in class (see Fig. 2, left). In this scenario, as students sign into the system, they can indicate if their partner is absent that day. If both members of the partnership are present, the system automatically prompts them to begin working (desires 2 and 6). If a student’s partner is absent, the system automatically finds a new partner for the student (desire 2) and notifies the teacher (desire 5). The teacher is able to still change the groups if needed (desire 3). This orchestration support allows the teacher to focus on supporting students at the beginning of class rather than having to figure out who is present. Additionally, this way of making changes to social groups also applies to students who are late or who are dismissed early, which is the focus of S2. In S2, the students are working on the activity, and three students are missing from class (see Fig. 2, right). Students with missing partners begin to work on the activity with another partner, and as the missing students arrive in class, they are paired with their original partner and automatically begin to work (desire 2). When a student leaves class early, the partner is automatically moved to a new partner for the rest of the activity (desire 2), with the system showing the pairing to the teacher in both instances and the teacher being able to change pairs if needed (desires 3 and 5).

In the between-activity fluid social transition scenario (S3), students work on an individual activity followed by a collaborative activity with a partner who was assigned during planning (see Fig. 3, left). In this scenario, as students finish their individual activity, they are automatically moved to the next activity and given instructions (desires 2 and 6) when their partner is ready as well. If a student’s partner is not yet ready, the student is asked to wait, as the system can monitor the progress of both students (desire 4). The system makes the teacher aware of the progress of the students (desire 4/5), and if a student is taking too long to finish, the teacher can manually advance the waiting partner or ask the student who is still working to move on (desire 3). If, during planning, the teacher selected to have pairing happen just-in-time for students, students will be teamed up and advanced to the collaborative activity as soon as another student is ready (desire 2).

Fig. 3

Examples of S3 (left) and S4 (right) where the teacher can monitor fluid social transitions. On the left, students are automatically moved to the next activity when their partner is ready. On the right, students may get paired or divided within an activity by the system if it would be beneficial for their learning

Within the fluid within-activity transition scenario (S4), the system can use information about the student state (desire 4) to automatically have students switch social levels within a activity if this transition would be beneficial (desire 2; see Fig. 3, right). Recognizing when transitions might be beneficial is an open area for research. In S4, students who are struggling with an individual activity are paired with one another to complete the activity (desire 2), the students are directly informed of this change (desire 6); the teacher is notified and can intervene, as needed (desires 3 and 5). While working with a partner on a new activity, students who are not being productive in their pairs are separated to work individually by the system. Although the question of how social levels should be adapted remains open, this scenario was intended to demonstrate the division of roles with a co-orchestration system and how it could support this process.

Design and Procedures

Each scenario review session was done individually with each teacher. A session lasted for one hour and was semi-structured to give teachers time to interact with the five tasks described in the previous section. The first half of the scenario review sessions focused on the lesson planning, in order to test the orchestration support and to understand how well the support aligned with how teachers currently think about lesson planning. We first instructed the teachers to plan an activity on paper, as they would for their regular class in which the students are working individually for part of the activity and collaboratively for part of the activity. We then asked the teachers to take their planned activity and design the same activity in the orchestration system. This allowed us to see how an activity that was not designed within the constraints of the system could be implemented with the planning support and how the range of social transition options impacted the design. During the planning process, the teachers were asked to think aloud, and we subsequently conducted a short interview with the teachers to prompt a discussion regarding the shared control.

For the second half of the sessions, the researcher walked the teachers through the four real-time scenarios, with the ability to pause after each action, such as a student completing an activity or students getting paired, to allow for an ad hoc discussion to occur regarding the co-orchestration support. The teachers were informed that each of the scenarios assumed that the students would be working on their own device, allowing the student-system interactions to be tracked, by the software, during the activity. The researcher introduced each of the scenarios and, as actions happened in real time, pointed out and explained the changes. At the end of all four scenarios, the researcher conducted a short interview with the teachers about the real-time co-orchestration support for social transitions. During the interview, the researcher prompted the teachers to express what they found surprising or confusing and to explore their understanding and feelings of control.

Data Analysis

Screen captures with audio data were collected from each of the sessions. As with Study 1, we used thematic analysis to analyze the results (Aronson 1995; Fereday and Muir-Cochrane 2006). We again followed an iterative process using affinity diagramming (Miles et al. 2013), which involves the grouping and re-grouping of individual pieces of data to find common themes (Aronson 1995). The data that were used for grouping in the thematic analysis were extracted by a researcher who reviewed the screen capture videos and transcribed on-topic dialogue into separate notes. Each note represented a complete thought expressed by a participant during the scenario review sessions. Across the seven sessions, we gathered a total of 627 notes. These notes were compiled into overarching themes, which were then reviewed by a team of three researchers and restructured as needed.

Findings

In Study 2, the co-orchestration scenarios of social transitions provided the teachers with an opportunity to reflect on how they may share the orchestration responsibility with a system regarding concrete actions and situations involving social transitions. Although each of the scenarios focused on the social transitions, there was a continued theme from Study 1, with the teachers focusing on how these scenarios would support student learning and how the orchestration support would allow them to spend more time supporting students. Through the use of the orchestration system, the teachers felt that they would “be able to teach more and to teach more in depth” (T2) when some of the orchestration load was shared. Additionally, they “had a better idea of how to use [their] time” and “would be able to target the kids that [they] need to see” (T4). Below we discuss the balance of the system roles in terms of the primary orchestration elements within the literature (Prieto et al. 2011b) and if this balance aligned with teacher expectations.

Lesson Planning

From Study 1, we developed the desires that in co-orchestration systems, the teacher should be responsible for the lesson planning in order to retain initial control over the student learning (desire 1). In the lesson design task, as mentioned, the teachers were provided with a template for their lesson design, along with different features for the options. All of the teachers expressed that they “did not think that [any option] was missing” (T3). The teachers “like[d] having the flexibility” to plan an activity based on what would work for their class (T4). Additionally, when first engaging with the real-time scenarios, all of the teachers questioned where the lesson settings had come from and indicated that these were decisions they would want to have already set during the lesson planning, emphasizing the importance that teachers place on planning.

Besides the teachers’ view that the orchestration system did not encroach on their planning control, six of the seven teachers indicated that the template from the system actually made them reflect more on their planning. The teachers liked the system “because it broke it down into chunks and steers [them] towards thinking about group/individual” (T7) and “gives you different ideas of what to group the students by and then how to transition” (T6). This support provided the teachers with more ideas about what might work for their class. In particular, the teachers found the idea of the transitions novel, commenting that they “don’t typically ever transition with something like that” (T2) and “had not thought of transitioning in that way” (T3) but thought that it was something “interesting” and “new” to think about. This broadening of ideas came about when the teachers transferred their paper planning into the system; after seeing the options in the system, they changed their planning of transitions (see Fig. 1).

An aspect of planning that did not surface until the reflection on the fluid social transition scenario, S3, was students’ waiting time between activities when they are working at their own pace. Four of the seven teachers asked “when [the students] are waiting for their partner, is there anything that they are doing?” (T7) and stated that they were “not necessarily a fan of them just waiting” (T5). All four of these teachers suggested that this is something they could plan for by having a task in place that “if it is right there for [the students], they could just click over and it is not disruptive to the kids around them” (T4). Additionally, there were some automatic decisions made by the system that the teachers wanted to guide during planning, which we address in the following section. These requests for the ability to make additional decisions during the planning so that the system can carry them out in the classroom, support the idea that if teachers have a larger orchestration role during the planning, they are able to share more of the role of real-time orchestration.

Teacher Control and Automatic Decisions

Based on Study 1, we recommended that the co-orchestration responsibilities be split, with the system taking real-time orchestration action within the bounds of the lesson design (desire 2), enabling teachers to focus on student learning. In other words, the system informs the teacher of a decision but does not wait for them to approve it before taking action. At the same time, the teacher is able to take any orchestration action if necessary, to adapt the lesson design or revert a system decision (desire 3). In the scenario review sessions, all of the teachers expressed pleasure that the system kept the class moving forward, with one teacher stating that the fluid social transitions “is ideal in an ideal situation” and it “would be fantastic if [they] were able to do [it]” (T2). The system was able to “make [the] transitions a little more seamless, a little more quick” (T4). In terms of groups, the teachers liked the fact that they were no longer the bottleneck. One teacher commented that when a student is absent, “you do not have to deal with the ‘Wait, your partner is not here. Well, let’s wait and see.’ It is just like ‘bang’, there is a partner” (T1). The same sentiment was expressed for when a student leaves early, in which case “[the students] are not like, ‘[teacher], what do I do?’ They are just able to go and join a group” (T2). The teachers felt that “if anything, [they thought] that it would run better than if [they were] doing it on [their] own” (T6), because the system would allow students to work at their own pace and get the support they need, which teachers cannot provide when they are also having to manage the transitions.

In the scenario review sessions, the teachers also focused on how the system support for real-time decisions allowed them to focus on other aspects of teaching. The student pairing support from the system helped to “take away [teacher] bias” and allowed teachers to not “waste any brain power on reassigning [students]” (T3). In addition to reducing the cognitive aspects for the teachers, they saw the support as “huge time savers in pairing students and transitions and grouping methods” (T2) and stated that it would “take some time off of [their] plates” (T5). When the system took over “those little bits and pieces of [the teacher] managing it” (T3), the teachers could instead “focus on the misconceptions” (T1) to support student learning.

However, there were moments during the real-time scenarios when the teachers were concerned about a decision that was made. In these instances, the teachers were curious about what actions they could take in terms of orchestration to address such a decision. For example, when the students had more choice, such as marking other students as absent or setting their own pace, the teachers were concerned that the students may just be clicking without any intent and expressed that they “would want to be able to override that” (T2) in order to undo an action or move students back to complete an activity. The system provided the teachers with the ability to take orchestration actions, and they felt they “could pretty much do whatever [they] wanted” (T4) once the action was shown to them. Partially due to the ease of this control, where the teachers “did not have to type in and justify why,” a level of trust was built with the system, insofar as the teachers felt “it trusts [them] and [they] are in charge” (T3) even though the running of the lesson was being implemented by the system.

Nevertheless, there were still instances of automatic fluid transitions in which the teachers felt they did not have enough say. When adapting the use of social level to student needs, there are many metrics that could be adapted to (e.g., motivation, strategies, prior knowledge, metacognition), and when forming new groups, there are many ways to get the new group started when the students may be in different places in the activity. Five of the seven teachers explicitly mentioned they would want to “choose how the students are going to be adapted to” and paired (T5). This choice is something that they would wish to have during the planning while the system still automates in real time. Such a division would provide the teachers with some control over the automation while not having to carry any of the orchestration load during the lesson.

Awareness and Monitoring

Within the orchestration system, for the system to support automatic transitions, it needs to monitor and understand the state of the students (desire 4). In addition to all seven of the teachers finding this general monitoring “so cool” (T5), over half of the teachers also commented on how it supplemented the monitoring that they could undertake in the classroom on their own. In the classroom, when teachers are working with students, they “sometimes struggle with knowing when a group is ready to go on” (T7). The teachers felt that they “could walk around but it is faster to just have it on the screen” (T6), allowing teachers to spend more time with the students. Additionally, through the system monitoring, “there is…an expectation and there is a paper trail or data left behind so there is some accountability” (T4). This accountability can prevent the students from “hiding” (T4), which the teachers cannot prevent on their own.

Through the system monitoring and displaying information to the teacher, the teachers also expressed having a greater awareness of what was happening in the classroom than they would otherwise have (desire 5). The orchestration system made time-related aspects of the learning more visible, as a teacher “could look up and realize that [a student] has been waiting three minutes” (T2). The information provided the teachers with “a quick way to look” (T3) without interrupting what they were doing. Additionally, when the system was able to make the teacher aware of why it had performed an orchestration action, “that really made [the teachers] trust it” (T3).

However, the information regarding the system’s orchestration actions was often still not sufficient, and the teachers wanted the system to be more explainable. Five of the seven teachers wanted to know “how the system would know” some information (T6). The teachers all wished for a way to understand how the system had made its decisions, enabling them to judge whether the decisions were correct or whether more support was needed. Specifically, the teachers wanted to have a better understanding and “get notified about all of the misconceptions that [the students] are having all the time” (T1). Although not related to the orchestration of social transitions, as the system relieved teachers of much of the burden of managing transitions in real time, the teachers felt they would be able to give more thought to student learning support. The teachers wanted the option to see “where [student] misconceptions were” (T2) and “what their approach was” (T7) so that they could “see who is having trouble” (T7).

Student Communication

Finally, five of the seven teachers commented on the reduced reliance on the teacher that the system provided through directly communicating with the students who their partner is, what their next activity is, and when to transition (desire 6). When the system gives directions directly to the students, “it really keeps it off of the teacher” (T3), enabling them to use their time for other things. Additionally, the teachers appreciated these direct instructions from system to student, as “that way [students] do not interrupt [the teacher]” (T2). The teachers also liked that the directions come from a different source “because a lot of [students] don’t want to just listen to [the teacher]” (T4). Thus, the system can provide another source of information for the students.

However, the teachers did not want all of the orchestration communication to come from the system. Although they appreciated the system’s support of communication regarding the social transitions, over half of the teachers expressed a desire to communicate with the students through the system and give the students small encouragements and reminders about behavior. The teachers wanted to use the system to prevent students from “get[ting] embarrassed in front of their friends and then get[ting] confrontational” (T1). In this way, the teacher could continue “working with other students” (T2) and could send “something that is quick and easy” (T5).

Study 2 Discussion

Through the scenario review sessions, we aimed to encourage teachers to reflect on the co-orchestration design desires from Study 1, in order to explore whether the divisions still aligned with their values. To help with this reflection, we presented the desires as realized in an early orchestration system prototype. The teachers reacted positively to the orchestration system. At the beginning of the sessions, the teachers were “a little hesitant, but after seeing some of the sessions in action, [they] feel like it is something that [they] could do” (T4). Once they had been through the different social transition scenarios, the teachers wished to have a co-orchestration system in their classrooms, with one teacher saying, “I hope I am alive when this happens and still teaching” (T2). By taking some of the orchestration load off of the teachers’ plates, they felt they would be able to focus more on supporting student learning.

Overall, the teachers liked the balance achieved by the teacher doing more of the upfront planning work and the system orchestrating the real-time flow with teacher oversight. This sharing of the orchestration responsibility allows for fluid transitions that would not have been possible before, without a dedicated co-orchestration system. The teachers felt that with this system, they could focus more time on helping the students, which was a primary value for them. Even where the scenarios fell short, such as in setting the adaptation metrics and having non-productive wait time for students, the teachers merely wanted to have more control during planning to specify these terms. They then felt secure in the system automatically orchestrating the lesson in real time within these parameters.

Additionally, the teachers liked the support that the system monitoring could provide in terms of being able to orchestrate the fluid social transitions and the awareness support with which it provided them. However, they did not want the system to be a black box. As they were still responsible for their classroom, they wished for greater awareness of what information the system used to make its decisions, such as when to pair students for an activity that is set to be done independently. This would enable them to comprehend the system’s decisions and ensure that they were correct for their class. To account for this need, we would extend our fifth design desire to include teacher awareness of system decision-making processes and our fourth design desire to include system explainability (Putnam and Conati 2019).

Although the early prototype may have presented some usability issues that could have influenced the teachers’ experiences and viewpoints regarding the support the system provided them, we fixed any issues that were discovered before the later sessions. Across all sessions, we supported the teachers in the use of the system and could answer any questions that they had in the moment. We could not tie any of our results to usability issues.

General Discussion

In our studies, we found that teachers want to spend as much time as possible supporting student learning in the classroom while maintaining classroom control. This finding aligns with what other research has found around the orchestration support of single activities (Holstein et al. 2017). For the orchestration of fluid social transitions, because of the focus of the orchestration actions are different from those focused-on help-giving, the division of roles supported by the system also needs to be different. When considering orchestration support for social transitions, our sample of teachers were willing to relinquish the real-time orchestration of a planned lesson as long as they had been able to set, ahead of time, the transitions for the lesson as well as the student characteristics that the system would use as the basis for adaptation, and had the awareness and power to override any decisions as needed. This finding is in contrast to previous research in which often the system plays a mirroring, alerting, or advising role. Moreover, through Study 1 and 2, we found that our sample of teachers did not want to make all of the decisions or undertake all of the communication in real time if this meant that they would have less time to spend with their students one-on-one. From these findings, we derived six desires on how to divide the responsibilities for a co-orchestration system around social transitions, with roughly the teachers having responsibility for planning and adapting the lesson and the system carrying out the orchestration of this plan.

We found that our teacher participants, after they interacted with a prototype orchestration system that embodied these desires, were open to a range of social transitions that they had not explicitly supported in their classrooms in the past, but which may have been occurring spontaneously. Initially, the teachers were hesitant about sharing the support with the system. However, after seeing the greater range of flexibility that the social transitions provided, they felt that these were ideal scenarios in the classroom that would not have been possible without co-orchestration. Four of the teachers specifically mentioned that students in their classroom work at different paces and that this system would give them the flexibility to design around this fact. Unlike most previous orchestration systems, through the use of co-orchestration that balances the orchestration load, our work supported teachers in beginning to see more complex scripts as viable in their classroom rather than only supporting current practices.

Although our work focused on the orchestration support of social transitions, we found that this support could not be completely disentangled from other types of orchestration support within the classroom. As mentioned, the teachers’ main priority was to support student learning in the classroom. Once we were able to take much of the load of orchestrating social transitions off their minds, they wished to receive support (e.g., tracking of misconceptions, student-made artifacts) that would enable the teachers themselves to provide better support for students. Our focus was not on this learning support within an activity, so we purposely left it out of our orchestration scenarios. However, a current focus within the orchestration literature is on providing such support for both individual and collaborative learning through teacher dashboards (Holstein et al. 2018a; van Leeuwen 2015). Across these studies, similar teacher needs and values have been identified but with different implications for system designs depending on the focus of the orchestration support.

The focus on teaching was also reflected in teachers’ desires to be able to send messages to the students through the system. These types of communications are already supported in some orchestration systems (Manathunga et al. 2015) and would provide teachers with a faster way of communicating a short message to students when they are engaged with helping others. In future work, by bringing these parallel lines of research work together, teachers may be able to more productively orchestrate both the overarching lesson as well as the learning support for each activity.

Despite our promising findings, our work is limited by the fact that so far we have not collected data in actual classrooms. Although we worked with teachers, we did not observe their practices in the classroom, which may provide valuable insights into teachers’ current practices and restrictions. Furthermore, we have not yet applied our desires to a system which is actually used in the classroom. Using an orchestration system in the actual chaotic environment of the classroom may reveal further design constraints or provide a slightly different perspective on the division of the co-orchestration responsibilities that do not appear outside of the classroom. Furthermore, this work focuses on the perspective of the teacher, but when considering co-orchestration, the student is also involved. It is important, therefore, to find a role balance that takes into account both the teacher and student perspective (Holstein, McLaren, & Aleven, 2019). In future work, we, therefore, wish to apply these design desires to an orchestration system that can be used in the classroom and to bring students into the design process to balance their perspectives with that of the teacher. As our work involved 2nd-7th grade teachers, it would not be appropriate to cluster all of these students together when considering how their interactions with the system would take place, even if they would support the same general principles. In this case, it would be important to consider the socio and cognitive development of the students.

Additionally, these design desires were conceived with a limited context in mind and a small sample of teachers. In this work, we specifically focused on supporting primary school teachers and focused on the co-orchestration of social transitions. However, the needs and classroom constraints of larger classrooms or older students may differ from those found in our analysis. Future work is needed to understand how our design desires may be combined or modified with design recommendations for these different classroom contexts and orchestration focuses. Our work takes an important first step in this direction by investigating the shared control within a co-orchestration system that supports social transitions from the teacher’s perspective. In respect to our sample size, although we only worked with a total of eight teachers, we found, as noted above, some commonalities with orchestration research in other areas (Dillenbourg and Jermann 2010; Holstein et al. 2018a). The presence of overlapping findings from multiple orchestration contexts – those focusing on fluid transitions and those focusing on student intervention – suggests the generalizability of the findings (Hellström 2008). Our findings address a novel context while expanding on previous orchestration research, as fluid social transitions have not been the primary focus of most orchestration research. The work contributes novel insight into what the co-orchestration, between system and teacher, of social transitions entails.

Conclusions

In this paper, we present a transition taxonomy and design desires for a system that can be used to co-orchestrate the range of transitions presented in the taxonomy, derived through design with teachers. These design desires extend characterizations and factors to be considered within orchestration systems in previous literature (Dillenbourg 2013; Dillenbourg and Jermann 2010; Prieto et al. 2011a), by addressing how these responsibilities in a co-orchestration system might be divided to support social transitions. Although the current literature provides guidelines for developing teacher-centric support, our work begins to address how to strike a good balance between the automation that the system can provide and the teachers’ autonomy over their own classrooms, which very little research has addressed. Although our focus in the present paper was on supporting the orchestration of social transitions, we believe that these desires would also provide a good starting point for other types of co-orchestration systems. Our desires were developed around our teachers’ value of spending their time supporting student content learning. The focus of the respective orchestration system and the extent to which it directly supports this value would determine how our desires could be applied.

Overall, orchestration is an important aspect for educational research to consider in the future, as it impacts the learning interventions that can be taken out of a lab setting and placed in the classroom. For lesson interventions to be effective in the classroom, it is important to not only support the individual activities in isolation but also to consider how they fit into the lesson as a whole. There is an interplay between individual and collaborative learning activities, as we have seen in the abundant use of scripts in which students work on multiple social levels (Dillenbourg 2004; Dillenbourg and Tchounikine 2007). It is important to consider how we can support students’ transitions across these social levels in the classroom. Specifically, as the use of technology in the classroom is increasing, and students can be supported in working at their own pace, we need more recommendations on how to make this orchestration manageable. Co-orchestration can be used to support the use of these more complex learning scripts in the classroom by the system or students sharing some of the orchestration load with the teacher. Once we have been able to support manageable orchestration loads for teachers in the classroom, we can begin to consider the effectiveness that these scripts have for student learning.