1 Introduction

Sam and Alex are primary school teachers who have seen a new educational robot come into the market, providing an original approach to teach math concepts to children. They think this pedagogical approach may be more engaging for children to learn math and they are curious to have them in their class. At the same time, they are thinking about the consequences: would children feel so excited about the robots that it would make their class out of control? Is it going to be a problem for them to understand what students are doing? How much should they guide students in the activity and how much should they let them discover it independently?

The use of educational robots along with other technologies, like augmented and virtual reality (Rojas-Sánchez et al., 2023), has grown significantly in recent years in schools (Chevalier et al., 2016). There are many teachers, like Sam and Alex, who are interested in having robots in their classrooms, but who are, at the same time, worried about losing control of their class. According to a study (Chevalier et al., 2016) involving more than two hundred teachers in Switzerland, the perceived utility of the robot decreased significantly after teachers used it in their classes. The researchers postulated that there must be some unforeseen difficulties that prevent teachers from adopting robots in the real classroom conditions. The accessibility of educational robots due to the advances in their hardware and software technologies (Mubin et al., 2013) on one hand and the emergence of new topics in school curriculum related to computer science and robotics, like computational thinking (Ioannou & Makridou, 2018), on the other hand, have made schools more inclined to invest on educational robots (Alam, 2022). However, as the research above suggests, teachers may have reservations about or need support to adopt the robots in their classrooms. This work is an attempt to support teachers in dealing with the difficulties in management or orchestration of their classrooms arising from the presence of robots.

Orchestrating a classroom refers to all the necessary efforts that should be made to ensure learning for students in the classroom, considering the limited class time, teachers’ attention and learning materials (Dillenbourg & Jermann, 2010; Phiri et al., 2016). Researchers on classroom orchestration have tried to understand what makes teachers feel that “a technology works well in their classroom”; i.e., what makes a class be well-orchestrated (Dillenbourg & Jermann, 2010). According to Prieto’s model, highlighting the most important aspects to characterize the orchestration process (Prieto et al., 2011), a well-orchestrated classroom requires 1) managing the class resources, like groups, time, available technical resources, etc. 2) being aware of what is happening in the classroom and within the learners’ minds 3) and accordingly adapting the lesson plan on-the-fly to the context of the classroom. We argue that supporting teachers in orchestrating classrooms, where educational robots are used as a learning tool, will make them feel more competent because the classroom will be better orchestrated and, as a consequence, teachers will feel more motivated towards integrating educational robots in their teaching practices.

Educational robots (ER) are physical learning technologies equipped with actuators, sensors, and interfaces to interact with the learner, and typically used in the context of in-person classrooms (Shahmoradi et al., 2019). They can be divided into two categories, as shown in Fig. 1: The first category are social robots which are used, for example, as a peer or as an instructor to teach foreign languages (Chang et al., 2010). The second category are table-top robots utilized in classrooms in two ways: 1) robotic platforms with programming interfaces suitable for practicing computational thinking (CT) skills in the form of open-ended, project-based activities (Eguchi, 2014; Seckel et al., 2022), and 2) handheld robotic activities (the focus of this work), that are based on physical learner-robot-interaction (Mubin et al., 2013), for teaching diverse disciplines such as mathematics (Khodr et al., 2020), physics (Özgür et al., 2017b), handwriting (Asselborn et al., 2018) and so on. Compared to other technologies used in classrooms they are considered expensive, so it is common practice for teachers to make students work with robots in groups, to accommodate for the reduced number of robots w.r.t. students (Chevalier et al., 2016).

Fig. 1
figure 1

Different categories of robots used for education: From left to right: NAO (Lopez-Caudana et al., 2022), Thymio (Chevalier et al., 2016) and Cellulo (Özgür et al., 2017b) robots used for education. Thymio and Cellulo are both educational robots, with Thymio being a tabletop platform for Computer Science (CS) and Computational Thinking (CT) skills acquisition, while Cellulo is a robot aiming to enhance and mediate the acquisition of disciplinary content via students’ interaction. NAO belongs to other category of social robots, more expensive and less used in classroom settings, used for specific learning contexts (Lopez-Caudana et al., 2022)

Full size image

Research on ER has shown that teachers are motivated to use robots in their teaching due to their rich capacities for learning, including:

  • Teachers feel that by using robots in their teaching they learn something new, or they can be professionally more efficient (Karim et al., 2015). Teachers who could adapt robots in their classrooms mostly had strong intrinsic motivation to be a competent teacher (Chevalier et al., 2016).

  • Helping learners to experience hands-on learning activities (Mataric et al., 2007) which translate the abstractness of lessons in mathematics and science to real-world problem-solving activities (Nugent et al., 2010).

  • Robots are engaging and attract the attention of all levels of learners (Riedo et al., 2012). They have been successful in increasing students’ participation and as a result could enhance learning some specific skills such as sequencing skills (Evripidou et al., 2021).

  • Robots facilitate applying certain learning theories in classrooms, like constructivism (Fosnot, 2013). Students construct their knowledge while programming the robot or solving the underlying problem (Alimisis, 2013). More importantly, robots were an inspiration for developing constructionism, a learning theory, based on constructivism that emphasizes on hands-on learning with focus on tangible outputs for learners (Alimisis & Kynigos, 2009).

Literature reviews on the usage of ER in classrooms have suggested supporting teachers in class time (Mubin et al., 2013) and particularly studying classroom orchestration in this context (Ioannou and Makridou, 2018) as necessary research directions in the field. Robotic classrooms are an interesting and different context to study for classroom orchestration, since they have specific features that could create orchestration difficulties for teachers, including:

  • An important part of the orchestration difficulties comes from the fact that robots are complex technologies for teachers compared to other technologies in classrooms. Teachers must prepare the robots before the class, for example by making sure they are charged enough. Moreover, robot failures (that can occur unpredictably due to technical issues) could lead to a waste of students’ time and even force teachers to extensively change their lesson plan (Shahmoradi et al., 2019).

  • Although having robots as publicly viewable technologies on children’s desks might help teachers’ awareness (compared to on-screen technologies), an important part of the robotic activities may still include information on screen-based technologies. Subsequently, due to the distribution of information between the screen and physical spaces, having an aggregated view of the class status could be difficult for teachers (Shahmoradi et al., 2020). Besides, it is hard for teachers to follow students’ work since students may use a variety of learning strategies, creating an unpredictable learning environment (Jormanainen, 2013).

  • As mentioned earlier in this section, robotic activities are often constructivist activities. Therefore, as required of teachers in constructivist classrooms, they should actively guide the students in making the connections between the learning activities and the underlying conceptual knowledge by creating discussion in the classroom during debriefing lectures. During such lectures teachers are required to orchestrate a feedback and discussion session to aggregate the students’ reflections on their activity, to identify new ideas from the students’ work and build on it in their instruction (Kriz, 2010; Hand et al., 1997). Here, the teachers’ challenges are, firstly, to debrief the lesson at the right moment (Schwartz & Bransford, 1998); so teachers need to have control over the sequence of the activities and be able to pause/resume students’ activities easily. Secondly, teachers need to adapt the discussion based on the students’ contributions which requires real-time data (from the students’ interactions with the robotic activity) (Do-Lenh et al., 2012).

In order to enhance TEL classroom orchestration, researchers have developed tools to assist teachers, which are called orchestration tools (Holstein et al., 2019; Amarasinghe et al., 2020) or learning dashboards (Martinez-Maldonado et al., 2014; Schwendimann et al., 2016; Gauthier et al., 2022; Prokofieva, 2021). For simplicity, in this work we refer to both of these as orchestration tools. An orchestration tool consists of several orchestration functionalities that, depending on the context, are designed to assist teachers in one or several aspects of classroom orchestration; for example, by providing students’ traces of the learning activity (Verbert et al., 2013) or making teachers able to control the activity flow (Amarasinghe et al., 2020). To design an orchestration tool for teachers, evaluating its usefulness during class time and incorporating teacher feedback to improve the design is crucial to ensure that the proposed solution effectively addresses the orchestration challenges and is adopted by teachers. The evaluation process must focus on two important aspects: usage (how much the tool has been used by stakeholders) and usefulness (its positive effects on the teaching/learning process) (Jivet et al., 2018; Bodily and Verbert, 2017). Most studies use self-reported data as the main data source for evaluation, in the form of interviews (Ruiz et al., 2016) or feedback surveys (Kim et al., 2016). However recent research has used data tracked during class time as well (Amarasinghe et al., 2020).

Within this approach, logging or observing teacher’s interactions with the orchestration tool has been the most used data source (Schwendimann et al., 2016), but researchers have also used richer data sources, such as class audio and video (Amarasinghe et al., 2020), possibly in combination with the students’ learning gain (Kim et al., 2016). These data sources have enabled researchers to explicate the benefits of orchestration tools, for example, at directing teachers’ attention to the pitfalls in individual or group activity, who needs help or which group is less collaborative (Martinez-Maldonado et al., 2014), or to improve the learning gain (Holstein, 2019).

Despite the emphasis in the literature on evaluating the usage and usefulness of orchestration tools in the context of robotic classrooms as a necessary research direction in the field (Ioannou and Makridou, 2018; Shahmoradi et al., 2020), to the best of our knowledge, such a study has not been done yet. This paper contributes to this research direction by evaluating and presenting novel insights on using classroom orchestration tools in the context of a robotic classrooms. (a classroom in which children are engaged in learning activities with table-top educational robots to learn a specific topic in math, or science) The results will contribute to the design of orchestration tools able to mitigate teachers’ challenges in robotic classrooms, thus ultimately helping to improve the orchestration of robotic classrooms and the adoption of educational robots. This work may also offer new insights on the usage and usefulness of orchestration tools in TEL classrooms in general. Specifically, we answer the following Research Questions (RQ):

  • RQ1: How do teachers use the functionalities of an orchestration tool in a robotic classroom? (Examines the usage of the functionality)

  • RQ2: What did teachers perceive as the effect of using the functionalities of the orchestration tool on the teaching process in the robotic classroom? (Examines the usefulness of the functionality)

Answering the above-mentioned research questions will help us understand if the orchestration functionalities have been effective in the context of our experiments. However, as a further step, we also need to understand the teachers’ purposes of using those functionalities, based on teachers’ feedback and their teaching behaviors. This is important since there could be a functionality that, even though used, teachers don’t find necessary to use. On the other hand, there could be a functionality which is not used due to some reason, but teachers still find useful, if modified. So, we also investigate:

  • RQ3: What was the relationship between the usage of the functionalities of the orchestration tool as identified in RQ1 and teachers’ orchestration behaviors in a lesson of a robotic classroom? Answering this question will help us to know whether the functionality can be a promising solution to teachers’ orchestration challenges in robotic classrooms in general.

The rest of the paper is organized as follows: we review the orchestration functionalities that have been proven useful in other TEL classrooms in the second section. Building on literature findings in Section 2, we design an orchestration tool, named CelloRoom, including functionalities aiming at addressing the highlighted teachers’ orchestration challenges in a series of robotic learning activities. We describe the design and implementation of the robotic learning activities and the CelloRoom in sections 3 and 4 respectively. In section 5, we explain how the experimental setup was tested in authentic school sessions. Next, we evaluate the usage, usefulness and teachers’ purposes towards the CelloRoom in the sixth section. We conclude the paper by discussing the results and finally review the findings in the conclusion.

2 Literature review

Although orchestration tools are not yet widely used in schools, there is a growing interest in designing them to support teachers in various contexts of TEL classrooms (Holstein, 2019; Amarasinghe et al., 2020; Schwendimann et al., 2016). TEL classrooms are classrooms equipped with various technological tools, such as computers, projectors, interactive whiteboards, and other digital devices, including education robots, to enhance the learning experience of students and support different teaching methods (Rojas-Sánchez et al., 2023; Lee and Choi, 2017). Therefore, in order to design an orchestration tool for robotic classrooms, it is necessary to review the common practices for designing orchestration tools in literature for TEL classrooms.

To characterize these tools, Holstein et al have proposed a framework, named Teaching Augmentation (TA) framework (An et al., 2020), which proposes a five-dimensional design space for orchestration tools: the first three dimensions describe the range of how, when and to what extent the orchestration tools are available in classrooms. For example, orchestration tools can be distributed in the classroom over students’ desks, like Lantern (Alavi et al., 2009), be visible to everyone in the class and available all the time, like TinkerBoard (Do-Lenh et al., 2012) or, on the opposite side of the spectrum, be private and available only when the teacher interacts with it, like tablet- or web-based orchestration tools (Haklev et al., 2017).

In terms of the content of the orchestration tools (the information they display or the functionalities they provide for teachers), the TA framework mentions functionalities ranging from perception (what to notice) to action (what to act upon). Teachers can perceive visualizations of traces of students’ work to help them have an aggregated view of class progress. The traces could be the time spent on the activities (Faucon et al., 2020), students’ artifacts in game-based environments (Gauthier et al., 2022), their answers to quizzes (Verbert et al., 2013), and so on. In the middle of the range, there is information aiming to help teachers make decisions, e.g., visualizations of group activities that show groups in need for help (Martinez-Maldonado et al., 2014; Voyiatzaki & Avouris, 2014) or showing students’ misuse of the learning system (Holstein et al., 2019). Finally, at the end of the range, there are functionalities aiming to help teachers take actions in the classroom, to orchestrate the activities and control the activity flow. The most common actions in orchestration tools are the ability to control the activity flow (start/stop/pause the activity), deciding when to project students’ artefacts on the board and changing the difficulty level of the activities (Amarasinghe et al., 2020).

Orchestration functionalities can be helpful based on the context. For example, pausing action has been found helpful to increase students’ reflection in the context of a tangible tabletop simulation system to support the training of vocational apprentices in logistics (Do-Lenh et al., 2012), however the same mechanism was not used by teachers in the context of collaborative activities at university level (Amarasinghe et al., 2020). As another example, visualization of students’ activity could be used to create class discussions (Beauchamp & Kennewell, 2013), especially in the context of problem-solving or constructivist activities (Verbert et al., 2014). Do-lenh (2012), in the context of a tangible tabletop simulation activity, showed that class discussion is enhanced by the presence of a student-facing dashboard (TinkerBoard) which helped teachers by providing aggregated views of all the students’ solutions, so the teacher was able to compare students’ simulations, leading them to reflect on their experience. Using TinkerBoard led to a more engaged classroom atmosphere and better learning gain, compared to the control group. The importance of learning context in the success of the above-mentioned orchestration functionalities, highlights the importance of designing orchestration tools according to the underlying context.

In the context of robotic classrooms, although novel, there is an increasing trend of research for finding orchestration functionalities indicating students’ progress. For instance, Orlando et al. (2020) have used educational data mining to calculate students’ scores in a robotic activity and created a dashboard to show this information to teachers. In similar research, Jormanainen (2013) tried to support teachers in a robotic programming activity by classifying the students’ status into four categories (indicated by colors from green to red) with decision trees classification method. These studies emphasize the importance of showing students’ progress in a robotic activity to teachers. However, none of these works has evaluated teachers’ opinions and their usage of the produced data in a classroom, which is a necessary step in designing and for ensuring adoption of orchestration tools (Holstein, 2019). To address this research gap, the aim of this work is to provide design guidelines for an orchestration tool for robotic classrooms based on teachers’ evaluation in real-time usage of the tool.

3 The robotic learning activities

The main goal of the robotic learning activities is to help students, aged 7-10, learn the notion of coordinate systems. The activities are designed in a sequence such that it allows students to discover the benefits of coordinate systems as they progress along the sequence. The activities are linked by the overarching theme of space training missions for astronauts, where each mission, i.e., single activity, is played to achieve a particular training goal, for example, catching all the stars or lost astronauts, as shown in Figs. 2 and 3. The learning activities are designed as collaborative activities involving teams of three students each (as shown in Fig. 4).

Fig. 2
figure 2

The first and second robotic mathematical learning activities, with respective task and learning objective

Full size image
Fig. 3
figure 3

The third and fourth robotic mathematical learning activities, with respective task and learning objective

Full size image
Fig. 4
figure 4

The experiment setup: students are playing in teams of three and the teacher is managing the classroom. The orchestration tool, called CelloRoom, is visible to the whole class on the beamer. The teacher is observing groups activities

Full size image

3.1 The activity setup

The activity setup, as shown in Fig. 4, consists of two parts, including the physical workspace and the tablet. These two parts are explained below:

  1. 1.

    Two students (called red and blue players) will hold one Cellulo robot each and move it on the augmented map (shown as Cellulo red and blue in Fig. 4) as the physical workspace of the activity. Cellulo robots (Özgür et al., 2017b) are hand-held, hexagon-shaped robots, capable of holonomic motion and absolute global localization (position and orientation) within an “augmented” paper sheet (we refer to it as a map), specifically designed for the activity. When placed on the map, each robot can self-localize itself with sub-mm accuracy via an image sensor placed underneath the robot. Cellulo robots have been used for several learning activities, such as chemistry (Özgür et al., 2017b) or physics (Özgür et al., 2017a).

  2. 2.

    The tablet app, held by the guider, which displays the necessary information about the activity, as shown in Fig. 5. It shows the position of the spaceship (which is updated together with the movement of Cellulo robots on the physical map). The guider (as shown in Fig. 4) will have access to additional information (like the position of the stars/astronauts to collect/save in activities 2, 3 and 4) that would enable him/her to guide the red and blue players who hold the robot.

Fig. 5
figure 5

A view of the tablet app that includes the virtual map of the third activity. The app is used by the guider to guide the teammates during the robotic activities

Full size image

In the activity design, we have used a collaborative-script to improve the collaboration by which, at any given moment, each team member has a specific role relative to other team members (Rojas et al., 2022). Students rotate the roles at the end of each turn. As mentioned above, these roles are defined according to the part of the game students interact with (Cellulos Red or Blue and the tablet).

The learning sequence is organized in four activities, described in detail below. Each activity is organized in three turns. At each turn, one child plays the role of the guider, while the two others play the role of red and blue players. Children change roles at each turn, so that they all practice different roles within each activity.

  • In the First Activity, the team is in charge of guiding spaceships to catch all the stars appearing on the virtual map, without time limit (Fig. 2 left). red and blue players independently drive one spaceship each, via their Cellulo robots, but they cannot see the position of stars on the physical map. The guider (who holds the tablet) can see the stars, but cannot move any spaceship, and should thus provide guidance to the other two. The goal of this activity is to get students familiar with the tablet app when playing the guider role and moving Cellulos when playing the red and blue players.

  • In the Second Activity, the team has the same goal of the first activity, however there is only one spaceship which is jointly controlled by the red and blue players to catch the stars. Specifically, the red Cellulo can only move the spaceship horizontally (left or right) while the other one can only move it vertically (up or down) (representing the roles of variables X-Y in a coordinate system). Instead of directly introducing the concept of two-axes, we want children to construct this knowledge on their own.

  • Third Activity follows the same structure as the second activity but introduces time constraints. This time, children have to save astronauts lost in space, and have a limited time to reach each of them. The idea is to prepare students for the fourth and final activity, where they can see the numbers on the axes and the grid on the map, as shown in Fig. 3. This new feature of the game pushes students to play the game faster and discover the best strategy to play the game, since they have only a limited amount of time to save the lost astronauts.

    Fourth and third activities involve the same task but with different space representation available to the students, and the rationale is to motivate students to think about what this new representation - the grid and numbers- brings to the task. This design allows to later on compare different representations (axes with vs without numbers) and thus make the advantages of using coordinate system emerge.

3.2 Iterative design and validation of the robotic learning activities

Before designing the orchestration tool, we needed to consider teachers’ opinions in designing the robotic learning activities and validate the robotic learning activities with teachers. To do so, we conducted an iterative design process consisting of three phases, which are explained below and summarized in Table 1.

Table 1 The iterative design process of the robotic learning activities
Full size table

In the first iteration, four teachers played an online version of our activities with their personal computers, since due to COVID-19 pandemic the physical gathering was not an option. In this session, they provided feedback about their experience. In general, they appreciated the collaborative aspects of the activities; however they raised concerns about the technical complexity of the learning environment (as a teacher said: It’s frustrating when you have to restart the game. Quick fix technology support is needed). Additionally, they focused on the practical issues of running these activities in their classroom, for example highlighting that the sounds in the game could create too much noise in classroom. This feedback motivated us to make the activities playable on table that is more suitable for classrooms. Additionally, we adjusted the sounds in the game, so that the activity sounds can be turned off in the classroom, if needed.

In the second iteration, a teacher volunteered to run the activities in his classroom to evaluate the teacher’s experience in terms of classroom usability (Dillenbourg & Jermann, 2010). In this class, nine students, aged 13-15 (and thus already familiar with the concept of coordinate systems), played the activities in groups of three, while the teacher was monitoring students by going around the classroom. After the session, students mentioned the need for more description of roles in the collaborative activity and commented that the activities are better suited for younger children. Also, the teacher mentioned the importance of the game to be readable and easy to follow for children, so it reduces teachers’ workload to explain the game environment for students. Based on this feedback and feedback from students, we added more visuals and instructions on the screens to define the roles while playing the game, as shown in Fig. 5.

In the last iteration of the design process, a teacher managed an online session with our activities including three students to teach coordinates system. Although students practiced all the activities successfully, according to the teacher, the fear of losing time forced her to use direct instruction rather than letting students discover the underlying concept on their own. We realized this could be incoherent with the nature of the robotic activities, aiming to promote a constructivist learning paradigm. After reviewing the literature, we realized that showing students’ scores in the learning activity can be a promising way for increasing students’ motivation for discussion (Do-Lenh et al., 2012) which is a necessary element of constructivist classrooms. Therefore, we tried to enhance the discussion in the classroom by adding students’ game scores in the orchestration tool (Fig. 7) which will be further explained in the next section.

4 CelloRoom: An orchestration tool for robotic classrooms with cellulo robots

The orchestration tool, called CelloRoom, is provided to teachers in the format of a web application. CelloRoom consists of four orchestration functionalities in two interface pages, which are shown in Figs. 6 and 7. Teachers can switch between these pages from the left-side menu. Below we explain each of the orchestration functionalities and the rationale for including. It is important to notice that each functionality is designed in a way that clearly addresses one or more of the teachers’ needs in robotic classroom, according to the literature (Amarasinghe et al., 2020, Do-Lenh et al., 2012, Shahmoradi et al. 2019, 2020).

  • Progress bar: In orchestration tools, the necessary information to support teachers in monitoring the students’ progress is called progress-related data (Wang et al., 2018), that are calculated based on the students’ pace in the activity, for example the number of completed steps in the activity (Schwendimann et al., 2016) or more advanced indicators like students’ status (Holstein, 2019) and students’ progress predictors (Faucon et al., 2020). In a robotic classroom, due to the diversity of students’ learning strategies and teachers’ limited attention, they need these data to have an overview of what is going on in the class (Shahmoradi et al., 2020). In our design, a tabular progress bar shows the number of completed steps for each group across the four activities and each turn. Each square in the table shows the group progress (Fig. 6). There are three possible colors that a square can take: white denotes turns that students have yet to reach, blue means in-progress and a completed turn is shown in green. The tabular design is one of the most-used visualization types (Schwendimann et al., 2016) that allows teachers to see in one glance the progress of all students. Due to its importance, progress-related data is shown on the first page of CelloRoom.

  • Robot Failure Indicator: Shahmoradi et al. (2020) mentioned the fear of robot failures as one of the most important teachers’ orchestration difficulties in a robotic classroom. Also, one of the teachers in our iterative design emphasized this issue. To address it, we embedded a feature such that if one of the groups’ robots turns off due to a technical problem, CelloRoom highlights that group with a red color on both pages to alert the teacher (for example the way group 5 is indicated in Fig. 6). Similar to MTDashboard (Martinez-Maldonado et al., 2014), in which the groups at need for help are recognizable by the dashboard, this functionality can potentially help teachers to identify groups with technical problems easier and faster.

  • Management of the activity sequence (class/group level): According to Amarasinghe (2020), being able to control the activity sequence is one of the basic functionalities for a teacher orchestration tool. The ability to control the activity sequence at both group and class levels gives teachers different possibilities to manage the class: while some teachers might prefer that all groups progress through activities at a similar pace, others might allow advanced groups to go on their own (Dillenbourg & Jermann, 2010). In our design, students can only advance through the activities when the teacher allows it, while moving from one turn to the next within the same activity can be done by each group on their tablet. This design puts more control in the hands of teachers. It is interesting to see how teachers would use this control. Due to the importance of these functionalities, all the activity control actions are provided in the first (main) page of the CelloRoom, as shown in Fig. 6.

  • Artefact-related data: As mentioned in Section 3.3, during the iterative design process we realized the need for assisting teachers in creating discussions over the activities. Visualizing the scores of each student/group on their learning activity could help teachers’ need of having information on the robotic activity for reflection (Do-Lenh et al., 2012). Game-related data about students’ activity is thus shown on the second page of CelloRoom (as shown in Fig. 7). In the first and second activities, the number of collected stars and in the third and fourth ones the number of astronauts per minute are shown as indicators of students’ scores. In a robotic classroom, this feature is foreseen to be used for two possible scenarios: 1) Increasing students’ engagement by comparing groups’ scores: it is possible for teachers to make a ranking to increase students’ motivation by fostering competition between groups. 2) Relating students’ activities to the learning goal by comparing over activities: this can be beneficial a way to link the learning activity to its learning goal, for example in the fourth activity, students use the map with grid and numbers while in the third activity, only the grid is shown (as shown in Fig. 3). Teachers could compare the students’ scores over the activities to discuss whether using a coordinate system is beneficial in these activities.

Fig. 6
figure 6

The first page of the CelloRoom interface. Each orchestration functionality is described in section 4

Full size image
Fig. 7
figure 7

The second page of CelloRoom, that shows artefact-bar data of students’ activity scores. This orchestration functionality is described in Section 4. (The students’ name are not real.)

Full size image

5 Participants and data collection

5.1 Participants

Six primary school teachers from schools across Switzerland volunteered to run the learning activities and use CelloRoom in one class session each. Each class had at least 12 students and three of them had covered a learning topic related to coordinates system in their curriculum prior to the session while the rest of classes were novice to coordinates system. All the teachers and students provided informed consent to participate in the study (Table 2).

Table 2 The design of CelloRoom principles
Full size table

Approval for the study was obtained from the Human Research Ethics Committee (HREC) at Swiss Federal Institute of Technology Lausanne (EPFL), under reference number 018-2021. According to the ethical protocol, all teachers’ names were anonymized and instead fake names (Albert, Beatrice, Clara, Daniel, Emily, Fabian) are used for reference to teachers in this article. Table 3 summarizes the information about the participants. All teachers were accustomed to managing technologies in their classrooms, like presentations with beamers in their teaching and tablets in their learning activities; however, none of them had before used any form of orchestration tool for management and robots for learning in their classroom.

Table 3 The detailed information about the teachers who participated in the experiments. (Teachers’ names are assigned and not real)
Full size table

5.2 Procedure and data collection

For each teacher, the researcher first presented the activities in a mock session with a duration of an hour to an hour and half. Teachers were asked to play the activities themselves, so they would get a sense of what their students would experience later. The researcher was playing the role of the teacher for two purposes:

  1. 1.

    Teachers would get an idea of how the session could go (although they were never obliged to repeat the exact same actions the researcher did as a teacher).

  2. 2.

    They would see how CelloRoom could possibly be used in classrooms. After this, teachers were asked to familiarize themselves with the orchestration tool.

Before and after the robotic learning activity, pre-test and post-test were performed to measure the students’ learning gain to evaluate the pedagogical effectiveness of the learning activities. The pre-test contained five questions and the post-test contained six questions related to the concepts of coordinates system. Both tests were validated by teachers for clarity and coverage of concepts related to coordinates systems.

The researcher was present mainly to answer possible technical questions the teacher could have about the robots (for example, when they needed to change the robots). The teacher was allowed to ask any questions from the researcher; however the researcher was not in contact with students directly and only communicated with teachers in case they had follow-up questions.

During the session, teachers wore a microphone designed for recording lectures and interviews (Røde smartlav+ Footnote 1) to record their dialogues and interactions with students. To complement the audio recording, throughout the session the researcher observed their interactions with CelloRoom. Specifically, the researcher recorded these events: 1) if they look at or interact with CelloRoom on the beamer. 2) if one of the groups’ robot turns off.

After the session, researchers conducted semi-structured retrospective interviews with the teacher about the session, lasting around thirty minutes. The goal of the interview was to collect teachers’ perceptions about CelloRoom, as well as to know how teachers felt about the activities and the session.

To initiate the interview, teachers were asked their feeling about the recent session (e.g., Was there any moment you feel stressed? Did it go according to your plan for the session?). Then, to answer the first and second research questions, the researcher asked more detailed questions about the usage and usefulness of each CelloRoom functionality, for example Did you look at the progress bar in the dashboard? Was it helpful?

5.3 Data analysis

As mentioned above, the collected data includes teachers’ discourse in the session, their interviews in audio format, the students’ pre-test and post-test data in the form of paper sheets and the researchers’ notes in spreadsheet format. Table 4 summarizes the information about the data sources and how they have been used to answer our research questions. Researchers’ observation notes were directly used in results. The main researcher graded pre/post test data, based on the prepared rubric.

Table 4 The source of data and the corresponding analysis to answer the research questions
Full size table

For analyzing the interviews, firstly, one member of the researcher team transcribed the interview audio files to text using Otter.ai, a speech-to-text applicationFootnote 2. Teachers’ responses during the interview were segmented into 40 notes, where each note corresponds to a teacher’s response to a particular question. The main researcher categorized teachers’ interview question responses into one of the related orchestration functionalities, mentioned in Table 2 (progress bar, robot failure alert, activity sequence management and artefact-bar). Moreover, the teachers’ opinions regarding each functionality were considered as Positive or Negative: If teachers mentioned the advantage of using the functionality in the interview, it was considered as positive, for example “It [progress bar] was useful because it helped me to keep track of [students’ progress],”. On the other hand, if teachers mentioned a reason for not using the functionality or a disadvantage of using it, it was considered a negative opinion, for example “I didn’t use the [artefact-bar], since it might make kids more competitive”.

The class discourse was analyzed at two levels:

  1. 1.

    The researcher segmented the class time into the following categories:

  • Logistics: includes all teachers’ efforts to setup the activity, including the group management or technical management (setting up the robots, tablets and physical maps).

  • Pre-activity, Introduction includes all teachers’ efforts to introduce the learning activities, i.e., to explain what tasks students are supposed to do.

  • Intervention: includes all teachers’ interactions with the students while they are playing the activities, to answer questions and make sure the students play the activity as intended.

Discussion: includes all teachers’ efforts to engage the students in a reflection on their experience of playing the robotic activity and connecting to the learning goals. The discussion normally happens in a group/class level initiated by the teacher.

  1. 2.

    The classroom discourse was annotated using ELAN (a professional tool to manually annotate and transcribe audio or video recordings)Footnote 3 to delve deeper into the usage and effects of the proposed orchestration tool. Firstly, the researcher segmented and categorized teachers’ discourse (including the interaction with students) into one of the related orchestration aspects, mentioned in Table 2 (Awareness, Activity Management and Discussion).

Table 5 shows the number of extracted notes for each orchestration aspect. The notes in each aspect were clustered using affinity diagramming to summarize the qualitative patterns (Holtzblatt & Beyer, 1997). For each aspect, specific themes emerged from the data, like” Activity follow up” theme in discussion aspect. These themes help us to have a clearer reasoning about teachers’ usage of the orchestration functionalities. For example, if they do not use the CelloRoom to create discussions, we can see whether it is because they use another approach to do so or whether they don’t intend to make discussions at all.

Table 5 Teachers’ discourse analysis: emerging orchestration aspects and related number of notes
Full size table

6 Results

6.1 Sessions overview

Before answering the main research questions of the paper, we will briefly review the timeline of the six experimental sessions, which took on average 56 ± 5 minutes per session. Figure 8 shows the timeline for each session. In general, teachers started the session with an introduction to the goal of the session, the robots and the activity rules, including explaining the three roles that students would play in the activity. Albert and Diana, firstly, assigned the students to the groups and afterwards made sure each group had the necessary equipment for starting the activity while the rest of participants did these steps in a reverse order. In each session, a significant amount of time was spent on logistics issues, such as changing the activity maps in between one activity and the next, as shown in Fig. 8. For the initial technical setup, each group had to go and receive two robots and a tablet from the researcher. The maps were already placed on the groups’ tables by the teachers before starting the sessions.

Fig. 8
figure 8

The comparison of timelines of teachers’ actions in their sessions highlights similarities and differences between their activity management patterns

Full size image

Afterwards, teachers would explain the login process on the tablets and monitor each group in the process by going to their desks.

Teachers were guiding the students to advance through the activities. Firstly, as a warm-up activity, teachers would ask students to move their robots on the physical maps to discover the relation between the physical robots and the spaceship on the virtual maps on the tablet. This step was specifically designed to make students familiar with the activities’ environment and check the technical system before starting the main activities. Afterwards, teachers made students advance through the remaining activities one by one. At the end of the sessions, all teachers devoted a significant length of time to class discussions.

According to the results of a paired sample t-test on the pre- and post-test scores from the ninety-one students in our experiments, the normalized change (Marx & Cummings, 2007) was significantly positive (+21.5 percent, p-value = 0.0015, (+23.2 for fifty-one novice learners of the topic and +20.5 for forty learners who had had a lesson related to the topic before)).

6.2 How teachers orchestrated the classrooms

In this section, we analyze in detail the aspects of teachers’ orchestration behavior (Prieto et al., 2011), but only the ones in relation to the orchestration functionalities described in Table 2. This section will provide a guideline for understanding the reasons behind the usage of each functionality and its usefulness.

6.2.1 Awareness

To become aware of students’ progress, teachers mostly chose to walk around the class and directly observe each group, rather than monitoring them from afar or via the progress bar on the beamer. As reported in Table 5, teachers were directly asking students about their progress, via questions such as “what do you guys notice about the activity?” or “how many astronauts did you catch?”. They listened to the conversations between team-members, commented on the students’ positions next to each other and intervened when necessary. Another common interaction between teachers and students was related to the completed steps in the activities. While most of the time teachers were asking students their status in the activity, sometimes students themselves would volunteer to go to teachers and report the recent progress.

6.2.2 Adaptation

In general, teachers did not skip any activity, although Albert decided to make the first activity shorter to have more time for the other activities (see Fig. 8). Charles and Fabian repeated one or two activities because of different reasons: Charles repeated the activity whenever a group had to wait for other groups to finish. Fabian repeated the fourth activity, so students have an opportunity to practice their skills in the activity.

Two patterns of activity sequence management were observed:

  1. 1.

    Albert, Charles, Diana, Emily and Fabian had a synchronous class-level management where the whole class was advancing through the activities at the same time and teachers were verbally or through the orchestration tool asking students to stop the activity to initiate discussions in between one activity and the next. These teachers found it easier to manage all the teams together, as Charles says: “When every student is not on the same level, (this group is [doing an activity] and the other group is [doing another activity]. [I] have to always adapt myself with. I think it’s the best if every group [would be] in the same activity”. While in most cases, students were almost paying attention to the teacher when they were asked to stop/start the activities, in Emily and Fabian’s classes, they had difficulty capturing students’ attention, as children were continuing their interaction with the robots or the tablet, when not supposed to.

  2. 2.

    Conversely, Beatrice preferred to manage the class asynchronously, letting some groups be more advanced than others. As a result, there were no class discussion moments between the activities (See Fig. 8). In both patterns (synchronous or asynchronous), when a group has finished their activities, they would ask the teacher about the next steps. However, in the second pattern (asynchronous management), the teacher would explain the next step and advance the group.

6.2.3 Class/group discussions

Teachers were mostly conducting the discussions as a class-level activity and students were participating in the discussion. However, Albert, Diana and Fabian also encouraged group discussions. One such moment of group discussion is captured by the following conversation excerpt:

  • -Albert: why did you all guys become better? why did you improve?

  • -Group 2: because we found the numbers and we had strategies

  • -Albert: you found that there is something (a strategy [which is] more efficient).

Four types of class/group discussions, in terms of the content and teachers’ purpose emerged among the sessions:

  1. 1.

    Activity follow-up: After an activity, teachers would often ask the students

"How did it (the activity) go?" or "Did you find the activity hard?" to initiate a discussion. For example, Albert was asking students to compare the difficulty of activities. The following conversation is an example of one such discussion:

  • -Albert: Firstly, all of you are saying that this activity was easier.

  • -Student 1: I think the four [activity] was the easiest because we could say go to 5 (we could use numbers), but this one (third activity) you have to just move around and you don’t know (exactly).

  • -Albert: Did all of you hear what she said? Did having numbers make it easier to play the game? How many of you agree? How many disagree?

  1. 2.

    Reflection on students’ game strategies: one topic that arose during such discussions were about the words to use when guiding the blue and red players, the position of the guider person to better play the game, collaboration tips, etc. For example, in the following conversation, the teacher is asking students about different strategies that students used to play the second activity in our activity sequence:

Charles: How did you guide your friend?

Student 1: We used left, right, up, down.

Teacher: Ok! So, directions! Did somebody use another word?

Student 2: Diagonal.

Teacher: Yes, diagonal! Did somebody else use diagonal? How about other strategies?

Student 3: Yes, other side.

Teacher: So, we observed there are different ways to guide on space.

  1. 3.

    Showing the students’ progress: Albert and Fabian were discussing the students’ game scores to show their progress and motivate them to improve their scores using CelloRoom. For example, he would ask students whether they can make their scores better and which strategies would they use to do so.

Albert: Are you ready to see the results? Oh! Look at group number 3! You did 8 astronauts in 2 minutes. Group number two, you did 10 astronauts in 3 minutes.

Student 1: I think my group is [as good as] the other group, because we did more astronauts in more minutes, but they did less astronauts in less minutes.

Albert: Have you noticed we don’t have any group just collecting 1 or 2 astronauts? Now let’s compare the activities: This team did 7 astronauts, then [they] did 3 astronauts in 5 minutes, 4 astronauts in 2 minutes. Do you think they improved a lot? Did you all improve?

Class: Yes!!

  1. 4.

    Connection with the learning goal: Albert, Charles, Diana and Fabian were asking, after the activities, if anyone could remember a mathematics topic that relates to the activities, for example:

Fabian: Okay, some great skills in your group communication. What about the math skills? The robots may have taught us something?

Student 1: Placement.

Student 2: Position.

Fabian: Placement? position? Two very important words. Position matters. Definitely. And also, in awareness of starting in the right position.

Student 2: [Where to] Start. Position. Being accurate.

Fabian: Excellent. So today, you’ve actually been learning all about something called coordinates. So, knowing the coordinates, it make finding the position much easier for you. right?

Class: Yes!

Fabian: So the goal of this lesson was to help you understand what coordinates are and let you see how useful they are. [It’s] much easier saying the coordinates, rather than saying "a teeny weeny bit to the right". I hear "a bit to the left", "two fingers to the right" [which are] some brilliant creativity. I love them but do you think using the coordinates was easier?

Class: Yes!

6.3 Teachers’ usage of the functionalities in the orchestration tool and its usefulness

All teachers had the CelloRoom on the beamer connected to their personal computers, as shown in Fig. 4. Only Fabian used the tool also on the tablet. Below, we describe in detail the usage of each orchestration functionality, its effects on teachers’ behavior in the classroom and the reasoning behind teachers’ behavior, based on their post-session interviews.

6.3.1 The progress bar

According to the researchers’ observations, four teachers looked at the progress bar as follows: Beatrice (4 times on the beamer), Diana (2 times on the beamer), Emily (6 times on the beamer) and Fabian (he looked at the tool on his tablet while his visits of groups). On the importance of the progress bar, in the interviews, Emily said:

“It [progress bar] was useful because it helped me to keep track of [students’ progress], Should I wait for everybody to finish [their activity]? Because most of the time I had one group that was late and it [progress bar] helped me to know [which group is it], because sometimes when you’re teaching, this [finding students’ progress] is taking a really long time”.

On the other hand, the other teachers (Albert, Charles and Diana) did not mention any interest in the progress bar in the interviews. Two possible reasons for this behavior have emerged from the interviews:

  1. 1.

    Teachers felt that by only looking at CelloRoom, they will miss important aspects of students’ learning progress, as Albert said: “I prefer to visit the groups. I can see what they are doing. If I only use the [CelloRoom], I miss the conversation. I miss the body language”.

  2. 2.

    They don’t feel the progress bar would inform them of something that they could not understand by visiting groups. For example, to decide about students’ transition to next activity, Charles preferred to listen to students, as he says “They [students] were telling me [that] it’s finished. I said, so the majority is done. So, we can now go on to the next stage.”

    These reasons align with teachers’ preference of moving around the class and directly monitoring the groups one by one rather than remotely monitoring them all via CelloRoom, as mentioned in Section 6.2.1.

6.3.2 Robot failure alert

In total, there were six cases in which one of the groups’ robots had a problem that needed teachers’ intervention. Table 6 shows the distribution of the failures among the sessions. Noticed by the Alert column shows the number of times in which teachers noticed the failure by looking at the CelloRoom while Validated from the Alert column is referred to the failures in which teachers only look at the CelloRoom after noticing the failure by students to validate their claim. The rest of times teachers were noticed by students about the failure. Mostly teachers were alerted by the CelloRoom about the robot failure. For example, in one of these cases, Diana looked at CelloRoom and then she said “where is group 3, there is a problem with your robot”. Then, she asked the researcher to replace the robot with the technical problem. Fabian and Emily had the same reaction when they noticed the failure alert. In other two cases, teachers only noticed about the robot failures after the students raised the issue. Another interesting usage of the failure alert was observed when Beatrice looked at CelloRoom to validate students’ claim that their robot was not working. From teachers’ behaviors and interviews, it seems that a combination of the orchestration tool and students’ direct feedback were the sources for teachers to get aware of technical problems. Both approaches find support in the teachers’ interviews: Beatrice said “When one of the robots turned off, the whole group [bar on the tool] became red. And it was very clear, for me to know that something is wrong”. On the other hand, Diana mentioned that the information about robot failures came from students and there is no need for CelloRoom to alert that: “they (students) say it right away. They would tell me [that] this (the robot) is not working. So please fix it.”

Table 6 Teachers’ Usage of the Robot Failure Alert
Full size table

6.3.3 Activity sequence management

Teachers used this feature every time they needed to move the students between activities (to move the entire class or just a group) (as described in Section 6.2.2). Having the ability to control the sequence of each group individually empowered teachers to run the class both in synchronous or asynchronous manner, as Beatrice said: “I like the fact that I can send students to different activities individually”. Teachers preferred to have the control over students moving between the activities mainly because each activity requires an introduction, however they still like to have the option that at least some groups move on autonomously, as Albert said: “That would depend on the group dynamics. For these two groups, I don’t want to have control, so they can go on their own. With that group, I want more discussion and coming back to group number 2 and asking them why there are successful?” The other theme that emerged from teachers’ interviews was that teachers preferred to be able to control the activity sequence themselves, as Charles says: “I think the dashboard is very important, because without the explanations they would be asking too many questions. What do we need to do? And I think it would be horrible to deal with that.”

Two teachers (Emily and Fabian) used the pause/resume buttons on CelloRoom. They found this functionality helpful to get students’ attention when they are distracted by the robots as Fabian said: “If I had to get all their attention to have a little discussion between activities it could be difficult, because children at this age easily get distracted. So, with the pause button, it was really useful that I got their attention”. Other teachers grasped students’ attention by calling out to them verbally. Although in some cases teachers had to call students few times to get their attention, they did not report having a major difficulty in doing so.

6.3.4 Artefact-bar

Albert and Fabian used the artefact-bar on the beamer at the end of the fourth activity. They showed students their scores in the activities to indicate their progress; for example, Albert said: “Now let’s compare the activities. This team did 7 astronauts then did 3 astronauts in 5 minutes, 4 astronauts in 2 minutes. Do you think they improved a lot? Did you all improve?”

Students were excited to participate in discussions and were trying to provide reasons for their success/failures as a student said: “I think my group is same (as the other group) because we did more astronauts in more minutes, but they did less astronauts in less minutes.” In case of Albert, using the artefact bar led to students’ engagement and creation of a summarize-discussion moment in the classroom, while for Fabian only it was used to motivate students to improve their scores by repeating the fourth activity.

Even though only Albert and Fabian used the artefact-bar in their teaching, all teachers mentioned their interest in using the artefact-bar and said that if they were to do the same session again, they would use the artefact bar. Emily and Fabian doubted using the artefact-bar while students were playing the activities as it might get them into unhealthy competition, as Emily said: “I didn’t use the [artefact-bar], since it might make kids more competitive. I was worried that it would get them very frustrated. They would end up in some arguments instead of having a healthy competition. I was afraid that [children think] is my group doing better [than others?] and would end up in frustration.”

6.3.5 Patterns of Teachers’ usage of celloroom and their opinions about its usefulness

For an overview of the patterns of teachers’ usage of CelloRoom and their opinions about its usefulness, Table. 7 categorizes the usage and usefulness data. To do so, this table simplifies the analysis by only showing whether a teacher used the functionality (indicated as yes, Y or no, N) and their general opinion about it (indicated as positive, P or negative, N). Teachers’ usage and their opinions were found to be different among the functionalities. For instance, all teachers used the activity sequence management of the orchestration tool to move the students between activities and found this functionality useful. Similarly, we can say that all teachers viewed the robot failure alert functionality as a reliable source for getting informed about technical failures.

Table 7 The patterns of teachers’ usage and their opinions about the usefulness of each functionality in CelloRoom. In the usage columns, “Y” indicates that the functionality was used in that session and “N” indicates no usage. In the usefulness columns, “P” is an abbreviation for positive and N for negative opinion about the usefulness of the functionalities
Full size table

The awareness functionalities (artefact and progress bars) were less used by teachers, however the relation between their usage and usefulness were not the same. Despite most teachers’ interest in showing the artefacts to their students, some teachers (four out of six participants) did not use it. There seems to be a gap between teachers’ opinion about the functionality and their real usage. The least used and favored functionality was the progress bar, however teachers’ patterns of usage and opinions were found to correlate, except in the case of Diana. The two teachers who used the functionality found it useful while others did not use and favor it.

7 Discussion

Our study aimed at evaluating the usage and usefulness of the four orchestration functionalities we designed based on the teachers’ needs in the context of a robotics classroom (first and second research questions) and understand the relationship between the functionalities and teachers’ orchestration behaviors (third research question). The corresponding results are summarized in Table 8. In this section, we firstly discuss the theoretical implications of this work, i.e., how our results contribute to the literature on classroom orchestration. Secondly, we provide the orchestration implications, i.e., which actions should be considered when designing orchestration tools for robotic classrooms and lastly, we mention the limitations of these implications.

Table 8 Evaluation of the orchestration functionalities: Each column answers one of our research questions
Full size table

7.1 Theoretical Implications

7.1.1 The contextual factors affecting teachers’ usage of CelloRoom

Discussing the contextual factors affecting teachers’ usage of orchestration tools is of great importance in research on classroom orchestration (Amarasinghe et al., 2020; Holstein et al., 2019). We believe that in our study two factors played a significant role in the teachers’ usage of CelloRoom, summarized in Table 7. Below we describe the role of these two contextual factors and their connection to existing literature:

The educational level

All our samples of robotic classrooms have been recorded in primary schools, for children aged 8-10 years old, where the teacher-student relationship is closer and there are fewer students per class than in typical university settings, (Spilt et al., 2012). We believe that this matter has a great effect on the usage of CelloRoom; for example, regarding the progress bar, an alternate awareness source for teachers who did not use it was to have the information directly from students, as mentioned in Section 6.2.1.

Another point to consider is teachers’ control and authority in primary schools (Forrester, 2000). An example in our study is when students would go and ask a teacher if they can transit to the next turn in the activity (which they could do even without asking the teacher) as Beatrice said, “I would normally prefer for them to go on their own. But I saw it as a theatrical opportunity, you know, like for them to have some kind of a closure, like a full stop at the end of the sentence. So, I go and they press the button, but it’s not at all necessary!”. Similarly, teachers’ control over class engagement is necessary to continue the class activities (Bergdahl & Bond, 2022), since excessive engagement from students in primary schools can take the class out of teachers’ control which hinders the learning process. As a result, sharing the data from orchestration tools, like the data in the artefact bar, is only suitable if it does not create undesired chaos in class. Similar results were observed by using FireFlies, a distributed orchestration tool, which students only found it useful if there are no stigmatizing effects (they don’t feel they are labeled) (d’Anjou et al., 2019).

The learning technology

Robotic activities (and in general tabletop activities), unlike screen-based activities (Shahmoradi et al., 2019), provide an easy-to-monitor setting for teachers; thus paying attention and monitoring students’ desks becomes more important, therefore teachers consider checking/using CelloRoom on a tablet or computer, a sort of a barrier rather than an assistant, as Diana said: “I don’t like it when every time I have to go and click on my PC [to interact with the dashboard]”. Similar results have been obtained from teachers’ interviews in literature (Amarasinghe et al., 2020; Holstein et al., 2019).

7.1.2 The importance of using orchestration tools for class-level discussion

We emphasize the importance of further research on the usage of artefact bars in creating class discussion in literature. Among the four discussion themes that teachers created in the classroom (mentioned in Section 6.2.3), in our study, the artefact bar was only used for showing students’ progress and making them engaged with the activities. We argue that the usefulness of the artefact bar could be extended to other discussion themes. Similar usage of showing students’ artefacts for supporting reflection in the class has been successful for creating connection with the learning goal in constructivist activities (Do-Lenh et al., 2012), and it can similarly happen in robotic activities.

7.2 Orchestration tool design implications

Based on the results of this work, a number of recommendations for designing orchestration tools in the context of in-person robotic classrooms in primary schools can be drawn:

  1. 1)

    In terms of indicators for providing awareness, both in-depth indicators (such as the artefact bar in our case) and generic ones (such as the progress bar in our case) are necessary, even though the usage time seems different. In-depth indicators allow teachers to motivate students in improving their performance and thus support them in creating rich and engaging discussions after the learning activities, while generic indicators provide light information during the learning activity to make necessary decisions, like transitioning to the next activity. These results are supported by the literature finding students’ completed steps as an important indicator for learning dashboards (Schwendimann et al., 2016; Haklev et al., 2017; Faucon et al., 2020). Robot failure alerts can help teachers prioritize assisting groups in need and act as a reliable source of information concerning the status of the robots in the class. In all cases, teachers changed their order of visiting groups to quickly reach the groups with a problem. Similar orchestration behaviors of prioritizing helping groups in need, according to the data from orchestration tools, have been reported in literature for detecting collaboration deficiencies (Martinez-Maldonado et al., 2014) or students’ misuse of the learning system (Holstein et al., 2019). What makes the robot failure alert different from the alerts mentioned in literature is the fact that students themselves were a source of alert awareness for teachers, which is not the case with students’ misuse of learning systems. According to some teachers in our study, relying on students could replace the orchestration tool, but even in that scenario, the data provided by the tool can still act as a validation. As a next step, the orchestration tool should provide more information on the action that the teacher should take (for example, to change the robot).

  2. 2)

    Concerning the management of the activity flow, it is important to provide teachers with multiple options, so they can freely choose the activity sequence management approach best suited for their students. As we observed in our study and as also seen in similar works in literature (Olsen et al., 2018), teachers don’t like to have the activity sequence control all the time and, in some cases, they prefer to keep the students autonomous. Thus, they appreciate being able to send individual groups to specific activities or making students advance autonomously. According to interviews, a missing action in CelloRoom was the possibility to repeat the activity which was mentioned by two of our teachers (Charles and Fabian). This functionality was important for them to keep the groups who finish the activity faster involved until the rest of the class is ready for the next activity or to make students practice their skills.

  3. 3)

    We re-emphasize the necessity of adapting the underlying technology of the orchestration tool according to the teachers’ orchestration behaviors for taking orchestration actions and informing them about urgent class events (Alavi et al., 2009). A burden to use the tool for being aware of technical failures (or in general urgent events) comes back to the point that teachers don’t look at the tool while going around the class, as mentioned earlier in Section 7.1. Wearable tools, like smart watches, could solve this issue by notifying teachers about urgent events without requiring their attention all the time (Quintana et al., 2016).

7.3 Limitations in the Generalization of CelloRoom to Other Robotic Classrooms

Although we have tried to derive and present the results as generally as possible, two main limitations should be considered for generalizing the usage of CelloRoom to other robotic classrooms:

  1. 1)

    Regarding the implementation of the progress bar and activity sequence management, it’s worth to mention that the activity sequence was pre-determined (we implanted the sequence in the software of CelloRoom before the sessions), which might not be possible for other learning topics. Therefore, further studies are needed to design orchestration tools for robotic activities that make the teacher able to design their own robotic lesson script before running the session, as allowed e.g., by the FROG orchestration tool (Haklev et al., 2017).

  2. 2)

    Regarding the artefact bar, for every activity, the measures that help teachers create a reflective discussion about the learning goal should be carefully designed based on the students’ interaction with the robots and the learning goal. For example, in a robot programming activity the measure should be based on the relationship between the students’ program and the corresponding robot behavior (Shahmoradi et al., 2020).

8 Conclusion

This paper presents the results of the first classroom evaluation of orchestration tools in the context of robotic classrooms. Classroom orchestration tools can be designed to support teachers in dealing with the difficulties related to the management complexity of new technologies in a classroom. Contributing to the literature of orchestration tool design, this work examines teachers’ usage of an orchestration tool in a novel context. The design of the proposed orchestration tool, called CelloRoom, was inspired by an analysis of the current orchestration tools for TEL classrooms and adapted to address teachers’ needs in the context of educational robotics (Shahmoradi et al., 2020). The developed robotic activities are organized in a sequence that relies on a tangible educational robot named Cellulo (Özgür et al., 2017b). Teachers would manage a classroom where students play those activities in groups of three to learn a mathematics-related topic.

Our findings indicate that the orchestration tool gave teachers flexibility in controlling the activities sequence synchronously (class-level) or asynchronously (group-level) and provided various levels of awareness indicators that helped teachers in making activity-transition decisions and summarize-discussion moments. Moreover, it served as a reliable source for detecting hardware problems with the educational robots. While, in general, in all classes positive learning gains were observed and teachers had an overall positive perception of the way robotic activities worked in their classes, teachers had different usages and perceptions of the same functionality in our orchestration tool. For example, regarding the orchestration action of keeping the students’ attention (by pausing the activities) to create discussions, while some teachers did use traditional methods (mostly verbally), two others found the pause button helpful in getting students’ attention. An orchestration functionality (named progress bar) that is used by some teachers was not favored by the others or the same orchestration functionality (activity sequence management) was used differently by the teachers. These are examples of the complexity of generalizing orchestration functionalities over teachers, as has also been discussed in the literature (Schwendimann et al., 2016).

Nevertheless, our findings regarding the teachers’ usage of the orchestration tool can be extended to other robotic classrooms, which emphasizes the need for future research in the two following directions:

  1. 1)

    Studying the interplay between class contextual factors and teachers’ orchestration practices for the adaptation of the orchestration tool: in this paper, we only discussed the effect of contextual factors, like the educational level and the underlying learning technology, on teachers’ usage of CelloRoom. However, teachers’ orchestration practice should be considered in designing awareness and managerial functionalities. For instance, regarding the usage of the product bar for creating engaging class discussions, it is important to firstly study what are the teachers’ practices to create class discussion. We plan to do so by analyzing teachers’ classroom discourse and finding relationships with the teachers’ usage of the orchestration tool.

  2. 2)

    The developed orchestration tool (CelloRoom) along with the robotic activities can be used as a benchmark for further research on teachers’ usage of orchestration tools in robotic classrooms or their effect on students’ learning. Furthermore, it is necessary to study, in future studies, the teachers’ self-efficacy as a result of using orchestration tools. We plan to extend our research and associated evaluation methodology to higher-level concepts in mathematics, hoping to have larger numbers of teachers and children of older ages, to verify the validity of the insights presented here.

Lastly, several limitations affecting the results of the presented work are important to highlight.

  1. 1)

    The small sample size calls for more studies to be done to validate the results.

  2. 2)

    Each teacher used the CelloRoom for only one teaching session, which caused them to be overwhelmed by the cognitive load of using a new tool while managing the classroom. Repeating the activities with each teacher multiple times will lead to long-term and more reliable results.

  3. 3)

    Three of the teachers involved in this experiment had already taught the learning goal (coordinate systems) to their classroom prior to taking part in the experiment. The results might thus change if teachers use this activity as an introduction, rather than a consolidation, of the learning concept.

  4. 4)

    The researcher was present during the class to provide technical help to the teacher, upon request. Studies without the presence of a researcher should be conducted, to ensure a greater ecological validity of the findings.