1 Introduction

Teacher dashboards are a specific application of learning analytics: modern technologies that display information about students to inform teachers and to increase teachers’ awareness of the situation. Teachers’ need for such support may be especially high in classroom situations in which teachers monitor and support collaborative student learning activities. The focus of the present paper is on what role teacher dashboards should fulfill to optimally support the teacher in monitoring student collaboration.

Dashboards can take on different functions, ranging from solely displaying information to also alerting teachers to important events or advising teachers in interpreting the situation (Soller et al. 2005; Van Leeuwen and Rummel 2019). Overview studies (Sergis and Sampson 2017) suggest that teachers may be best served by dashboards that fulfill a more advanced function than solely displaying information. However, such systems may have a profound impact on the teacher’s actions, and subsequently, on teacher-student interaction and resulting student learning. In the present study, we therefore first investigate teachers’ perceptions of different scenarios of role division between teacher and dashboard through interviews using various elicitation techniques. Based on the results of the interviews, we conducted an experimental study that investigated whether different types of dashboards can aid teachers in detecting groups that may be in need of support by asking teachers to judge standardized vignette situations. The results shed light on how various dashboard types may enable or constrain teachers’ guidance of student collaboration in different classroom situations.

1.1 Collaborative learning and the potential role of teacher dashboards

Collaborative learning (CL), situations in which two or more students work together on a shared goal (Dillenbourg 1999), is increasingly recognized not only as an effective means to increase performance on a range of skills, but also as a useful and necessary skill in our current society (OECD 2017). Even at a relatively young age in primary education, children can benefit from collaboration by for example engaging in joint practice of skills (Olsen et al. 2014). In the present paper, we specifically focus on collaboration between pairs of students in the context of primary education.

Computer-supported collaborative learning (CSCL) environments (Stahl et al. 2006) are increasingly used to support collaborative activity. CSCL environments offer multiple advantages: a CSCL environment provides a shared frame of reference for students to focus on, it stimulates attention to the task, and it can provide a basic level of automated guidance. A specific example of how a CSCL environment can be employed as a means for collaboration is displayed in Fig. 1, where two students are working with the Mathtutor software (2018) on fraction assignments. Each member of the dyad can manipulate a separate part of the assignment, which creates a level of interdependency that stimulates equal participation. Employing programs such as Mathtutor have been shown to be equally effective to individual practice when it concerns students’ learning gains, with the collaborative setting even showing faster progress (Olsen et al. 2014).

Fig. 1
figure 1

Dyadic computer-supported collaboration in primary education with Mathtutor (2018)

Full size image

While such programs offer a basic support structure, the teacher remains central for structuring and orchestrating the classroom activity and for offering essential higher-order support, such as in-depth explanations of mathematical constructs (Saye and Brush 2002; Slotta et al. 2013). However, facilitating the collaboration by monitoring and supporting students’ activities, imposes a high orchestration load on teachers. There is a considerable number of dyads to monitor, which could be working on different types or levels of tasks. Furthermore, the students require support at multiple levels: not only concerning the task itself, but also with regard to the regulative and social aspects of collaboration (Van Leeuwen et al. 2013; Xing et al. 2019). As a result, losing overview is a commonly encountered obstacle for teachers (Gillies and Boyle 2010; Muñoz-Cristóbal et al. 2018). Maintaining overview during students’ collaborate activity is thus a challenge for teachers, but a necessary skill for providing effective support (Kaendler et al. 2015). By monitoring the collaborative activity and detecting groups that may need support, teachers can subsequently examine whether support is indeed necessary, and if so, select the appropriate pedagogical intervention (Webb 2009). Given the dynamic nature of the collaborative classroom, teachers are generally under pressure to keep up with all activity and have to continuously decide which group receives their attention at any given moment (Greiffenhagen 2012). The teacher’s decision about which group to interact with in the classroom has consequences for whether students display on-task or off-task behavior, not only for the group the teacher directly interacts with, but also for the groups that are in close proximity to the teacher, who can be influenced by the mere near presence of the teacher (Chiu 2004). It is therefore highly important that teachers detect in which groups they may be needed without disrupting students’ collaborative processes (Lin et al. 2015). It is this initial phase of detecting groups that may especially need support that we focus on in this paper.

CSCL environments usually track all student activity. These digital traces can be used to inform teachers by analyzing and visualizing those traces of learning in the form of teacher dashboards (Verbert et al. 2014). Similar to dashboards aimed at supporting students, teacher dashboards are technological artifacts that contain a visual display with information about some aspect of the learning process (Verbert et al. 2014). In case of CSCL, by informing teachers in real time of students’ collaborative activities, teachers could be supported in maintaining a more complete and more accurate overview of the activities each collaborating group engages in. Teacher dashboards are therefore a form of learning analytics (Van Leeuwen et al. 2022), which is a field of research concerned with studying traces of learning and the analysis of these traces to improve learning processes. In this paper, we focus on the specific role a dashboard may fulfill for supporting teachers’ monitoring of collaborative activity.

1.2 Teacher dashboards fulfilling different functions

As several reviews describe (Van Leeuwen and Rummel 2019; Sergis and Sampson 2017; Verbert et al. 2014), dashboards could aid teachers in several ways depending on the division of labor between teacher and dashboard in terms of who manages or is responsible for each task of supporting students (Soller et al. 2005). On the far left of the spectrum, dashboards merely take on the initial process of collecting and analyzing data about students, and show the resulting information to the teacher. Any interpretation and decision concerning subsequent action is then left to the teacher. This type of support is called mirroring. Dashboards may also check the students’ activities for deviations from a specified standard, and alert the teacher to those events, in which case the dashboard fulfills an alerting function. Taking the role of dashboards further, they may additionally provide advice on how the teacher should interpret the current situation, in which case the dashboard has an advising function. On the far right of the spectrum, dashboards can take responsibility for the whole process, including automatic intervention to support collaborating students, for example by sending a message to stimulate a group to ask each other more questions. Interpretation of and decision about appropriate interventions are thus performed by the dashboard, in which case the dashboard fulfills an intervening function.

The majority of research in this area has investigated teacher dashboards with a mirroring or alerting function, with mixed results (Sergis and Sampson 2017; Van Leeuwen and Rummel 2019). On the one hand, teachers find mirroring dashboards a helpful tool to enhance their insight into students’ activities (e.g., Melero et al. 2015; Wise and Jung 2019), and such tools were shown to improve teachers’ ability to detect meaningful events (e.g., Casamayor et al. 2009). On the other hand, there are also studies showing no improvement in this ability (Van Leeuwen et al. 2015), with the explanation that teachers may draw different conclusions from the same information, with the result that they do not always prioritize the group that is most in need of support. In their review, Sergis and Sampson (2017) suggest that instead of only providing information (i.e., mirroring), teachers may benefit more from dashboards that provide more extensive types of support. Martinez-Maldonado et al. (2015) compared a mirroring to an alerting teacher dashboard and found that only in the alerting condition, the teachers’ feedback significantly influenced students’ achievement. It could mean that teachers were better able to detect relevant information, upon which they could successfully act. However, Van Leeuwen et al. (2019) compared mirroring, alerting, and advising dashboards, and did not find a difference in teachers’ ability to detect the group that was in need of support.

Thus, there is a scarcity of research that compares the usefullness of different dashboard functions to each other, and the little research that is there does not provide conclusive results. Recently, there has been an increased interest in human-centered approaches to learning analytics (Buckingham Shum et al. 2019), meaning that authors are focusing more on why and how applications of learning analytics are (not) used, and how this relates to human factors. In light of this development, we argue that focusing on teachers’ perceptions of different dashboard roles could move the field forward in several ways. First, to investigate which type of dashboard is most helpful for teachers, it is important to investigate in-depth what teachers’ needs for support are during a complex situation such as student collaboration. Models of teacher decision making in the classroom have shown that teachers’ professional vision is highly important (Seidel and Stürmer 2014). Professional vision is teachers’ ability to notice and interpret significant features of classroom situations. In a way, teacher dashboards could be described as a tool to complement teachers’ vision because they capture data about what is happening in the classroom. Studying which information teachers use during collaborative situations to make decisions, could shed light on what kind of information on dashboards teachers could benefit from. Second, deciding about the appropriate type of dashboard also requires an investigation of their consequences for classroom processes. Studying teacher perceptions of each scenario of implementing dashboards is a valuable tool in mapping such possible consequences. Third, teachers’ uptake or adoption of technology is heavily influenced by their perceptions of the affordances of the technology (Gómez Domingo and Garganté 2016; Zhao and Cziko 2001). As described in the technology acceptance model, the core factors underlying the acceptance of technology are perceived ease of use and perceived usefulness (Granić and Marangunić 2019). Therefore, asking teachers to reflect on a number of situations in which dashboards are used can help to uncover what kind of dashboard uses teachers perceive as useful and easy to use. To conclude, we start our investigation with a qualitative exploration of teachers’ perceptions of different types of dashboards.

1.3 The present paper

As explained above, teacher dashboards may fulfill different roles (mirroring, alerting, advising, intervening) to support the teacher in monitoring student collaboration. In this paper, we focus on a comparison of these different dashboard roles. The design of this study consists of two phases that can be described as an exploratory sequential design (McCrudden et al. 2019): we first perform a qualitative investigation that leads to a follow up quantitative intervention.

First, a qualitative investigation of primary teachers’ perception of different types of dashboards was performed to obtain insights into what factors determine whether a dashboard is actually useful in the process of monitoring student collaboration. Interviews were conducted to take a human-centered perspective and discover in more detail teachers’ perceptions of different types of dashboards, in particular whether and why teachers show acceptance of dashboards performing different roles in the classroom. Subsequently, based on the results from the teacher interviews, we formed the hypothesis that the earlier conflicting findings about different dashboard roles may have been caused by a factor related to the classroom situation, namely time pressure in deciding which collaborating group to offer support to. We therefore performed an experimental study in which we compared a mirroring and advising dashboard and manipulated time pressure.

The following research questions were formulated:

Phase 1: What are teachers’ perceptions of the usability and acceptability of different types of teacher dashboards?

Phase 2: In the context of time pressure, what is the effect of the function of a teacher dashboard on teachers’ monitoring of collaborating groups?

Taken together, the two phases in this study aim to contribute to a better understanding of teachers’ perceptions of and interaction with different types of dashboards.

2 Phase 1: Interview study

A qualitative study using three interview techniques was performed to study primary teachers’ perceptions of different dashboard roles in administering CSCL.

2.1 Participants

As Onwuegbuzie and Leech (2007) note, sample sizes in qualitative research should be large enough to achieve saturation, which means that the collected data provides all relevant insights and further data collection will not yield new insights (Saunders et al. 2018). At the same time, the sample should be small enough to allow for a thorough analysis. Following these guidelines, we selected ten primary school teachers to participate in this study (Onwuegbuzie and Leech 2007). All participants were informed about the purpose of the interviews and signed informed consent. They received a monetary compensation for participating. The interviews were audio recorded so that they could be transcribed at a later point. After transcribing, the audio recordings were removed. The transcriptions were stored anonymously on a secured server.

Our aim was to obtain insights about teachers’ perception of dashboards for a wide range of teachers. In the sample of teachers, thus, variance was sought concerning age, teaching experience, and experience with the implementation of collaborative learning. The teachers’ mean age was 30.2 (SD = 3.9). On average, the teachers had 8.2 years teaching experience (SD = 7.4). Five teachers had less than 5 years teaching experience (range 1–3 years), the other five teachers more (range 5–13 years). Eight teachers were female; this majority of females is representative of primary school teacher populations in many countries (World data bank 2022), including the Netherlands, where the present study was conducted. Four teachers indicated they had relatively little experience with collaborative learning in their classrooms, the other six had considerable or a lot of experience. Four teachers indicated they had experience with dashboards that were part of software they used in their classroom for student practice of curricular material. As a measure of how the participants judged their own skills with using technology in the classroom, they filled in the perceived ease of use of technology questionnaire (Teo 2009) at the start of the interview (α = 0.75). On a scale from 1 to 5, all teachers scored at least 3.

2.2 Instruments

Three specific interview techniques were used to illicit teachers’ perceptions of different dashboard roles.

2.2.1 Contextual inquiry

To get a sense of how teachers detect and decide which collaborating group to offer support to, a mix between contextual inquiry and directed storytelling was used (Hanington and Martin 2012). A pressing source of load for teachers during CL is the amount of synchronicity in terms of multiple groups of students requesting assistance (Gillies and Boyle 2010). We therefore selected four situations that differed in the amount of help that the collaborating students requested from the teacher: 1) the collaborating students are at work without any group asking for help, 2) one group raises their hand to ask a question, 3) multiple groups raise their hands to ask a question, 4) the teacher is listening in on the conversation of one particular group and has the impression the group might have a misconception about the task. For all situations, the teachers were asked whether they would provide help (if any) and to what group, and what help they would provide.

2.2.2 Storyboards of different dashboard roles

In the second part of the interview, we investigated teachers’ perceptions of different dashboard roles, namely mirroring, alerting, advising, and automatically intervening dashboards. For this purpose, storyboards were used. A storyboard is a visual sketch of how intended users may interact with the object under study (Hanington and Martin 2012), in this case a teacher dashboard. The storyboards were presented to the participating teachers in the form of drawings. Drawings can show the right amount of context without moving visual details that distract participants from the purpose of the storyboard (Hanington and Martin 2012).

Four storyboards were created that depicted classroom situations in which students collaborated supported by technology (see Appendix A). We made use of the computer-supported context to gauge teachers’ perceptions of the automatic capture and analyses of student activities. In each storyboard, the teacher had access to a dashboard that either fulfilled the mirroring, alerting, advising, or intervening role, respectively. The storyboards were drawn from a third-person point of view. They included four panels of drawn images, each accompanied by a short text explaining the panel of drawings. The storyboards were given to all teachers in the same order. After each storyboard, teachers were asked for their perceptions concerning the function the dashboard fulfilled and the behavior of the teacher in the storyboard.

2.2.3 Reflective questions

The last part of the interview concerned general, reflective questions about teachers’ ideal vision of the implementation of teacher dashboards. These questions were purposefully placed at the end, because by that time, the teachers had been primed to think about a multitude of situations in which dashboards could be used during student collaboration, which enabled them to reflect on their overall thoughts. Examples of these questions were “What are your ideas and opinions about teacher dashboards?”, “Which tasks, when it concerns supervision of collaborative learning, would you entrust to a teacher dashboard?” and “What demands would a teacher dashboard have to meet to be regarded as trustworthy?”.

2.3 Analyses

A grounded theory approach (Boeije 2010; Strauss and Corbin 1994) was used to analyze the results, which fits the explorative goal of the paper and the qualitative nature of our data. Three phases are distinguished, namely open coding, axial coding, and selective coding (Boeije 2010). All interviews were transcribed and subsequently analyzed using the NVivo software. Before starting analysis, all participants were sent the transcript of their interview so they had the chance to make corrections (i.e., perform a member check).

Treating all interviews as one dataset, we first engaged in open coding, where tentative labels or codes are created for excerpts from the data. The codes were both based on constructs derived from the theoretical framework of the study (such as the role division between teacher and dashboard, Soller et al. 2005), but also for a large part based on what meaning emerged from the data itself (bottom up). To increase reliability (Boeije 2010), open coding was carried out on a subset of the data that contained answers from all three interview techniques by the first author together with a student assistant. This was done until saturation (Boeije 2010), that is, until no more new codes seemed to emerge. The first author coded the remaining data with the created codes, which led to only a few more new codes. Open coding resulted in a total of 90 codes.

Next, axial coding was performed: the codes resulting from open coding were sorted, compared, and condensed into overarching codes. In naming the overarching codes, literature was again used as input. For the overarching categories that could not directly be coupled to the sensitizing concepts, further literature was sought, such as the concept of teacher agency (Biesta et al. 2015) and teacher trust in technology (Lankton et al. 2015). This process resulted in the overarching codes considered to be the most important themes within the data. Because the code tree (Boeije 2010) had reached a manageable and concrete form after axial coding, interrater reliability was calculated at this point. A subset of 40 interview fragments were independently coded by the first author and a student assistant, leading to 90% agreement. The data were then fully coded by the first author (see Appendix B for the coding frequencies).

Finally, the phase of selective coding is meant to determine the relationship between the overarching codes and to establish the core themes that emerge from the data. An overview was made of which codes occurred most frequently for the data belonging to each of the three employed interview techniques. By comparing similarities and overlap between the codes, four main categories of relevant aspects mentioned by the teachers were identified (see Fig. 2).

Fig. 2
figure 2

Main and subcategories resulting from the interviews

Full size image

2.4 Results

With the interviews, our aim was to answer the question “What are teachers’ perceptions of the usability and acceptability of different types of teacher dashboards?” Fig. 2 shows the four main categories of relevant aspects mentioned by the teachers that determined their view on the acceptability of the dashboards, with subcategories for each category listed as well. We discuss each main category below.

2.4.1 Dashboard design

Teachers generally found that mirroring dashboards were acceptable, and in some cases thought so of the other types of dashboards as well, as long as two requirements concerning the dashboard design were fulfilled. First, the information offered by the dashboard had to be information that teachers could not figure out on their own, such as aggregated information at group or class level, or a different type of information than teachers have access to. A dashboard can provide insight into the development of the collaborative process, which is hard to do for every group for a teacher. Also, behavioral indicators that teachers observe are not always a reliable representation of what is actually going on. P8 gave the following example: “It could look like they [a collaborating group] are hard at work, but when you look closely and take a seat beside them you find out that one of the group members is doing the work for the other.”

Another essential aspect of dashboard design is usability. Most teachers indicated that having a mobile device such as a tablet was absolutely necessary, as it enables the teacher to remain mobile in the classroom as well (in contrast to a stationary desktop computer). The design of the display itself also has to be high in usability to avoid an overload of information. P3: “You can also have too much information, and then it becomes a disturbance. It would be good to be able to indicate what information should be displayed and what information should be disabled.” Thus, a certain level of customizability of the dashboard was deemed desirable by the teachers.

2.4.2 Teacher characteristics

Although the teachers generally showed acceptance of the mirroring dashboard, teachers’ views on the acceptability of the other types of dashboards showed more variation. Views on acceptability were found to be related to several teacher characteristics.

First, teachers’ beliefs of what unique skills teachers possess, in comparison to dashboards, played a role. The participants thought that a teacher is able to notice aspects of collaboration that a dashboard cannot easily capture, such as affective and behavioral indicators, as well as students’ collaborative skills as they develop over time (such as taking initiative and negotiating or coordinating with a collaborative partner). In the teachers’ opinion, this means that teachers should not solely rely on information from the dashboard. Going even further, some more skeptical participants remarked that acting on the dashboard’s information could lead to teachers losing the ability to objectively or neutrally observe students, because the dashboard might already steer them in a certain direction of judgement. Teachers’ knowledge of their students was also mentioned as a highly important teacher skill that influences decisions in the classroom. For example, knowledge of students’ past performance, students’ emotional states, and the specific combination of the collaborating group influences teachers’ prioritization of students’ help requests, as well as teachers’ choices for specific interventions. Most teachers thought that teachers, in contrast to computers, are able to adapt to specific students and circumstances in the classroom.

Another defining teacher characteristic that separated acceptable from unacceptable dashboards roles was the importance teachers assigned to agency, that is, the teacher’s active contribution to shaping his or her own activities and to influence the classroom environment (Biesta et al. 2015). The storyboards in the interviews were more often judged as acceptable when the dashboard made a suggestion and the teacher made the careful decision to act on the information or not, whereas unacceptable storyboards involved the teacher following the dashboard’s suggestion without proper thought. The second aspect of teacher agency is that teachers like to see the teacher being the one making the final decision about what happens in the classroom. P8: “I think the dashboard should primarily provide me with information and that I decide what I do with it. So I think it is most important that the teacher remains in control”. A third aspect of agency related to teachers’ identity. P4 for example stated that a dashboard “takes away part of the teacher’s professionalism”, and there thus seems to be an underlying sense of danger of dashboards replacing teachers in the classroom. The participants again voiced the importance of the teacher being in control over the dashboard and not the other way around, so that teachers avoid being on “automatic dashboard pilot” (P8).

The third teacher characteristic we identified was a sense of trust (or lack thereof) in technology (Lankton et al. 2015), in this case in the dashboards. The first indication of trust in dashboards was that teachers questioned the origin of the advice from the dashboard. Some participants openly expressed their doubt and their first reaction was to ask what the advice that the dashboards gave was based on. P7: “The most important thing is that the teacher goes to that group and interprets whether it [the dashboard’s assessment] is the case … because other things might be going on”. In contrast, some teachers would have liked to have even more substantive advice. Similarly, there were differences between participants concerning the alerting function of the dashboards. Some participants willingly accepted that the first alert should receive highest priority, and would correspondingly let other groups know they have to wait. P10: “The dashboard signalled group 1 first. They simply were first in line.”

2.4.3 Possible consequences of dashboard use

Another aspect influencing dashboards’ level of acceptability is teachers’ description of possible consequences for the classroom when they employ a certain type of dashboard.

The first category of consequences we derived from participants’ responses concerned collaborative processes and resulting student learning. In the specific case of the intervening dashboard, half of the participants indicated they would find it positive for students’ collaborative skills when groups receive automated support prompts and have to discuss among themselves what to do with the prompt. Another aspect that participants thought might influence students’ learning processes is the insight provided to the teacher. On the one hand, participants valued the idea of teachers being able to offer groups help even when students themselves are not yet aware or are not yet asking for support. On the other hand, according to the participants, being able to monitor one’s own learning is an essential skill, and the opportunity to develop that skill might be taken away when the teacher is alerted of every problem in advance without students’ own explicit reflection on their activities.

The second category of consequences concerned classroom dynamics. Although participants indicated they could imagine that a dashboard could improve the flow of activities in the classroom, in the last part of the interview they reflected on undesirable consequences for the relationship between the teacher and students. Especially the intervening dashboard was thought to lead to a different position of the teacher, since students could start to rely on the support offered by the dashboard, thereby again echoeing the fear of being replaced. The intervening dashboard was also thought to lead to a loss of interaction between teacher and students in general, and thus to less social cohesion.

2.4.4 The role of the classroom situation

Teachers face different challenges depending on the specific situation in the context of student collaboration they are in. In line with this finding, the classroom situation was a central theme in whether the use of the dashboard was perceived positive or negative by the teachers. In particular, most participants judged the intervening dashboard to be acceptable although it seems to contrast the earlier discussed theme of agency. In the teachers’ opinion, when the situation becomes highly demanding in terms of the amount or type of assistance required by the collaborating groups, they found it to be understandable that a teacher would choose to let the dashboard take a larger role by providing temporary automated support. In those cases, the participants gave priority to the teacher in the storyboard being able to finish his explanation and activate automated support for the other groups, even though “you would rather go the the groups yourself” (P8). Similarly, P6 states: “Sometimes as a teacher it is very important to finish your explanation, and then you choose to activate it [the intervening dashboard]. I think you should not do that every time. But when it is needed, it is good that the option is available.”

When explicitly asked to reflect on the difference between several situations during student collaboration, teachers typically mentioned the contrast between a low demanding situation, in which all groups are at work without any direct questions and where the teacher might walk around the class or even be seated at his desk, and a high demanding situation in which many groups request support. In the low demanding situation, teachers typically report that they would be most aided by knowing the status of each group, which provides them with the additional set of eyes to compare groups and to detect potential problems that they would otherwise remain unaware of. In the high demanding situation, the teachers’ reported need for aid is more about maintaining the flow of classroom activity and making sure they can help groups as adequately and efficiently as possible. By effectively allocating the teacher’s attention, the students’ requests for help can be adequately dealt with to make sure students do not remain stuck for too long. For example, teachers thought the dashboard could help them prioritize which groups are most in need of help, and aid in estimating how long they might need to help a particular group. Also, the dashboard could help teachers to determine whether multiple groups are facing the same problem, and thereby decide to give additional instruction to a subset of groups to avoid having to explain the same issue multiple times.

3 Discussion of interview study

In this qualitative investigation with a sample of 10 teachers, we identified patterns that could explain teachers’ attitudes towards different types of teacher dashboards. In line with previous studies, the teachers in this study generally were positive about the underlying idea of dashboards as teacher support tools, especially when it concerned the mirroring dashboard (e.g., Melero et al. 2015). The teachers remarked that dashboards make it possible to be constantly aware of the activities students are engaged in, which enables teachers to initiate interaction with students as well as use the information they gain from the dashboard to adapt their support when students have questions. Teachers thus see the potential of dashboards to enhance adapted teacher support towards each collaborating group both in a proactive and a reactive way (Onrubia and Engel 2012; Wise and Jung 2019). There are certain preconditions that a dashboard has to fulfill to be considered useful by teachers at all, such as being mobile enough to be moved around in the classroom. This is in line with the central assumption of usability in the field of human-computer interaction (Dix et al. 2004) and confirms other studies that identify usability as a factor that influences teachers’ implementation of technology in the classroom (e.g., McCulloch et al. 2018).

Besides usability issues, we also found that characteristics of the teacher played a role in teachers’ view of the acceptability of dashboards. In particular, trust in technology, teachers’ beliefs about what unique skills they possess in comparison with dashboards, and teachers’ need for agency were found to be influential. What it means for a teacher to trust a dashboard presents a complicated case. On the one hand, teachers mentioned that they felt the dashboards sometimes took away from their ability to remain objective towards their students, as the dashboard advises which collaborating group needs support. On the other hand, teachers also mentioned that they considered teachers’ subjective knowledge of their students a key teacher asset, as it enables them to tailor their support to specific students. This finding is an interesting confirmation of teachers’ overestimation of their own abilities to monitoring student collaboration, which they are in general not that good at (Kaendler et al. 2016). In terms of trust in dashboards, it underlines two issues in the implementation of dashboards. The first is that any dashboard reflects the designer’s choice of which student metrics to visualize, thereby implying the importance of optimizing or minimizing certain student behavior. Those choices need not necessarily correspond with teachers’ values of what constitutes desirable or effective student behavior. Recent studies show that what aspect of student behavior teachers base their pedagogical decisions on, are not always the same aspects of student behavior that actually relate to or predict student learning (Oudman et al. 2018). Teachers may therefore feel like the dashboard makes a wrong or non-optimal judgement, leading to a low level of trust. Related to that, the second issue is that teachers may feel less trust in a dashboard when they do not know or understand why a system makes a certain judgement, as the teachers in this study also indicated. Implementing a dashboard therefore may also imply a phase of informing or educating teachers as well (Matuk et al. 2016).

A key insight gained from the interviews we conducted is that the influence of these teacher characteristics on teachers’ views of the acceptability of dashboards seemed to be moderated by the classroom situation. Maintaining agency was found to be a core theme in teachers’ perceptions of the storyboards, and this directly related to the classroom situation. Even when teachers in general reported being hesitant about their willingness to accept alerting or advising dashboards, that judgement changed when confronted with a classroom situation in which the pressure to respond and tend to multiple groups was high. We therefore hypothesize that the experienced amount of pressure to quickly make a decision about which collaborating group to offer support to, influences the usefullness of dashboards with different functions.

As mentioned in Sect. 1, in an earlier study it was investigated whether there was a difference in teachers’ ability to detect the collaborating group in need of support when teachers interacted with different types of dashboards (Van Leeuwen et al. 2019). In that study, teachers were shown information about collaborating groups on either mirroring, alerting, or advising dashboards. The vignettes were manipulated so that they contained groups with need for support, for example a situation in which one of the group members was slacking off. Teachers’ task was to identify which of the groups was in need of support. The results showed that there was no difference in detection accuracy; in all conditions teachers were equally well able to detect a group with a need for support. In the design of that study, however, teachers could take all the time they needed to study the dashboard and make a decision. When we looked at how the teachers had used the dashboards, it was evident that the teachers generally took a generous amount of time to do so. They looked at the information from each of the collaborating groups, sometimes multiple times, before reaching their decision. We hypothesized that the fact that teachers had all the time they needed cancelled out the potential benefit of the support from the alerting and advising dashboards and explained why there was no difference with the mirroring condition.

The interviews in the present study pointed to the role of pressure to quickly make decisions in how teachers regard and are willing to use teacher dashboards. Therefore, in the follow up experimental study, we decided to operationalize that pressure by limiting the amount of time teachers have to monitor groups and decide which group is in need of support. Our expectation is that when teachers face time pressure, they are more likely to benefit from an advising dashboards compared to a mirroring dashboard and that differences in detection accuracy will occur. The experimental study served to test this hypothesis, and is described in Sect. 4.

4 Phase 2: Experimental study testing the prototype dashboard

An experimental study with two conditions (mirroring and advising dashboard) was carried out. The research question that was formulated for this study was: In the context of time pressure, what is the effect of the function of a teacher dashboard on teachers’ monitoring of collaborating groups?

4.1 Participants

The study had a sample of 35 participants, who were either pre-service primary school teachers or primary school teachers who had recently finished their teacher education. Participants signed up for the experiment voluntarily, they all signed informed consent, and received a monetary compensation for their participation. All data collected in the study was stored anonymously (by using participant IDs) and safely on a secured server.

Participants were randomly distributed over two conditions, leading to 17 in the mirroring dashboard condition (1 male) and 18 in the advising dashboard condition (2 male). Their mean age was 21.4 (SD = 2.2), and on average they had 30.4 months teaching experience in primary education (SD = 15.3). No significant differences between the conditions were found regarding these variables (p > 0.6 in both cases). In both conditions, 5 teaches indicated they had worked with a teacher dashboard before.

4.2 Procedure

The participants individually took part in the experiment, which was conducted fully computer-based. At the start of the experiment, participants watched a video that explained the procedure of the data collection and the layout of the dashboard they were about to interact with. The experiment continued with questions concerning teachers’ background variables (age, sex, teaching experience) and a yes/no question whether teachers had worked with dashboards before.

Participants were explained that their task was to imagine they were a teacher in a 4th grade class, in which dyads of students were collaborating on fraction assignments concerning the skills of naming fractions, simplifying fractions, and adding and subtracting fractions. The eight situations were derived from the existing software Mathtutor, a program designed to practice mathematics (including fractions) for both individual and collaborative settings (Olsen et al. 2014). With Mathtutor, both students have their own computer screen, but the interface they control is the same for the two members of the dyad. Because the software logs all student activity, these log-files can be processed and used as input for teacher dashboards.

It was explained to the participants that they would be shown eight dashboards showing information about 5 dyads of students who collaborated through Mathtutor. In the eight situations, which were shown in random order, the participants had to imagine that they consulted the dashboard to check whether any group might be in need of teacher support. It was stressed that participants should try to imagine they were in an actual classroom, even though the situations were fictitious.

Participants could click all class overview and group overview pages (see below) as often as they wanted and could end the situation if they had made a decision by clicking the ‘Finish’ button. To increase the pressure to quickly monitor the collaborating groups, the participants had 50 s in each situation to decide. If they did not press the ‘Finish’ button themselves before that time, the situation ended automatically.

4.3 Instruments

4.3.1 Independent variable: Mirroring and advising dashboard

The dashboards were designed based on a literature review and a co-design phase with teachers, see Van Leeuwen et al. (2019). Information was available for the teachers to browse concerning the following six indicators: 1) the number of completed assignments, 2) the number of attempts a dyad needed to solve an assignment, 3) the chance that a dyad displayed trial-and-error behavior on an assignment, 4) the amount of talk for each dyad member, 5) dyads’ proficiency on fraction skills, and 6) a display of a dyads’ activity over time. The dashboards displayed information about the collaborating students at class and at group level. Fig. 4 shows example dashboards for the mirroring and advising condition. On the left, buttons are displayed with the five dyad numbers. On the top row, six buttons are available for each of the six indicators. When a group button is clicked on, a group overview opens that displays information on all six indicators for that particular group. When an indicator button is clicked on, a class overview opens that displays information concerning that indicator for each of the five groups, see Fig. 3.

Fig. 3
figure 3

Screenshot of the dashboard with an opened group overview page, with information about six indicators (marked with numbers 1–6)

Full size image
Fig. 4
figure 4

Screenshots of the mirroring and advising dashboards. a Mirroring, b Advising

Full size image

The dashboard situations were designed in such a way that one of the five groups displayed a specific problem, based on literature about the characteristics of successful and less successful collaboration (e.g., Kaendler et al. 2015; Meier et al. 2007). Two situations showed a cognitive problem (e.g., a dyad being stuck on a problem), two situations included a social problem (e.g., a dyad showing lack of discussion), and two situations showed a combination of a cognitive and social problem. By setting up the values on the six indicators for one of the five groups in a particular way, the problematic situations were created. The remaining four unproblematic groups’ values were kept average. Finally, two situations did not include a problematic group, so all five groups showed average values.

The dashboards in the mirroring condition displayed information about the five groups for each of the six indicators (Fig. 4a). The dashboards in the advising condition displayed the same information, but also included a visual cue (an exclamation mark) that denoted the group that was in need of support (Fig. 4b). The group overview of the marked group also contained a text box that explained why the dashboard had marked the group as being in need of support.

4.3.2 Dependent variables: teacher monitoring of collaborating groups

Similar to the earlier study without time pressure (Van Leeuwen et al. 2019), participants answered the following questions after each situation: (1) which group they thought had faced a problem, (2) what type of problem the group faced, (3) whether and how they would intervene in these situations, (4) how much effort it took to answer these questions, and (5) how much confidence participants had in their answer. The amount of effort, which can be regarded as an indicator of experienced cognitive load, was measured with the widely used scale developed by Paas (1992), ranging from 1 (very, very little effort) to 9 (very, very much effort). The confidence question was measured on a scale from 1 (very unsure of my answer) to 10 (very sure of my answer). For purposes of the current study, we used questions (1), (4), and (5) for analysis.

4.4 Analysis

Out of the eight dashboard situations, we extracted the teachers’ detection accuracy, meaning the number of situations in which the teachers correctly identified the group that had faced a problem (or correctly identified that there was no problematic group). Furthermore, we calculated the average cognitive load and confidence level associated with selecting a group after each vignette. Because none of the teacher background variables differed significantly between conditions, detection accuracy, cognitive load, and confidence level were compared across the two conditions by means of independent samples t‑tests. The significance level was set to 0.05.

4.5 Results

The leading question in this experimental study was: “In the context of time pressure, what is the effect of the function of a teacher dashboard on teachers’ monitoring of collaborating groups?” Out of eight vignettes, participants in the mirroring condition on average identified the problematic group (or lack thereof) correctly 6.65 times (SD = 1.06), versus 7.50 in the advising condition (SD = 0.62). An independent samples t‑test showed this difference was significant, t (33) = −2.934, p = 0.006, d = 0.99. The average reported cognitive load was significantly higher in the mirroring condition (M = 4.52, SD = 0.97) than in the advising condition (M = 2.77, SD = 0.89), t (33) = 5.586, p < 0.001, d = 1.89. The average confidence level associated with selecting a group was significantly higher in the advising condition (M = 7.94, SD = 1.20) than in the mirroring condition (M = 6.74, SD = 1.06), t (33) = −3.118, p = 0.004, d = 1.05. In summary, the functionality of the dashboard had a positive effect on teachers’ detection of groups: in the advising condition, groups were more often detected, it cost teachers less effort, and teachers had more confidence in their decision.

5 Discussion of experimental study

Based on the interviews, the subsequent question addressed in the experimental study was whether teachers’ detection of groups would improve when they interact with an advising dashboard compared to a mirroring dashboard under the circumstance of time pressure. In an earlier study (Van Leeuwen et al. 2019), we found no difference in detection of groups, amount of cognitive load, or level of confidence in the decision which group to offer support to between these dashboard types. Informed by the insights gained from the interviews, a potential explanation for that finding was that teachers in the earlier study had all the time they needed to attend to the information on the dashboard. In the current study, we therefore manipulated the classroom situation to include time pressure, and we indeed found that in this case, compared to the mirroring condition teachers in the advising condition benefitted from having the dashboard point them in the direction of groups that may be in need of support in terms of a higher detection score. This may seem trivial because the dashboard was pointing out the group facing a potential problem, but it shows that teachers in this case were willing to follow this advice. Furthermore, it cost teachers less effort, and teachers had more confidence in their decision, again demonstrating that they followed the dashboard’s advice and that it strengthened their decision. If we regard the previous study (Van Leeuwen et al. 2019) and the present study together, the findings seem to confirm our hypothesis and extend earlier research by offering a direct comparison between a mirroring and advising dashboard. Of course, more studies are needed to show whether our findings are robust, but they offer initial indication that the advising dashboard is a form of support that aids the teacher in the crucial task of identifying collaborating groups that may need additional attention when time pressure is high. The way a teacher divides his or her attention and moves through the classroom can have a large impact on students’ on-task behaviors, and in turn, on their learning outcomes (Chiu 2004). Providing teachers with a dashboard that fits their needs in the classroom situation may therefore indirectly help teachers support student learning.

6 General discussion

In classroom situations in which students engage in collaborative activity, teachers may benefit from the support of dashboards that display information about their students to increase teachers’ awareness of the situation. In this paper we investigated the different roles dashboards may fulfill (mirroring, alerting, advising, intervening, see Soller et al. 2005) in two steps: we first examined teachers’ perceptions of different types of dashboards through interviews in which contextual inquiry and storyboards were used as interview techniques. Based on the results of the interviews, we conducted a follow-up experimental study that specifically compared mirroring to advising dashboards, mimicking a classroom situation in which teachers experience time pressure. The results of the experimental study showed that the advising dashboard indeed seemed to offer advantages to the teachers, because teachers were better able to single out groups in need of support, teachers experienced less cognitive load, and had more confidence in their decisions.

6.1 Discussion of findings

6.1.1 Teachers’ perception of control

If we look at the two studies in combination, the key issue that we can distill is the importance of teachers’ perception of control. From the interviews (phase 1) we infer that teachers may experience two types of control, namely chaos reduction and autonomy. During student collaboration, teachers’ aim is to keep everything under control in terms of the flow of activity in the classroom (i.e., chaos reduction), while they also want to be the ones who make the decisions and take responsibility for most tasks (i.e., maintain autonomy). Dashboards are systems that do what a teacher does, and therefore come close to a sense of “humanness”. Even though teachers in general would welcome dashboards that mimic their own behavior, teachers simultaneously perceive a potential danger of scenarios in which the dashboard controls or replaces the teacher. When the classroom situation is not heavily demanding and pressure is low, advising or intervening dashboards mostly form a threat to this autonomy, since chaos reduction is not necessary. However, when the situation does become demanding and pressure is high, the need for chaos reduction becomes higher. These types of high demanding situations are common during student collaboration, as there is typically a large amount of choices and open alternatives for pedagogical action in each moment (Prieto et al. 2015). In those situations, the teachers in this study report they are willing to temporarily share agency with the dashboard (phase 1), and as the experimental study showed (phase 2), the advising dashboard can play a helpful role in that situation. In the field of automated systems and the study of their usability, a related phenomenon is described that is known as the control paradox (Hargreaves et al. 2016; Randall 2003): the more advanced control capabilities a system provides, the more out of control users feel. This finding echoes the teachers’ fear of losing autonomy or agency in the face of dashboards that fulfill the advising and intervening roles that the teachers reported in the interview study (phase 1). Teachers’ sense of control, and how that relates to the classroom situation, is therefore is an important factor to take into account when implementing teacher dashboards.

6.1.2 The role of the classroom situation

The two studies presented in this article offer promising results in gaining a deeper understanding of the role of the classroom situation in how teachers view and make use of different types of dashboards. It is clear that a one size fits all approach is not enough and that teacher dashboards are more effective when they are tailored to the needs of the teacher in a specific context. The interviews showed (phase 1) that dashboards could be helpful tools for teachers especially when the dashboards have high usability, that teachers have different needs for support in different types of situations, and that the situation and the pressure that teachers perceive in that situation determines what teachers perceive as acceptable dashboard roles. The follow up experimental study (phase 2) confirmed the role of teachers’ perceived pressure. When teachers had only little time to make decisions (i.e., high pressure), the use of the advising dashboards led to higher detection accuracy than the mirroring dashboard. We therefore would recommend that future empirical studies and reflections on teacher dashboards take into account the role of the classroom situation.

6.1.3 Practical implications

Based on the findings from the present study, we formulate three practical recommendations for designing and implementing teacher dashboards. First, when school leaders or teachers are considering to start working with teacher dashboards, we recommend to start by analyzing the situation for which the school or teacher wants to use the dashboard. Is the envisioned use targeted towards informing teachers of their students during classroom action (when pressure can be expected to be high), or between activities or between classroom sessions (when pressure is probably lower)? The situation and associated pressure partly determines what type of dashboard is helpful and how teachers will reflect on the dashboard.

Second, when a choice for a particular dashboard is made, we recommend to invest time in explaining to teachers what the dashboard does exactly, what its intended use and benefits are, and how the function of the dashboard relates to their own autonomy as teachers. Discussing these issues in a teacher team may reduce a potential feeling of threat from the dashboard.

Third, a potential practical implication for the design of dashboards is to include the option for teachers to turn various roles on and off as the situation demands, so that teachers can decide when and to what extent to provide the dashboard with more responsibility. This form of customizability (Dix et al. 2004) would provide the teachers with the means to reduce chaos in high demanding situations, and to retain autonomy in low demanding situations.

6.2 Limitations

It must be noted that this study was conducted in a specific context, namely concerning primary education, small sized groups of collaborating students (i.e., dyads), and the specific domain of mathematics. Each of these characteristics influences the way students interact and how teachers interpret the collaboration. Given the complexity of monitoring collaboration, partly caused by the relatively young age of the students for whom it is difficult to collaborate productively, dashboards could be especially useful for teachers in this context. However, this also means that some of the thematic strands we identified in the results might not be generalizable to other contexts. For example, because the current study was conducted in primary education, students have less experience with collaboration and need more teacher support in this regard. The teachers in this study repeatedly voiced their need to check the activities in each group and the need to be the one to make the final decision concerning support offered to students. It could be that teachers in for example secondary education are more willing to initially let groups figure out arising problems on their own, and thus that those teachers may have different needs and opinions concerning the function of a dashboard. Therefore, we would recommend to replicate the studies reported here in other contexts to ensure the adequate design and implementation of teacher dashboards in those contexts.

Another potential limitation of the study is that we used interviews as a basis for the subsequent experimental study. It could be argued that interviews are less objective than for example observing what teachers do in the classroom and how they interact with different types of dashboards. However, because we were interested in teachers’ thoughts and perceptions, we decided interviews would be the best methodological option. We incorporated several interview techniques to elicit teachers’ thoughts and perceptions in a robust and valid way (Hanington and Martin 2012).

6.3 Directions for future research

The experimental study was conducted in a controlled lab setting and was therefore primarily focused on the teacher. An avenue for future research is executing studies with teacher dashboards in actual classrooms and over a onger period of time, thereby adding to the ecological validity of the study. In that case, time pressure and classroom context in terms of chaos could be measured not only in terms of teachers’ perceptions, but also in terms of the number of classroom stimuli that the teacher is subjected to. For example, one hypothesis to be tested is whether under conditions that have a relatively high amount of chaos, teachers are more inclined to consult a dashboard and/or more inclined to follow a dashboard’s advice. An approach such as the one taken by Knoop-van Campen and Molenaar (2020) could be relevant here. They looked at temporal sequences of teachers’ consultations of a dashboard and the feedback they gave to students as a proxy for teachers’ professional routines. These routines could be correlated to the amount of observed stimuli in the classroom to see if it predicts teachers’ dashboard consultation.

The second suggestion for future research is to take a step backwards in the chain of reasoning to discover what exactly causes teachers to feel time pressure and whether there are factors that would shield teachers from it. For example, teachers’ amount of experience with implementing collaborative activities may be of influence, since it leads to the development of specific mental schema and pedagogical knowledge that informs how and how effectively teachers monitor students’ activity (Kaendler et al. 2016). Other teacher-related characteristics that may predict the experiencing of pressure include teachers’ tolerance of ambiguity (Budner 1962; Prieto et al. 2015), which means that some teachers have a higher internal need for order and categorization and face more stress when presented with ambiguous stimuli or situations.

Research could also take a step forward and focus more on the consequences of a dashboard in a classroom in a broader sense. In the interview study, teachers also considered the acceptability of a dashboard in light of possible consequences for learning and interaction processes in the classroom. They mentioned for example that students’ skills of self-assessment (Panadero et al. 2016) may be negatively influenced when dashboards take over the responsibility of assessment and inform the teacher of students’ progress on tasks. The teachers also reflected on the role division between teacher and dashboard and how that may influence the teacher’s position in the classroom as well as the resulting interaction between teacher and student. These considerations point to the far-reaching consequences that introducing a dashboard in the classroom may have, and point to the need to consider the technology-enhanced classroom as a system in which teacher, student, and technology each take their own role while working together as a whole (Holstein et al. 2020). Teachers’ amount of teaching experience may therefore also play a role in whether or not teachers find different types of dashboards acceptable. Over time, teachers may come to learn and experience the potential advantages and disadvantages of using a teacher dashboard in more detail. As the present study included mostly teachers with little experience, repeating the study with more experienced teachers could be an avenue for future research.

6.4 Conclusion

To conclude, three core take away messages can be formulated. First, our results showed that situation-specific demands placed on the teacher during student collaboration determine what dashboard role is deemed acceptable by teachers and that under time pressure, teachers benefit from an advising dashboard compared to a mirroring dashboard. Second, our results give insight into teachers’ varying needs for control (in terms of chaos reduction and autonomy), which can lead to different teacher-dashboard interaction. Third, we outlined avenues for future research in the area of teacher dashboards, among which is a closer investigation of the aforementioned construct of teacher control, the demands placed on the teacher in the classroom, and subsequently, how dashboards may play a supporting role in different classroom situations.