Promoting regulation of equal participation in online collaboration by combining a group awareness tool and adaptive prompts. But does it even matter?

Abstract

Unequal participation poses a challenge to collaborative learning because it reduces opportunities for fruitful collaboration among learners and affects learners’ satisfaction. Social group awareness tools can display information on the distribution of participation and thus encourage groups to regulate the distribution of participation. However, some groups might require additional explicit support to leverage the information from such a tool. Therefore, this study investigated the effect of combining a group awareness tool and adaptive collaboration prompts on the distribution of participation during web-based collaboration. In this field experiment, students in a university level online course collaborated twice for two-weeks (16 groups in the first task; 13 groups in the second task) and either received only a group awareness tool, a combination of a group awareness tool and adaptive collaboration prompts, or no additional support. Our results showed that students were more satisfied when the participation in their group was more evenly distributed. However, we only found tentative support that the collaboration support helped groups achieve equal participation. Students reported rarely using the support for shared regulation of participation. Sequence alignment and clustering of action sequences revealed that groups who initiated the collaboration early, coordinated before solving the problem and interacted continuously tended to achieve an equal distribution of participation and were more satisfied with the collaboration. Against the background of our results, we identify potential ways to improve group awareness tools for supporting groups in their regulation of participation, and discuss the premise of equal participation during collaborative learning.

Introduction

Online distance education in the form of online-courses has received increasing attention over the recent years. Massive open online courses (MOOCs, Baturay 2015) however, face a number of challenges such as low participation and feelings of isolation that are partly caused by a lack of social interaction in the course (Khalil and Ebner 2014). Implementing collaborative learning activities can help increase interactivity within the course for the benefit of the students (Cherney et al. 2018; Rosé et al. 2015), as collaborative learning consists of meaningful interactions between learners. Research has accumulated compelling evidence that collaborative learning is an effective learning approach, both in face-to-face settings and in technology-enhanced settings (Chen et al. 2018; Pai et al. 2015; Tenenbaum et al. 2020).

Research on collaborative learning has shown that active participation of all members of a group is a conditio sine qua non because collaborative learning depends on students interacting to learn, for example by thinking aloud, giving explanations (King 2007) or by transactively building upon each other’s contributions (Weinberger and Fischer 2006). Further, coordination and regulation processes, which are key to successful collaboration, rely on students interacting with each other. For example Kwon et al. (2014) found that successful groups coordinated their collaboration early in the process, continuously engaged in socio-emotional interaction, and regulated their collaboration in a timely manner. Active and equal participation are further important for the regulation of the collaboration (Isohätälä et al. 2017). Especially for computer-mediated settings, interactivity is central as it affects social presence (Kim et al. 2011), which is one central challenge for virtual distance learning settings. This central role of productive interaction is reflected in the “interactions-paradigm” (Dillenbourg et al. 1996, p. 201), which highlights the need to analyze interaction processes during collaboration to gain insight into learning mechanisms and antecedents for learning. Yet, participation and effective interaction between the members of a group cannot be taken for granted (Kreijns et al. 2003). Rather, students benefit from additional scaffolding during collaboration so that they can engage in interactions that benefit learning (e.g., Rummel and Spada 2005). Additional challenges for the success of collaborative learning arise from unfavorable interaction patterns (Strauß et al. 2018), such as social loafing or free-riding. Social loafing is a term that describes situations where individual members of a group (intentionally) invest less effort than others into the collaboration and therefore rely more than expected on the efforts of the remaining group members (Aggarwal and O’Brien 2008; Karau and Wilhau 2020). In its most pronounced form, this can take the form of free-riding, that is, one or a few group members invest little to no effort while still benefitting from the others’ efforts. Since participation in general is a vehicle for constructing a social space (Kirschner et al. 2015) and knowledge (Kreijns et al. 2003), low and unevenly distributed participation leave fewer opportunities for coordination, regulation or building knowledge. This in turn negatively affects both the performance of the group and individual learning from the interaction. Furthermore, dysfunctional interaction patterns like unequal participation can also lead to dissatisfaction (Aggarwal and O’Brien 2008; Biasutti 2011; Capdeferro and Romero 2012; Keyton 1991). In groups, negative affect can reduce group cohesion and may lead to group members leaving the group (cf. Monzani et al. 2014; Peñarroja et al. 2017). As e-learning research has shown, students who are dissatisfied with the setting will more likely drop out of the course (Levy 2007), which further reduces opportunities for interaction among learners. Against this background, we argue that equal participation and satisfaction are pivotal to the success of collaborative learning. Since productive interaction relies on active and even participation by all members of the group, learning settings should support students in equal engagement of all members.

The present study was conducted in a university-level online course and focused on the distribution of participation in small groups and on students’ satisfaction with the collaboration. Specifically, we pursued two goals. First, we aimed to promote students’ regulation of participation during online collaboration with a combination of two types of collaboration support. Following the argumentation above, the collaboration support targeted the interaction in the group in order to achieve equality of participation and satisfaction with the collaboration. Secondly, since the antecedents of both equal participation and satisfaction are still understudied, we investigated the interaction patterns in the groups and how they relate to the distribution of participation and satisfaction. We aim to increase learning about how we can support groups in regulating their participation and about how interaction patterns relate to both the distribution of participation and satisfaction with the collaboration. This increased knowledge is meant to help educators improve the design of support and the implementation of adaptive collaboration support into educational practice (e.g., online courses).

The role of satisfaction and unequal participation during collaboration

Ortony et al. (1987) classify satisfaction as an affective state. In the context of learning, satisfaction can be understood as a positive affective reaction that results from the learning experience (Dewiyanti et al. 2007) if expectations are met or if needs are fulfilled (Hecht 1978). Following this general definition, satisfaction develops if the learning setting fulfills the needs of the learner or meets their expectations. While affective variables have received less attention in CSCL research than cognitive variables (Jeong et al. 2019), research from neighboring fields of study stresses the role of satisfaction underlying activity and engagement. On the one hand, positive correlations were found between satisfaction and perceived level of collaboration (So and Brush 2008) or a sense of community (Liu et al. 2007). Further, students report a higher satisfaction with their group learning experiences if the group members had developed a shared perception (Melzner et al. 2020). On the other hand, with regard to the level of interaction, especially unequal participation has been consistently reported as a challenge for collaboration and a source of dissatisfaction (Aggarwal and O’Brien 2008; Capdeferro and Romero 2012; Goold et al. 2008; Khalil and Ebner 2014; Le et al. 2018; Monzani et al. 2014; Peñarroja et al. 2017). Specifically, students’ self-reports showed that they expect all members of a team to contribute equally and “pull their weight” (e.g., Capdeferro and Romero 2012; Freeman and Greenacre 2011), because unequal participation is perceived as unfair (e.g., Price et al. 2006). Social loafing (i.e., group members exert less effort when working in a group as compared to working alone) or free-riding behavior (i.e., individual group members invest little to no work in the joint task but benefit from their team’s efforts) are more likely to occur in large groups (Aggarwal and O’Brien 2008), if the individual contributions are not identifiable (Johnson and Johnson 2009; Karau and Williams 1993; Price et al. 2006), or if students do not perceive the joint task as meaningful (Karau and Williams 1993). Furthermore, the negative effects of perceived social loafing on satisfaction are stronger in collaboration settings that employ low-fidelity communication channels such as text-based instant messaging (Monzani et al. 2014).

According to Kerr (1983), if group members perceive that the norms of collaboration (equity, social responsibility and reciprocity) are violated, they will reduce their efforts during collaboration to avoid being exploited. As Hütter and Diehl (2011) showed, group members were more likely to reduce their efforts in reaction to a loafing group member if they had strong sensitivity regarding equity. Against the background of the definition of satisfaction, we conjecture that students will experience dissatisfaction if their expectations of equity, social responsibility and reciprocity during collaboration are not met. In consequence, these students will reduce their efforts, that is, participate less in the joint work. In the worst case, low levels of satisfaction with a course can also lead to dropout from the course (Khalil and Ebner 2014; Levy 2007). If students disengage, reduce their participation or even exit the learning setting, there is less space for interaction that benefits collaboration knowledge construction, coordination, regulation, social presence or a sense of community. Harding (2018), as well as Robert (2020), showed that groups who reported less perceived social loafing showed better group performance. A further finding was that individual learning from the collaboration is indirectly affected by the increased group performance. Hence, unequal participation can be viewed as a central challenge for collaborative learning since it not only immediately reduces opportunities for collaboration, but also because it leads to dissatisfaction, which in turn can further reduce the space for productive interactions between learners. Instructional design can help reduce the risk of social loafing by increasing the social interdependence between learners (Johnson and Johnson 2009). During collaboration, however, learners still need to coordinate and to regulate to make sure that all group members are actively participating.

Supporting regulation of collaboration

During collaborative learning, learners not only need to regulate their own learning (self-regulation) but also have to scaffold the regulation of their peers (co-regulation). At the same time, the group as a system needs to regulate the behavior, thinking, motivation and emotions of its group members (socially shared regulation) (Hadwin et al. 2011). Research has shown that groups require support to engage in regulation of the collaboration and that technological support can potentially enhance regulation (Chan 2012; Järvelä et al. 2015). Following Järvelä et al. (2015), support for socially shared regulation should increase learners’ awareness of learning processes, facilitate externalizing these on a social plane, promote sharing and interaction in the group, and prompt the acquisition or activation of regulation. In the following section, we argue that adaptive prompts may complement group awareness tools and that a combination of both can support regulation of participation.

Regulating collaboration

As noted above, group members need to scaffold the regulation of their peers (co-regulation), while the groups as a system need to regulate the behavior, thinking, motivation or emotions of their group members (socially shared regulation) (Järvelä et al. 2018). For example, group members can align their task-perceptions in order to formulate subgoals (Hadwin et al. 2011; Hadwin et al. 2018). Soller et al. (2005) proposed the “collaboration management cycle” to model how groups monitor and regulate their collaboration. The model relies on the cybernetic idea of homeostasis and as such understands a group as a system that continuously tries to achieve an equilibrium. In particular, the model specifies five phases. During the first phase, the system (i.e., the group) collects data on the interaction, for example information on the participation of the individual group members. In the second phase, the group aggregates data into indicators that characterize the current state of the collaboration in terms of the desired aspect (e.g., the distribution of participation). The third phase consists of a comparison between the current state of the collaboration and a desired goal-state (e.g., equal participation). The goal-state can be set by the group itself (descriptive collaboration norm) or by external agents such as the teacher (prescriptive collaboration norm). In the fourth phase, the group initiates regulation if the current state and the desired state differ. The fifth phase consists of an evaluation of whether these regulation processes were successful. Depending on the outcome, the group may restart the cycle.

Soller et al. (2005) present one type of computer support for each of these phases. For instance, group awareness tools facilitate monitoring the collaboration by mirroring aggregated information on the collaboration (i.e., phase two). On the other hand, adaptive prompts provide groups with explicit guidance for effective interactions, thus functioning as an adaptive system (i.e., phase four).

Tacit guidance: Group awareness tools

Regulating the collaboration requires information about both the group members and the current state of the collaboration. This knowledge about the current state of the collaboration is called group awareness (Bodemer and Dehler 2011; Dourish and Bellotti 1992) and can for instance include knowing what other group members are doing (i.e., social group awareness) or what knowledge they possess (i.e., knowledge awareness) (Janssen and Bodemer 2013). In computer-mediated settings, developing group awareness is more challenging since social cues are more difficult to obtain than in face-to-face settings (Gutwin and Greenberg 2001; Janssen and Bodemer 2013). Group awareness tools (GATs) (Bodemer et al. 2018) can support collaboration in this regard as they collect, process and visualize group awareness information for the group. The learners in a group then can use this awareness to regulate the collaboration. In other words, GATs raise awareness of collaboration processes and provide an externalization that the group can use for monitoring and regulation, and thus are one part of support for socially shared regulation (cf. Järvelä et al. 2015). Bodemer (2011) describes this type of support as tacit guidance since GATs only provide a visualization of the current state of the collaboration, without imposing specific regulatory actions on the learners. For this reason, Soller et al. (2005) classify GATs as mirroring tools.

GATs support groups in developing group awareness, that is, knowledge about the past and current states of the task and characteristics and states of the group members. This awareness in turn provides a basis for group members to adjust their behavior. While the research literature on GATs does not yet offer a framework describing how GATs support the collaboration, existing studies propose three mechanisms that help understand how GATs facilitate regulation of collaboration. First, visualizing information on the collaboration (e.g., distribution of participation) makes this particular aspect of the collaboration more salient (Bachour et al. 2010) and thus draws the attention of the group to this aspect. As a consequence, it can be expected that a group would monitor this particular aspect and regulate it. Secondly, group awareness information provides (negative) feedback to the group (Jermann and Dillenbourg 2008), which in turn facilitates monitoring and group processing (Johnson and Johnson 2009). If a group detects a discrepancy between the current state of the collaboration and the desired state, the group is challenged to reflect on the collaboration and engage in regulation. Subsequently, a GAT helps monitor the progression towards the desired state. Thirdly, a graphical representation of participation within the group creates individual accountability (Johnson and Johnson 2009) by making individual group members more visible. If the individual contributions can still identify the amount of work members have contributed, they are less likely to engage in social loafing (e.g., Price et al. 2006). In addition, being able to discern individual contributions may also motivate group members to contribute to the collaboration through social comparison and thus reduce social loafing (Michinov and Primois 2005).

Prior studies showed that GATs that visualized the distribution of participation in a group fostered social aspects of the collaboration such as mutual understanding (Janssen et al. 2007). In addition, students who had access to a GAT sent more messages related to planning (Janssen et al. 2007). In terms of group performance, existing studies have not found advantages of social GATs for group performance (Janssen et al. 2007, 2011). However, the results by Harding (2018) suggest that groups with less social loafing also perform better. So far, studies have not found direct effects of GATs on the equality of participation (Janssen et al. 2007, 2011), instead, group members with and without access to a GAT contributed equally to the group task. Janssen et al. (2011) hypothesized that the inquiry task used in their study already provided enough interdependence between the learners in a group, which led to equal participation. While the GAT did not directly affect the distribution of participation in this study, participation was more equally distributed the longer students had the GAT opened during collaboration (Janssen et al. 2011). Thus, Janssen et al. (2011) concluded that students’ use of the GAT was crucial for the effectiveness of the tool and suggested that future studies should explore how students view and accept this type of collaboration support. Finally, Bachour et al. (2010) found that groups who were supported with a GAT achieved more equal participation when the group members perceived equal participation, as important.

While the theoretical accounts about the mechanisms underlying GATs as well as the empirical evidence reviewed above suggest that social GATs can effectively facilitate the regulation of participation, existing research raises two questions about the boundary conditions for the effectiveness of these GATs. The first question concerns the type of the task. Tasks that create strong positive interdependence between learners might reduce the need for additional collaboration support because learners are already required to collaborate more equally to solve the task (Janssen et al. 2011). Janssen and colleagues (Janssen et al. 2007, 2011) further conducted their studies in schools and had students collaborate over a longer period of time (multiple weeks). Thus, students had not only more time to work on the task, but also worked at school, which may have increased the pressure to contribute to the task. This raises the question of whether a social GAT that mirrors the distribution of participation is more effective in settings that provide learners with more degrees of freedom (e.g., studying in an online course) where tasks create less social interdependence.

The second open question is whether a GAT alone is an effective means to help groups achieve an equal distribution of participation. In their study, Dehler et al. (2009) provided dyads with a cognitive GAT that displayed the distribution of knowledge. They found that not all dyads benefitted from the GAT. The authors hypothesized that the learners may have overlooked the potential of the GAT as a means to regulate the collaboration. In addition, they hypothesized that some dyads lacked the competence to utilize the information displayed to adapt their interaction accordingly. Based on their findings, Dehler et al. (2009) as well as Janssen et al. (2011) therefore suggested combining a GAT with explicit support that suggests adequate next steps based on the current state of the collaboration.

Explicit collaboration support: Adaptive prompts

Prompts suggest potentially productive (inter)actions in the form of hints, questions or sentence openers (Ge and Land 2004). For example, prompts can help students activate specific learning strategies in face-to-face learning settings (e.g., Glogger et al. 2009; Loibl and Leuders 2019), but also in online courses (Wong et al. 2021). For collaborative settings, for example, Renner et al. (2016) found that dedicated prompts encouraged practitioners to engage in collaborative reflection. Implementing prompts that are adaptive to the learners (cf. Rummel 2018) can reduce negative effects that stem from redundant support (expertise-reversal effect, Kalyuga 2007). In the terminology of the collaboration management cycle (Soller et al. 2005) adaptive collaboration prompts are an adaptive system that scaffolds collaboration by collecting, aggregating and analyzing information about the collaboration. Afterwards, the support system compares the current state of the collaboration with a desired goal-state and presents a prompt if a discrepancy between these states is detected (guiding-system, cf., Soller et al. 2005). Adaptive prompts have been shown to elicit effective collaborative behavior such as helping tutors to provide additional conceptual help to their tutees (Walker et al. 2011), focusing the discussion during learning with multiple visual representations (Rau et al. 2017), formulating sub-goals during interaction with a pedagogical agent (Harley et al. 2018) and adjusting their explanation behavior (Gweon et al. 2006). The effects of adaptive support such as prompts on the distribution of participation in a group have not yet been addressed by research. The studies reported above, however, provide promising evidence that adaptive collaboration prompts are a suitable means to help groups engage in instructionally desired behavior, such as regulating the distribution of participation.

Combining group awareness tools and adaptive prompts

So far, not many studies have examined a combination of GATs and explicit support. Phielix et al. (2011) had students rate each other’s collaborative behavior (e.g., reliability and quality of contributions). The results of this assessment were then visualized and students collaboratively reflected on the results. This included deriving a plan for regulating the collaboration. This co-reflection activity served as explicit guidance because it scaffolded both deliberate processing of group awareness information and deriving regulatory actions. This combination of a GAT and prompts positively affected the groups’ social performance and the convergence between self and peer assessment of individual behavior in the group (Phielix et al. 2011).

Hadwin et al. (2018) augmented a collaboration script with different versions of a (meta)cognitive GAT to investigate the effect of different visualizations of individual planning perceptions. During an individual planning phase, students answered questions on their task perception and perceived goals of the collaboration. During a collaborative planning phase before collaborating, students received a visualization of the results of the members’ individual task perceptions. The collaboration script specified the order of activities (planning, collaborative work and reflection) and on which social planes these activities had to be carried out. The script further prompted planning and reflecting the collaboration. As a result, students who received the script and the additional GAT reported planning challenges as less severe. Students further perceived regulating the collaboration in terms of these challenges as being less difficult.

Taken together, these studies investigated combinations of implicit and explicit collaboration support. However, the support in both studies did not target the same aspect of collaboration. Instead, the GATs were directed at students’ perceptions of the collaboration and group processes, while the collaboration scripts were directed at processing the information offered by the GAT (Phielix et al. 2011) or augmenting the collaboration script by displaying students’ task perceptions (Hadwin et al. 2018).

Whether two complementary types of support can be combined effectively on the same target (i.e., regulation of participation) has not yet been investigated. Against the background of the design principles offered by Järvelä et al. (2015), combination of a GAT and adaptive prompts can effectively support socially shared regulation, as it promotes awareness of collaborative processes and helps activate or acquire regulation strategies.

Research questions and hypotheses

The present study investigated whether adding adaptive collaboration prompts to a social GAT supports groups in regulating the distribution of participation more effectively, as compared to groups who only receive a GAT or no support (RQ1). Following Dehler et al. (2009), not all groups might be able to regulate their collaboration solely based on the information provided by a GAT because they lack the appropriate strategies that would allow them to take advantage of the GAT. Adaptive prompts fill this gap by suggesting appropriate actions to groups who struggle to achieve equal participation. After following the prompt, the group can monitor their regulation using the GAT. Thus, we hypothesized that groups who receive a combination of a GAT and adaptive prompts would achieve a more even distribution of participation than groups who only receive a GAT or groups in the control condition without any additional collaboration support (H1). Secondly, we analyzed the relationship between the equality of participation and satisfaction with the collaboration (RQ2). Because learners have reported unequal participation as a source of dissatisfaction or frustration (e.g., Capdeferro and Romero 2012; Goold et al. 2008), we hypothesized that students who work in a group that exhibits unequal participation would report lower satisfaction with the collaboration after finishing the collaborative task (H2).

In addition to these top-down research questions, we performed two exploratory analyses. The first exploration concerned students’ perception and use of the collaboration support, because previous studies showed that the effectiveness of the collaboration is associated with students’ use of the tools (Janssen et al. 2011). In addition, Dehler et al. (2009) hypothesized that some groups may not view the GAT as a means to adapt the collaboration. We therefore aimed at discovering user-related boundary conditions for the effectiveness of collaboration support (exploration 1). The second exploration focused interactions that relate to the distribution of participation and satisfaction with the collaboration (exploration 2). As research on collaborative learning has shown, interaction processes are pivotal during collaborative learning (i.e., interaction paradigm, Dillenbourg et al. 1996). Thus, we assumed that the interaction in the groups would also affect the distribution of participation (cf. RQ1) as well as the satisfaction with the collaboration (cf. RQ 2). Analyzing the interaction can help us uncover the mechanisms underlying these variables, but can also allow us to identify new targets for collaboration support.

Method

Sample

We conducted a field experiment in an online-course about computer-mediated communication. The course design generally followed the design of a massive open online course (MOOC, Baturay 2015; Breslow et al. 2013) but also incorporated collaborative assignments for small groups. Undergraduate students from various bachelor’s and master’s programs from a large German university participated for course credit. Enrolled students could volunteer to participate in the study and received monetary compensation for their participation. Students who consented in participation in the study were randomly assigned to one of three experimental conditions: A control condition, a GAT condition or a GAT + Prompt condition. Due to attrition from the course, the sample size decreased over time. In total, 115 students enrolled in the course and 70 students initially agreed to participate in the study. Out of these students, 61 students (M = 23.17 years, SD = 4.14; 62.3% female) participated in the first collaborative task (topic#3), which started three weeks after the beginning of the course, and 54 students participated in the second collaborative task (topic# 4) (M = 23.18 years, SD = 4.18; 61.1% female). Out of these students, 42 students completed the questionnaire at the end of the course (M = 23.00 years, SD = 4.42; 66.7% female). Complete data (pre-questionnaire, logged activity in both topics, both short questionnaires and the post-questionnaire) was available for 41 students. Unfortunately, attrition is common for e-learning courses such as MOOCs (Breslow et al. 2013; Jordan 2015).

Participating students worked in groups of four within each experimental condition. These groups were randomly formed at the beginning of a new topic (cf. Procedure and Materials) and stayed together for the two weeks of the topic. The present study was conducted during two consecutive topics (topic #3 and topic #4). Thus, participating students worked in two randomly formed groups over a period of two weeks each. Students who did not agree to participate in the study were assigned to groups that only consisted of students who also did not participate in the study. Whenever necessary, we formed groups of five students to avoid groups with less than four group members.

Table 1 provides an overview on the respective sample size for each condition and collaborative task.

Table 1 Overview of sample sizes in both collaborative tasks

Design and materials

The online course comprised six topics that were each covered over a period of two weeks. We conducted the present study during two subsequent course topics (topics #3 and #4) that started in weeks four and six of the course. Similar to a MOOC, the learning materials for each topic consisted of a short lecture video (ca. 15 min) in which the central concepts of the topic (e.g., “Common Ground and Grounding”) were summarized, basic readings, and a voluntary multiple-choice quiz. In addition, students worked on one collaborative task during each topic.

We designed the collaborative tasks such that they imposed medium levels of social interdependence, that is, students could work on individual parts of the task before having to discuss their individual findings and arrive at a shared conclusion. The first reason for this design choice lay in the research aim of the present study. Janssen et al. (2011) hypothesized that a high amount of social interdependence (e.g., in a multiple-week inquiry task) may reduce the need for additional regulation support since the task already imposes a high need for coordination. The second reason was pragmatic. Since we expected attrition from the online-courses (Breslow et al. 2013; Jordan 2015), we developed collaborative tasks that groups would still be able to solve after suffering from member dropout.

Each collaborative task put the students in the role of consultants who had to suggest a solution to a problem from educational or work practice. Each group had to develop a joint answer text of at least 600 words. The answer should a) present central theoretical points for the respective problem, b) evaluate different solutions for the problem, c) present a solution to the target audience, and d) provide arguments for the decision presented. Each group received a group forum for communication between the group members and a group wiki for writing the joint answer text. During their collaboration, groups received collaboration support depending on the experimental condition (i.e., no support, a GAT, or a combination of GAT and adaptive prompts).

Collaboration support

The group awareness tool and adaptive prompts were implemented in the form of Moodle-plugins that were controlled by an automated support system. Both types of support were always visible in Moodle’s sidebar on the main page, the group wiki, and the group forum. The support was automatically updated whenever a student had contributed to the group’s wiki or forum (cf. Constapel et al. 2019 on the architecture of this system).

Group awareness tool (GAT)

Students in both experimental conditions received a GAT that visualized the distribution of participation using bar graphs (see Fig. 1).

Fig. 1

The social group awareness tool displaying the distribution of participation in a (fictitious) group

The number of words contributed to the group’s forum and wiki were automatically counted for each student, summed and then displayed in the GAT. The height of the bars was set relative to that of the highest bar in the group. Below the GAT, a legend identified the group members. Students could toggle between displaying the word count for the group’s wiki, the group’s forum or the total amount of words by ticking checkboxes above the visualization. The legend also displayed whether a student was currently online (checkmark) and provided shortcuts to compose a message to be posted in the group forum (envelope symbol) or send a private message to individual group members (speech bubble symbol).

Combination of GAT and adaptive prompt (GAT + prompt)

Groups in this condition received a combination of the GAT and the adaptive prompt. The adaptive prompt was displayed in a text box above the GAT (see Fig. 2).

Fig. 2

Group awareness tool (bottom) with adaptive prompt (top)

The prompt suggested to the group a specific action that would help them regulate their collaboration in terms of the distribution of participation. Besides suggesting an action, the prompt ended with a motivating statement that explained why it was desirable to reach equal participation. This motivator was based on prior research that found that students were frustrated by unfair distribution of participation (e.g., Capdeferro and Romero 2012). The prompts did not include an assessment of the current distribution of the participation (i.e., that the distribution is currently unequal) as this information was already provided by the GAT.

For example, a prompt suggested to the students to “Coordinate as a group and define which work will be done by whom and how much every member should contribute to the group tasks and discussions. This way, the group work remains fair” (cf. Fig. 2). We developed a pool of six synonymous prompts (see Appendix A, Table 14) from which the support system randomly chose a prompt if the analysis indicated that the number of words was unequally distributed. This aimed at reducing the probability that students would ignore the prompt text after each automatic diagnosis because the prompt text looked the same. Each prompt was introduced with “Based on the data, there is a hint for your group”. The actual prompt text then suggested to explicitly negotiate expectations regarding participation and work distribution so that all group members ‘pulled their weight’ and thus were responsible for the joint success.

The collaboration prompts were adaptive to the distribution of participation. That means that the support system displayed a prompt automatically if the overall word count in the group was distributed unequally. This was assessed using the gini coefficient (Janssen et al. 2011), which describes the deviation of a given distribution from equal distribution, ranging from 0 (equal participation) to 1 (unequal participation). If the gini coefficient was equal to 0.5 or greater, a prompt was displayed. This interval was assumed to represent a degree of unequal participation that causes dissatisfaction with the collaboration. If the gini coefficient was below 0.5, the text informed the group that there are currently no new suggestions available for them. The first prompt was displayed three days after the collaborative task had started in order to reduce redundant prompts that result from the first group member opening the conversation.

Procedure

Before the start of the course and the study, participants filled out an online questionnaire on socio-demographic data (pre-questionnaire, see Fig. 3).

Fig. 3

Timeline of the course and time points of data collection. The present study was conducted during topics #3 and #4

During the course, students accessed the course materials and worked on the collaborative tasks. At the end of each collaborative task, students answered the short-questionnaire that referred to the collaboration during the last collaborative task. To obtain as many responses as possible, this questionnaire only contained a limited number of items for each variable (attention to the collaboration support, use and perceived usefulness of the collaboration support, satisfaction with the collaboration). Only students in the experimental conditions answered the items about the collaboration support.

Students further indicated if they were still active in the course and would continue the course. Only students continuing the course were assigned to a new group for the next topic. After the second collaborative task (topic #4), a post-questionnaire was administered in addition to the short questionnaire. The post-questionnaire asked students to refer to both collaborative tasks and contained items for the perceived use and usefulness of the respective collaboration support.

Measures

Participation was assessed by counting the number of words each student had contributed to the group’s discussion forum and wiki during each collaborative task. Equality of participation was assessed by using the gini coefficient (cf. Dorfman 1979; Janssen et al. 2007) for the group’s word count distribution. The gini coefficient measures the degree to which the distribution of a trait differs from equal distribution. The coefficient can range from 0 (equal distribution, i.e., each group member contributed the exact same amount of words) to 1 (unequal distribution, i.e., only one group member has contributed). We calculated the gini coefficient using the R-package “reldist” (Handcock 2016).

Students reported their satisfaction with the collaboration by answering two items after each collaborative task. These items covered willingness to stay in the team (Hertel et al. 2004) “I would like to work in this group again”, and overall satisfaction with the collaboration “Altogether, I’m satisfied with the collaboration”.

Students reported on their attention to the collaboration support by entering the number of times that they had closely examined the support during the previous two weeks. Likewise, students reported on their use of the collaboration support by indicating to what degree they used the collaboration support (i.e., GAT or combination of GAT and prompt) to adapt the participation on the individual level (self-regulation) or the group level (socially shared regulation). The items were: “I have used the information to alter how much I contributed to the group task” (self-regulation), and “We as a group used the information to adapt how we collaborated” (socially shared regulation), respectively. As noted above, “the information” referred to the collaboration support. After the last collaborative task (in the post-questionnaire), students answered the same items but specifically for the type of collaboration support they had received during the collaboration (GAT or combination of GAT and prompt). In these items, students indicated the degree to which they used the support for self-regulation and socially shared regulation of participation.

Students rated the perceived usefulness of the collaboration support (GAT and prompt) by rating the following statement: “It was helpful to receive the information on the Moodle platform”. Note that the phrase “the information” referred to the collaboration support, that is, the GAT or the combination of GAT and adaptive prompts. After the last collaborative task (in the post-questionnaire), students answered the same items but specifically for the type of collaboration support they had received during the collaboration (GAT or combination of GAT and prompt). Additionally, students answered an open-ended question on how they perceived the collaboration support.

Coding interactions

We coded the contributions in the group forum and group wiki in terms of their function during the collaboration (see Table 2). To this end, we adjusted the rating schema by Curtis and Lawson (2001) to fit the collaborative task in our study, that is, coordinating and monitoring the progress towards a short joint wiki entry.

Table 2 Categories for learner interaction during online collaboration (based on the coding scheme by Curtis and Lawson 2001)

We coded forum contributions that targeted future actions (i.e., intentions, plans) as “coordination”, while forum contributions that contained statements on past actions (i.e., signaling that a wiki contribution was made) as “monitoring”. A “major contribution” was any contribution to the group forum or the group wiki that focused on extending the joint answer text and exceeded 600 words. Shorter contributions that only added scant new information or focused on changes to the text (i.e., correcting spelling mistakes, style or formatting) were coded as “minor contribution”. Thus, major and minor contributions were not exclusively determined by the length of the contributions but also by their content.

Two coders independently coded 200 randomly selected actions (20% of all actions) which yielded an inter-rater reliability of Cohen‘s κ = 0.79. Subsequently, one coder coded the remaining actions. After coding the actions in the group forum and group wiki, a “gap” action was added for every 24 h in which none of the actions from the coding schema occurred. This “gap” served to identify episodes with no visible interaction. Note that since logging into Moodle and accessing the learning materials were not visible to the other group members, learner-content interaction did not affect categorizing a 24 h period as a “gap”.

Results

Research question 1: Equality of participation

In total, students contributed between 380.41 and 474.42 words during topic #3, and between 255.25 and 401.12 words during topic #4. To test our first hypothesis, we compared the conditions regarding the distribution of participation as measured by the gini coefficient. Note that low coefficients indicate equal participation while higher coefficients indicate an unequal distribution of participation.

Table 3 provides an overview of the equality of participation (gini coefficient) at the end of the two collaborative tasks.

Table 3 Equality of participation (gini coefficient) in the groups at the end of the collaborative tasks

Mean gini coefficients of slightly above 0.5 indicate that most groups reached a medium degree of equal participation. The minima show that some groups were able to achieve a rather equal distribution of participation.

Mapping the development of the average gini coefficient over time for each collaborative task (see Fig. 4) revealed a descriptive linear trend towards equal participation. This finding indicates that accumulating over time, the number of words contributed by each team member became more equal over time. As noted above, however, the mean values show that the participation was still unevenly distributed at the end of the 14 days of collaboration.

Fig. 4

Distribution of participation (mean gini coefficient) over time. Smaller gini coefficients indicated more even distributions of participation within the groups

However, a Kruskal-Wallis test revealed no significant difference between the conditions in terms of equality of participation at the end of topic #3 H(2) = 0.463, p > 0.05, or topic #4 H(2) = 0.91, p > 0.05. This result suggests that neither providing a GAT nor the combination of a GAT and adaptive prompts lead to a different distribution of participation. Since we chose a gini coefficient of equal to 0.5. or greater to represent unequal participation requiring regulation, we also inspected how many groups arrived at a gini coefficient of ≥0.5 at the end of the 14 days of collaboration. At the end of topic #3, 12 out of all 16 groups (75%) achieved a gini coefficient of 0.5 or greater, at the end of topic #4, the percentage dropped to 38,5% (5 out of 13 groups). As noted above, the majority of groups struggled to achieve somewhat equal participation during topic #3, however during topic #4 groups more often achieved a more even distribution of participation. Comparing the experimental conditions (see Table 4) revealed that out of all groups that struggled to meet the criterion, more than half belonged to the control condition.

Table 4 Summary: Groups with gini coefficients equal to or greater than 0.5 at the end of both topics

Since the planned analysis of the H1 did not reach statistical significance, we rejected H1. However, analyzing the distribution of groups that actually reached a somewhat equal participation revealed tentative evidence in line with our hypothesis, that is, groups that received collaboration support will regulate their collaboration more successfully and achieve a more equal distribution of participation.

Research question 2: Relationship between the distribution of participation and satisfaction with the collaboration

At the end of each collaborative task, students indicated their satisfaction with the collaboration of the current group. On average, students reported medium levels of satisfaction with the collaboration after topic #3 (M = 2.89; SD = 1.14) and topic #4 (M = 3.52; SD = 1.01). We calculated bivariate correlations (Spearman’s rho) to determine the relationship between the distribution of participation and the satisfaction with the collaboration. Since these variables are on different levels of analysis, we had to transform one variable to match the levels of analysis. We aggregated satisfaction with the collaboration (individual level) to the level of the group by calculating the mean value for satisfaction for each group. We chose this aggregation approach over assigning the groups’ gini coefficient to each student (disaggregation) because we wanted to avoid the risk of increasing the chance for a Type I error; note that aggregating increases the risk of committing a Type II error (cf. Janssen et al. 2013).

The analyses yielded strong significant negative correlations rho(15) = −.61 (p < .05) for topic #3 and rho(15) = − .72 (p < .05) for topic #4. In line with our hypothesis, this result indicates that satisfaction was lower in groups that yielded a more unequal distribution of participation (i.e., groups where the gini coefficient was higher, thus indicating unequal participation), and vice versa. Both correlations can be qualified as strong (cf. Cohen 1992). Thus, H2 was confirmed.

Exploration 1: Reported use and perceived usefulness of the collaboration support

The aim of the first exploration was to gain more insight into how the learners used and perceived the collaboration support, that is, the GAT and the adaptive prompts. After finishing both collaborative tasks, participants reported that they closely inspected the collaboration support between three and four times during the two weeks of the collaborative tasks (see Table 5).

Table 5 Descriptive statistics for the reported number of times the GAT or GAT + Prompt was viewed (student level; short questionnaire)

Descriptively, participants in the GAT condition reported looking at the support more frequently. However, this difference did not reach statistical significance for topic #3 (U = 83.5; Z = −.92; p > 0.05) or topic #4 (U = 67.0; Z = − 1.7; p > 0.05). This indicated that the students from both conditions did not differ in the number of times they inspected the support.

In addition, the participants of the experimental conditions rated how useful the collaboration support was for them (see Table 6).

Table 6 Descriptive statistics for perceived usefulness of collaboration support at the end of the collaborative tasks (student-level; short-questionnaire)

Average ratings of around four points on a five-point Likert scale indicate that students generally perceived the support as helpful. On a descriptive level, again, students perceived the GAT as more useful than the combination of GAT and prompts. However, students did not differ in their ratings for topic #3 (U = 110.0; Z = −.02; p > 0.05) or topic #4 (U = 81.5; Z = −1.04; p > 0.05). When examining this result in more detail, students who received a GAT and adaptive collaboration prompts perceived the GAT as more useful than the collaboration prompts (Table 7).

Table 7 Descriptive statistics of the perceived usefulness of the GAT and the prompts (post-questionnaire)

These results suggest that the students in both conditions inspected the support and also perceived it as useful. However, the prompts were perceived less useful than the GAT.

Further, we analyzed how students used the support to regulate the interaction in the groups. Our results showed that students reported using the collaboration support rather for self-regulation than for socially shared regulation (see Table 8).

Table 8 Descriptive statistics (post-questionnaire) for use of support for self-regulation (SRL) and socially shared regulation (SSRL)

Students did not significantly differ in their reported use of the GAT for regulating their individual behavior (U = 99.0; Z = −0.276; p > 0.05) or their reported use of the GAT for regulating the collaboration on the level of the group (U = 69.0; Z = −1.65; p > 0.05). This finding indicates that students from both conditions did not differ in their use of the GAT for the regulation of their individual participation or the distribution of participation on the level of the group.

Students also answered an open-ended item in the post-questionnaire to elaborate on their responses. In general, students stated that the GAT was helpful. For example, because the GAT helped them see the “exact” distribution of the participation within the group (student from GAT + Prompt) and to what extend the other members of the group were participating. This allowed students to “learn, which persons are reliable (and which are not)” (student from GAT + Prompt).

In line with the quantitative results, students reported using the GAT predominantly for self-regulation and social comparison. In particular, students from both conditions perceived the GAT as useful because it helped them assess if they themselves were participating enough, compared to the other group members. One student noted that they would increase their participation based on the comparison of themselves and the other group members (student from GAT + Prompt). Regarding the regulation of the group, students reported that the GAT did provide a useful basis for task division (student from GAT), yet, students found it challenging to leverage the information to regulate the collaboration (student from GAT + Prompt).

Students’ responses rarely addressed the prompts, however. The few students mentioning the prompts stated that they perceived the prompts as generic and thus ignored them.

Exploration 2: Interaction patterns during collaboration

RQ 1 and RQ 2 were based on evidence on the relationship between the interaction in the groups and students’ satisfaction with the collaboration. Against the background of the interaction paradigm (Dillenbourg et al. 1996) we assumed that interaction patterns also relate to the distribution of participation and satisfaction with the collaboration. To learn more about potential interaction patterns and their relationship with these variables, we explored the interaction patterns in the groups through clustering groups with similar interaction sequences. A preliminary analysis of the interaction sequences has been reported by (Doberstein et al. 2017; Hoppe et al. 2020).

Applying the coding schema described above to the contributions in each group process resulted in one interaction sequence for each group (see Fig. 5 for an example). Note that the length of each interaction sequence depends on the number of logged actions, mostly irrespective of how much time had passed, because only “gaps” were defined using elapsed time (i.e., 24 h without any action that is visible for the group).

Fig. 5

Action sequence for one group. Color-coding: magenta: start of collaborative task, purple: gap; green: coordination; orange: major contribution; blue: monitoring; yellow: minor contribution

Because the groups were randomly re-formed after each collaborative task, we treated the groups from both tasks as independent units and included all groups from both topics into the analysis. Hence, our analyses included the sequences of 29 groups (16 groups from topic #3 and 13 groups from topic #4).

To sort groups by similarity of their interaction sequences, we applied the so-called sequence alignment procedure (or optimal matching) which originally stems from bio-informatics but has also been applied in the social sciences (Abbott and Tsay 2000). We performed the sequence alignment using the R-package “TraMineR” (Gabadinho et al. 2011). Sequence alignment determines the similarity between sequences (i.e., the interaction sequences of each group) by performing pairwise comparisons between all sequences (i.e., the sequences of all 29 groups). The sequence alignment procedure compares two sequences by transforming the sequence of one group into the sequence of the other group by injecting, deleting or substituting elements (i.e., the coded actions). Each transformation (injecting, deleting and substituting) is associated with a ‘cost’ that has to be specified before the procedure. The costs for the transformations are accumulated until both sequences in the pairwise comparison are identical. For similar sequences, the transformation costs are low. For our analysis, we set the costs for adding or deleting any two actions to ‘1’, while the cost for substituting an action for a “gap” was set to ‘2’, in order to reflect the importance of visible (inter-) action within the group.

The sequence alignment procedure results in a distance-matrix that includes the transformation costs (i.e., distances) for all pairwise comparisons. To find clusters of groups with similar action sequences, we applied the partitioning around medoids algorithm (PAM; Kaufman and Rousseeuw 1987) using the distance values from the sequence alignment procedure. This clustering algorithm consists of two phases. In the build-phase, the algorithm chooses a set of k units (k is specified by the researcher) that are most similar to the other units. These medoids represent the center of the clusters. Subsequently, the remaining units are assigned to their closest medoid. In the swap-phase, units that are currently not selected as medoids are tested as potential medoids. Therefore, the procedure iteratively exchanges the current medoids with unselected units to determine which unit yields the smallest distances between all units of the particular cluster. The procedure is terminated as soon as no possible exchange will decrease the distances between the units any further.

For our data, k values between 2 and 4 were tried. Setting k to values greater than 2 resulted in one cluster that contained the majority of groups while the remaining cluster(s) consisted only of a single group. Hence, we chose a two-cluster solution (see Fig. 6).

Fig. 6

Comparison of the action sequences of the two clusters. Cluster 1: n = 19 groups; Cluster 2: n = 10 groups. Color-coding of actions: magenta: start of collaborative task; purple: gap; green: coordination; blue: monitoring; orange: major contribution; yellow: minor contribution

The clustering algorithm resulted in two clusters that contained 19 groups (Cluster 1; 65.52% of all groups) and 10 groups (Cluster 2; 34.48% of all groups) respectively. Both clusters consisted of groups from all three conditions (Table 9). Cluster 1 mostly consisted of groups from the control condition while all three experimental conditions are equally present in Cluster 2.

Table 9 Composition of Clusters (percentage and number of groups)

Table 10 below shows that most groups from the control condition and the GAT + Prompt belonged to Cluster 1.

Table 10 Distribution of conditions between the clusters

However, a chi-square test revealed no significant association between the conditions and the clusters (χ² (1) = 2.79, p > 0.05). These two findings indicate that the collaboration support did not have a significant effect on the sequence of interactions within the groups.

As intended, the sequence alignment procedure and cluster analysis revealed clusters of groups that differed in terms of the amount of logged actions (see Table 11).

Table 11 Descriptive statistics for the number of actions and number of words per group in both clusters

In comparison, groups in Cluster 2 are generally more active, especially in terms of coordination. Monitoring actions, however, were rare in both clusters. These groups also showed fewer gaps, that is, periods of 24 h without visible interaction. While differences in terms of the number of actions were expected given the sequence alignment approach, the clusters also differed regarding the equality of participation and the satisfaction with the collaboration (see Table 12), two variables which were not entered into the cluster analysis.

Table 12 Equality of participation and satisfaction with the collaboration in the clusters

In comparison, groups in Cluster 2 achieved a more even distribution of participation at the end of the collaborative task (as indicated by a lower mean gini coefficient). In addition, groups in Cluster 2 also reported higher levels of satisfaction with the collaboration.

The major benefit of using the sequence alignment procedure is that the clusters not only differed in terms of the number of actions but also in terms of the sequence of these actions. To learn more about interaction patterns that are associated with the distribution of participation and satisfaction with the collaboration, we analyzed the interaction sequences of the clusters. We did so by proposing interaction patterns that distinguished the clusters. For example, upon a quick visual inspection, groups in Cluster 2 appeared to exhibit fewer “gaps” between the start of the collaborative task and the first action that is no “gap”. Thus, we compared the clusters in terms of “number of gaps before the first non-gap action”. By counting the number of gaps before the first non-gap action of each group within each cluster and calculation the mean, we compared the clusters. Table 13 provides an overview of those characteristics that potentially distinguish the clusters and often these interaction patterns occurred on average in the groups within their cluster.

Table 13 Frequency of interaction patterns in both clusters (mean and SD)

In line with the overall frequency of the individual actions analyzing the interaction sequences in the clusters revealed differences in terms of gaps and coordination. Comparing the interaction sequences revealed that groups in Cluster 2 started the collaboration earlier and generally had fewer and shorter episodes without any visible interaction. This is indicated by fewer gaps before the first non-gap action, shorter episodes of consecutive gaps and fewer gaps after at least one coordination action and one contribution were made. Another striking feature of the interaction in Cluster 2 is the focus on coordination. Specifically, groups in Cluster 2 engaged in coordination more frequently and always began their collaboration with a coordination activity. Groups in Cluster 2 also engaged in prolonged coordination, as indicated by more as well as longer episodes and consecutive coordination actions. Finally, groups in Cluster 2 more often than groups in Cluster 1 coordinated before the first contribution to the joint solution text was made and more often monitored their contributions after coordinating and contributing.

Discussion

Implementing collaborative learning into online education bears great potential because collaborative learning is not only an effective learning setting (Pai et al. 2015) but can also increase the interactivity in the course. One prerequisite for effective collaboration is that all members of the group participate in the interaction and construct knowledge together. From this perspective, unequal participation poses a challenge as it reduces the opportunities for productive interaction between the members of a group and is also bound to cause dissatisfaction, which further hinders productive interaction among learners. In the present study, we tested the effects of automated collaboration support on the distribution of participation, and investigated the relationship between participation, interaction and satisfaction with the collaboration. To summarize and discuss our findings, we organize the discussion into two parts. In the first part, we summarize our main findings concerning the effectiveness of the collaboration support and discuss the analyses of interaction sequences. In the second part, we critically discuss the opportunities and challenges of our research design, and the assumptions underlying our research.

Summary of findings

We used behavioral data about group members’ participation and related them to self-reported satisfaction with the collaboration (RQ 2). In line with existing self-report studies (e.g., Capdeferro and Romero 2012; Freeman and Greenacre 2011; Goold et al. 2008), we found that students who worked in groups with unequal participation reported being significantly less satisfied with the collaboration (and vice versa). With this, our study showed that the relationship between the distribution of participation and satisfaction with the collaboration also holds true if considering behavioral data that is available in a real online learning setting (i.e., number of words). So far, existing studies predominantly employed self-reports or artificial collaborative tasks, and did not provide direct observational evidence about the relationship between these variables. Our finding supports the notion of the importance of promoting the regulation of participation during collaboration. To foster the regulation of participation during collaboration, we provided groups with a GAT and adaptive prompts. We chose a combination of these types of support because prior studies suggested that groups working with a GAT may benefit from additional explicit support (Dehler et al. 2009; Janssen et al. 2011). In particular, we investigated the effects of combining a GAT with additional adaptive collaboration prompts on the same aspect of the collaboration, namely distribution of participation (RQ 1). It should be noted, that our results need to be interpreted with caution due to the small sample size (cf. below for a discussion of the research design). Our results suggest that participation generally became more equally distributed over time, however, only few groups reached somewhat equal participation at the end of the collaboration. In line with previous studies with comparable GATs (Janssen et al. 2007, 2011), we did not find a significant direct effect of the GAT on equality of participation. While the difference between the conditions did not reach statistical significance, inspecting the distribution of the participation (i.e., gini coefficients) across groups revealed that those groups who struggled to achieve an even distribution of participation (i.e., groups that yielded gini coefficients of 0.5 or greater) predominantly belonged to the control condition. While this finding does not warrant the conclusion that the collaboration support helped groups achieve a more equal distribution of participation, it gives rise to the conjecture that the support had at least a small effect. Still, contrary to our hypothesis, combining a GAT with adaptive prompts did not significantly affect collaboration in comparison to the other conditions. We will discuss the premise of equal participation as a valid goal of regulation in the second part of our discussion.

In addition, we explored students’ use and perception of the collaboration support (exploration 1), as well as the interaction sequences in the groups (exploration 2). The first exploration provided insights into how students in an online course use collaboration support such as a GAT and adaptive prompts for regulation. Specifically, students perceived the GAT as helpful. They did, however, not use it to initiate regulation on the level of the group. Regarding the adaptive prompts, students reported having payed less attention because they perceived them as less helpful. These findings may provide explanations for our results on the main hypotheses, as we will discuss in more detail in the next section. The second exploration revealed interaction patterns that relate to the distribution of participation and satisfaction with the collaboration. Groups who started their collaboration early, interacted often and continuously, showed shorter episodes without any interaction, and focused on coordination before working on the joint solution, achieved a more even distribution of participation and also reported higher levels of satisfaction. This finding is in accordance with the results of the cluster analysis reported by Kwon et al. (2014) who found that the most successful groups exhibited early coordination, continuous socio-emotional interaction and timely regulation. Our analyses corroborate the finding that coordination should occur early in the process and that groups need to take time for this coordination. Regarding the distribution of participation, it can be hypothesized that groups who start to collaborate earlier and coordinate more often will achieve a distribution of participation that is more equal since they have a greater chance to evenly distribute tasks and responsibilities and also have a longer timeframe so that each member can contribute.

Promoting regulation participation with a combination of a group awareness tool and adaptive prompts

The planned analyses did not show the hypothesized effects of the collaboration support on the distribution of participation. In discussing our findings, we focus on boundary conditions and potential challenges for students when leveraging a GAT and adaptive prompts to regulate the distribution of participation.

In line with studies by Dehler et al. (2009), and Janssen et al. (2011), our results underscore that leveraging a GAT for regulation can be challenging for students and that students’ use of the collaboration support is crucial for its effectiveness. In this section, we discuss potential reasons for our findings along the phases of the collaboration management cycle (Soller et al. 2005). Recall that this model proposes that groups 1) gather data on the collaboration and aggregate it to develop indicators that allow assessing the collaboration, 2) take up the information and relate it to the collaboration (i.e., develop group awareness), and 3) process the information, that is, compare the current state of the collaboration with a desired state and diagnose if regulation is required. Based on this, groups 4) choose and enact strategies to regulate the collaboration, and eventually 5) control the success of the regulatory actions.

In the present study, groups in the control condition had to perform all these phases without additional guidance, whereas groups in both experimental conditions received support for the different phases of the collaboration management cycle. Both conditions received a GAT that supported phase 1 (collecting information about the collaboration) by displaying aggregated data on the distribution of participation. Groups in the GAT condition had to perform all subsequent phases without additional guidance. That is, they had to interpret information visualized in the GAT and decide whether regulation of the participation was necessary. Based on this assessment, the group could initiate adequate regulatory actions. Afterwards, the group members could use the GAT to assess whether their regulation had been effective. In contrast, groups in the GAT + Prompt condition received support for all phases. Beside the GAT, the support system showed an actionable prompt if the current distribution of participation indicated unequal participation. Groups could follow the prompt and use the GAT to assess if the regulation was successful. Consequently, groups in this condition received support for all phases of the collaboration management cycle except for the evaluation phase, where the group reassesses if regulation was successful. In the following, we will discuss potential reasons our findings separated by the phases of the collaboration management cycle.

Phase 2) Taking up group awareness information on the distribution of participation. In order to take up information from the GAT, learners have to pay attention to the visual representation provided by the tool and process the information that is being displayed. The GAT in the present study visualized information about the distribution of participation, which was defined as the distribution of uttered words. Through the visualization, this particular aspect of the collaboration was expected to become more salient for the students, thus making it more likely a target of regulation. Unlike in prior studies with similar GATs (e.g., Janssen et al. 2011), the GAT in our study was constantly visible during collaboration. Therefore, students did not have to deliberately open it to view the distribution of participation. As the GAT was always visible, we were not able to collect data regarding students’ actual use of the tool. Students reported having paid attention to the collaboration support, especially to the GAT. Yet, looking at the information displayed in the GAT does not automatically imply active processing. Hence, it remains unclear to what extent students processed the information with the goal of regulating the collaboration. To foster active processing of information, designers of collaboration support should consider findings from human-computer interaction research (cf. for example Jacko 2012).

Phase 3) Using group awareness information to assess the need for regulation. In order to assess the need for regulation, learners have to actively process the information visualized in the GAT with this very intention. While the collaboration management cycle takes for granted that the GAT triggers a corresponding reflection process, our results suggest that this is not necessarily the case. We have three hypotheses offering potential explanations about why groups did not achieve equal distribution of participation based on information from the GAT. First, as the collaborative tasks were part of the course requirements, task completion can be hypothesized to be a competing goal. If task-completion was the primary goal of the groups, students would focus on creating a text that fulfilled the requirements (e.g., length of the wiki article and covering certain topics) instead of achieving equal participation. Thus, regulating the collaboration would be less of a priority for groups. However, our data does not allow for clarifying whether this explanation holds true.

Secondly, some comments by the students indicate that the students assigned functional values to the GAT that we had not intended when designing the tool (cf. appropriation, Tchounikine 2017). We conceptualized the GAT as a means to develop group awareness and compare the current distribution of participation with a desired goal-state, in order to assess the need for regulation. One functional value that students perceived was using the GAT to assess which group members were reliable and therefore could be trusted in future collaborations. Developing trust during collaboration is important and especially difficult in virtual collaboration (Robert 2020). As research has shown, high levels of trust can reduce individual loafing behavior (Robert 2020; Williams and Karau 1991). With regard to regulation, however, we assume that this functional value may rather contribute to regulating their own participation behavior (e.g., social compensation instead of individual loafing, Williams and Karau 1991), instead of initiating regulation on the level of the group. This exploratory finding provides preliminary support for the hypothesis formulated by Dehler et al. (2009) that groups may not view a GAT as a means to regulate their collaboration and thus some groups may fall short to leverage information provided by the tool.

Thirdly, regulation of participation is less likely to occur if students perceived no need for regulation, that is, if they concluded that the distribution of participation is acceptable. While the GAT only visualized group awareness information about the distribution of participation, the adaptive prompt made suggestions to the members of the group in order to align their understanding of how the group should distribute the participation, and to distribute the workload accordingly. However, the prompt did not explicitly state that the current distribution of participation was unequal and required regulation. Thus, if students viewed the GAT and assessed the current state of the collaboration, but arrived at a different conclusion than the adaptive system, students may reject the prompts instead of initiating regulation. Students self-reports, however, did not indicate how large the difference has to be to spark an increase in participation to match the participation of the other group members. With regard to the adaptive collaboration prompt, this would imply that the threshold for intervention should be recalibrated based on collaboration data. For example, a linear regression could be used to determine the value of the gini coefficient that is associated with a high level of dissatisfaction. Unfortunately, the quality of our data does not allow to perform such regression analysis.

Phase 4) Regulating the collaboration to achieve equal participation. In order to regulate the collaboration and achieve equal participation, the learners of a group need to align their perceptions regarding the distribution of participation, decide on individual as well as joint regulatory actions and act them out effectively. The self-reports suggest that students would only regulate their participation (i.e., increase participation) if they noticed that they had contributed less than the members of their team. This is in line with Jermann and Dillenbourg (2008) who explain the effectiveness of a GAT through regulation processes that are triggered by social comparison. Why students rather initiated self-regulation instead of regulation on the level of the group could be interpreted against the background of the conceptualization of cognitive group awareness by Engelmann et al. (2009). The authors propose that (cognitive) group awareness exists on the level of the individual, unlike related concepts such as common ground (cf. Clark and Brennan 1991) or transactive memory (cf. Wegner 1995) that exist on the group-level. In other words, an individual group member can be aware of what their peers know, but this does not necessarily hold true for the other group members. If group awareness is only held by individual group members, regulation can be expected to occur rather on the level of the individual instead of the level of the group.

Moreover, some students commented that they perceived it as challenging to initiate regulation on the group level. These students hypothesized that this was due to the communication channel (forum) and due to the generally low level of activity in the group. This underscores the notion that groups need additional support to leverage a GAT for regulation. In our study, the combination of a GAT and adaptive prompts provided additional guidance that aimed at promoting regulation on the level of the group. However, our results indicate that students rather neglected the adaptive prompts since they perceived the prompts as not helpful.

Above, we hypothesized that students may have perceived no reason for regulation and thus did not adapt their collaboration. Another potential explanation focuses on students’ regulation skills. The combination of GAT and adaptive prompts on the same target (i.e., regulating the distribution of participation) was based on the assumption that students possessed regulation strategies that could be triggered by a prompt. The adaptive prompt in the present study asked students to negotiate the distribution of effort explicitly on the level of the group. If students were not sufficiently proficient in engaging in this behavior (i.e., mediation deficiency, Flavell et al. 1966; Hübner et al. 2010), regulation was unlikely to occur, irrespective of the experimental condition (GAT or GAT + Prompt). This would call for collaboration support that is more sensitive to students’ competencies (see below).

Using sequence alignment and cluster analysis to investigate factors associated with equal participation and satisfaction

The present study also explored interaction patterns (exploration 2). Due to the cluster analysis approach, our results do not allow for inferring causality. However, we hypothesize that the differences between the two clusters partly stem from the type and frequency of interaction in the respective groups. In particular, the groups in Cluster 2 coordinated frequently and interacted continuously. In addition, these groups showed a more even distribution of participation and were also more satisfied with the collaboration. The difference in satisfaction with the collaboration between the clusters can be explained by the correlation between satisfaction and equality of participation (cf., RQ2). In addition, the higher levels of interactivity between the learners in the groups in Cluster 2 may have increased students’ satisfaction. The relationship between interactivity during collaboration and satisfaction with the collaboration can either be immediate or mediated. Evidence for a direct relationship between interactivity and satisfaction comes from e-learning studies that found a positive correlation between interactivity and satisfaction (Sher 2009; So and Brush 2008). However, not all studies found this effect, or only small effects (Kurucay and Inan 2017). Alternatively, the relationship between interactivity and satisfaction can be mediated by social presence. That is, more interaction in the group would increase social presence, which in turn increases satisfaction with the collaboration. This relationship is, however, not well researched. Some studies have shown a positive correlation between interactivity and social presence (Kim et al. 2011; So and Brush 2008). For the relationship between social presence and satisfaction, the meta-analysis by Richardson et al. (2017) identified a moderate correlation. However, Weidlich and Bastiaens (2017), take a contrary position and argue that research in this area has been using convoluted definitions of social presence. Thus, the relationships between social presence and other variables may be less clear than previously assumed. For their study, the authors defined social presence as a feeling of non-mediation or a “[…] sensation of the other being “there” and “present”” (Weidlich and Bastiaens 2017, p. 482). Using a structural equation model, they investigated the relationship between social presence and related variables such as social space, impression formation or satisfaction in an asynchronous distance education course. They found a positive correlation between learner-to-learner interaction in the online course and social presence. This relation was mediated via the process of impression formation. This indicates that students formed an increasingly distinct impression of their communication partners through continuing interaction, which eventually lead to social presence. Contrary to existing studies (cf. meta-analysis by Richardson et al. 2017), Weidlich and Bastiaens (2017) found no direct correlation between social presence and satisfaction. Instead, they found that social presence is directly associated with perceived learning and social space (relationships between learners), which in turn are both associated with satisfaction (in this case, satisfaction with the course). Hence, the study by Weidlich and Bastiaens (2017) at least calls into question the often postulated direct relationship between social presence and satisfaction as well as between social interaction and social presence. For the present study, these results support the assumption that the higher degree of satisfaction with the collaboration partly stems from a higher degree of interaction (and the resulting social space within the group). However, we would welcome future studies that help to shed more light on the predictors and results of satisfaction with the collaboration.

Conducting a field-experiment in an online course – Pros and cons

We conducted a field experiment in which we tested the effectiveness of automated collaboration support. A strength of the current study is that we implemented collaboration support in an ecologically valid learning setting where students collaborated at a distance over a month in total while receiving automated support. Conducting a study over an extended period of time allowed us to collect data that reflected students’ experience during collaborative online learning. At the same time, conducting a field-experiment introduced limitations that affected the internal validity and statistical power. First, recruiting a large number of groups from an online course was challenging because the number of enrolled students was limited. In addition, voluntary study participation and attrition from the course over time (which has been reported for online courses; Breslow et al. 2013; Jordan 2015) formed a “funnel of participation” (Clow 2013) that reduced the sample size. In particular, out of all students who enrolled in the course, only a fraction was active during the first week of the course. Of these students, only a certain percentage agreed to participate in our study. In addition, not all of these students actively participated in the study and filled in the questionnaires. As a consequence, and despite monetary compensation and efforts to keep the workload caused by participating in the study to a minimum (i.e., filling in questionnaires that were kept short), we were only able to retain less than half of the enrolled students as participants. We want to stress that these phenomena pose a challenge for experimental intervention studies in the field of online education since they affect statistical power and the statistical methods that are available. In our case, we decided against conducting hierarchical-linear modeling (HLM) because the small sample size of less than thirty groups (second-level) can be expected to cause heavily biased estimates (McNeish and Stapleton 2016). However, conducting HLM would be preferable due to the nested nature of the data (i.e., individual students working in groups) as well as the combination of variables with different units of analysis (i.e., individual and group level) (Cress 2008).

Another limitation that stemmed from the field-setting was that we had to limit the number of items we could administer in the online questionnaires. This decision aimed at keeping the return rate in the online course as high as possible but might have reduced the overall reliability of the measurement. Similarly, assessing socially shared regulation with a single questionnaire item could only serve as preliminary exploration. Likewise, due to the overhead caused by the study as well as due to local requirements it was not feasible to collect individual pre-tests and post-tests in addition to the mandatory tasks in the course. Thus, we were not able to collect data about individual knowledge gains. A general drawback of self-reports is the social desirability bias. Using behavioral data to capture students’ perception and use of the collaboration support would have decreased the potential for this effect and would also provide a more reliable measurement. As the support was always visible, collecting log-data on students’ use of the support was not possible. To investigate how students used the collaboration support for socially shared regulation processes in more detail, qualitative analyses of forum discussions and interviews would be desirable. Despite the limitations discussed above, we believe that conducting field experiments that test the effectiveness of interventions in authentic learning contexts is valuable for both, research and educational practice. In the field, interventions such as collaboration support have to work without additional beneficial effects that are caused by the laboratory setting.

Promoting equal participation – But does it even matter?

At the core of the present study was the distribution of participation during web-based collaboration. Based on the numerous studies that consistently found unfavorable effects of unequal participation, social-loafing and free-riding behavior (Aggarwal and O’Brien 2008; Boren and Morales 2018; Capdeferro and Romero 2012; Freeman and Greenacre 2011; Goold et al. 2008; Harding 2018; Johnson and Johnson 2009; Karau and Williams 1993; Kerr 1983; Khalil and Ebner 2014; Le et al. 2018; Monzani et al. 2014; Peñarroja et al. 2017; Price et al. 2006; Robert 2020; Williams and Karau 1991), our study was guided by the premise that equal participation during collaboration is beneficial for collaboration. We acknowledge that using the number of words as an indicator for participation might not be optimal. Further, there may be additional nuances to equal participation as a desired state during collaborative learning. We therefore want to discuss our operationalization of participation, and also challenge the premise of equal participation during collaboration.

The number of words is a commonplace indicator for social GATs (e.g., Janssen et al. 2007, 2011; Jermann and Dillenbourg 2008) and as a dependent variable for measuring participation (Hrastinski 2008). Against this background, we, too, assessed participation during collaboration via the number of words that the group members had contributed. Furthermore, following Janssen and colleagues (Janssen et al. 2007, 2011), we used the gini coefficient to determine the distribution of participation in the groups. This coefficient indicates how far the distribution of participation deviates from perfectly equal participation. Because we assumed that students would use the extent of their peer’s work (i.e., how much they had written) as preliminary evidence for their peers’ participation, we expected that the number of words would play a role for their perception of the collaboration, and consequently would affect their satisfaction. We expected that any meaningful interaction between the members of a group requires measurable participation in the form of words that were exchanged. Using the number of words for the collaboration support, we conceptualized groups as successful regulators if they achieved an even distribution of the number of words. Arguably, however, using the number of words as an indicator for participation may not be optimal. On the one hand, visualizing the number of words may lead to unwanted effects. Following the argumentation by Wang et al. (2011), making the raw number of words available to the students may lead students to think that contributing large amounts of text is expected. Thus, emphasizing the number of words and their distribution may lead to process losses as it may result in text production becoming the short-term goal, instead of promoting meaningful interaction between group members, thus compromising the long-term goal of collaborative learning. Instead of providing sound arguments or contributions of higher quality, students may simply provide any text. In the worst case, this could also result in students ‘gaming the system’ by contributing nonsensical or very shallow text. However, our results did not indicate such gaming behavior. Indeed, it seems that the students used the number of words to determine whether they had to contribute more and if their peers were trustworthy.

On the other hand, using the number of words to operationalize participation for our analyses has implications for our findings. Existing studies on social loafing used behavioral indicators such as number of ideas generated, clapping or shouting (cf. Karau and Williams 1993) or students‘perception of participation and social loafing (e.g., Aggarwal and O’Brien 2008; Peñarroja et al. 2017) to measure participation and unequal participation. Therefore, while our results are in line with our prediction that unequal participation, measured by means of word count, is a source of dissatisfaction, not all of the effects found in prior studies may generalize to our operationalization of participation. By the same token, our analyses may not have been sensitive to process losses or challenges that do not manifest themselves in uneven participation assess by an uneven amount of words (e.g., developing a shared task-perception, Hadwin et al. 2018). The study by Wang and Rosé (2007) showed the benefit of including the quality of the ideas that were generated during a brainstorming task instead of only focusing on the number of accumulated ideas. For our case of collaboration in an authentic learning setting, using a more holistic operationalization (cf. Hrastinski 2008) of participation may contribute to discerning which aspects of the collaboration influence satisfaction with the collaboration. Indeed, Hrastinski (2008) critically notes that the concept of participation encompasses additional dimensions that are not captured by word count alone, for example participating offline (reading literature, preparing texts) or making contributions of high quality. With our conceptualization, we were unable to detect differences between the conditions that existed on higher level conceptualizations of participation.

Besides this methodological discussion, we want to critically discuss the premise of equal participation more generally. As noted in the results section, students were rather satisfied with the collaboration, although only few groups achieved an equal distribution of participation. Above we hypothesized that students would not initiate regulation if they did not perceive the distribution of participation as uneven. Against this background, it would be important to investigate under which conditions ‘objectively’ uneven participation (or participation that is perceived as uneven) leads to sizeable dissatisfaction. While existing studies showed that unequal participation is a source of frustration (e.g., Aggarwal and O’Brien 2008; Capdeferro and Romero 2012; Goold et al. 2008), given our results we assume that there may be nuances (i.e., moderating and mediating variables) to the question under which circumstances of unequal distribution of participation leads to dissatisfaction and regulation processes. Existing studies provide first evidence regarding the context and consequences of social loafing. Unequal participation can be caused by some students contributing less than their peers. As research on social loafing has shown, team members interpret unequal participation differently and consequently react to it in different ways. For example, research has shown that students’ reactions depend on the perceived status of the loafing group member and on the perceived reason for the loafing behavior (Boren and Morales 2018), as well as on individual characteristics of the other group members such as conscientiousness (Schippers 2014). Depending on these factors, the remaining team members may rather compensate for missing contributions, instead of reducing their own efforts (cf. Harding 2018; Williams and Karau 1991) or even pressuring the ‘loafing’ group members to contribute (Boren and Morales 2018; Freeman and Greenacre 2011). Note here, that the results by Freeman and Greenacre (2011) suggest that students may not be sensitive to the difference between students who contribute less due to a lack of competence instead of a lack of effort. Thus, the remaining group members treated the ‘loafing’ members similarly.

On the flipside, an unequal distribution of participation can also result from individual group members contributing visibly more than their peers. Conceivable reasons for this can be interindividual differences in terms of dominance, personality factors, work ethics, motivation or resources such as the time available for collaboration. For example, Harding (2018) showed that forming groups with members that have similar levels of motivation to work on the task and have with similar time schedules reduces social loafing and increases performance in comparison to self-selected groups. Regarding power dynamics, Simpson et al. (2017) investigated power and status in co-located collaboration using positioning theory. They reported on two case studies where differences in power and status resulted in unequal interaction patterns. If individual students dominate the collaboration they position themselves in a way that leaves less room for the remaining team members to participate and for fruitful interaction to occur. This raises the question if all group members have to contribute the same amount of work (i.e., number of words) to be perceived as someone who ‘pulls their weight’ and also how different constellations affect satisfaction. For example, dominant students may feel less satisfied with equal participation as this opposes their position of power. In situations, where group members compensate for a high-status group member (Williams and Karau 1991), an uneven distribution may not affect satisfaction as much compared to situations where group members attribute social loafing to low ability (Freeman and Greenacre 2011). Likewise, depending on status differences in the group, some group members may have to work more than their peers to be perceived as equal parts of the team.

Conclusion and avenues for future research

Our study set out to investigate the effectiveness of combining two complementary means of collaboration support in the context of online collaboration. Despite the limitations, the present study makes three contributions. First, by investigating participation, satisfaction and interaction patterns, we have focused on two aspects of collaboration that are prerequisites for any beneficial interaction to occur. Secondly, we combined complementary means of computer-support and implemented them into an authentic online course in order to foster the equal distribution of participation and, by extension, satisfaction. With regard to supporting collaboration, our results help provide first evidence that groups who have access to group awareness information might require additional support to engage in socially shared regulation. In this regard, investigating students’ use and perception of the collaboration support may provide more valuable insights for further enhancing the effectiveness of the support. Finally, with regard to research methods, the present study introduced sequence alignment (Abbott and Tsay 2000) in conjunction with cluster analysis. The analyses reported in this paper build on prior work of Doberstein et al. (2017) (cf. also Hoppe et al. 2020) and we are confident that this combination of methods is a valuable addition to the CSCL researchers’ toolkit, especially if these methods are complemented with qualitative analyses. For research like the present study, qualitative analyses could be employed to shed more light into students’ goals during collaboration or how collaboration support affects their perception of participation in the group.

While our findings need to be further corroborated in studies that address the limitations discussed above, our results point to two broad avenues for future research on supporting collaboration. The first avenue for future research focuses on boundary conditions for the effectiveness of collaboration support and regulation of collaboration. We used the collaboration management cycle (Soller et al. 2005) as a theoretical basis for studying the effectiveness of support and regulation processes during collaboration. Our results suggest that this model oversimplifies regulation because it neglects how groups leverage collaboration support. As a cybernetic model, the collaboration management cycle conceptualizes a group as system that pursues an equilibrium and reacts to imbalances with regulation. Our findings suggest that what can be considered an ‘imbalance’ is not as straight forward as suggested by this model. For example, students’ goals, their perceptions of participation and their actual use of the support are also a part of the equation and affect the regulation process. Our results stress that students’ needs during collaboration, their appropriation, and their perceptions of the support should not be neglected (cf. Tchounikine 2016; provocation #6, Wise and Schwarz 2017). For educational practice, our results imply that educators should be cautious in assuming that learners use the support as intended by the designers.

Against the background of our initial research question, we hypothesize that groups might benefit from explicit support that fosters goal directed processing of group awareness information and socially shared regulation. As we have argued, the regulation of participation is complex. Nevertheless, our findings are also in line with the interpretation by Dehler et al. (2009). One way to support the processing of awareness information is implementing a mandatory, and guided co-reflection activity that asks students to view the GAT, reflect upon the information provided by the GAT, and collaboratively derive regulatory actions (e.g., Peñarroja et al. 2017; Phielix et al. 2011). Activities like this provide students with more space to externalize collaborative processes on a social plane and engage in socially shared regulation (cf., Järvelä et al. 2015). However, students who are more proficient in collaborating may need less external guidance than inexperienced learners. Thus, the design of collaboration support should take into account students’ internal collaboration scripts (e.g., Fischer et al. 2013) as well as their regulation strategies for collaboration (Hadwin et al. 2018) in order to tailor the prompts to the students’ needs.

The second avenue for future studies focuses on interaction patterns during web-based collaboration and their relationship with equal participation and satisfaction. Our analyses indicated that equality of participation and satisfaction were associated with interaction patterns that were not exclusively promoted by the group awareness tool or the adaptive prompts. Further, while not all groups achieved equal participation, students generally reported being satisfied with the collaboration. Therefore, future studies should not only include both, behavioral data and self-reports and develop a more holistic operationalization of participation, but should also challenge the premise of equal participation as a desired goal state of collaboration. In this regard, future research may for instance investigate interaction patterns and boundary conditions that affect equal participation and satisfaction with the collaboration. This is especially important, because research has shown that satisfaction is associated with a number of variables that are relevant for learning in remote online education. However, theoretical accounts of the role of satisfaction during collaborative learning are still scarce. Thus, we would like to encourage researchers to pool their expertise to shed more light on the role of this affective variable. For example, future studies may investigate the relationship between personality factors (such as the big-five personality factors, e.g. Ülke and Bilgiç 2011), motivational states (e.g., achievement motivation, Hart et al. 2004), affective states such as satisfaction, interaction patterns and results of the collaboration. Also, diving deeper into collaborative interaction and considering factors such as power, status, personality traits and individual resources may provide insights needed to understand web-based collaboration processes better. Our study considered the distribution of participation, which was a valid first step because without any written contribution there is no space for productive interaction. However, investigating the relationship to resulting group performance and individual learning is a necessary future step. Investigating interaction patterns and how these relate to cognitive, motivational or affective variables can provide new starting points for adapting existing types of collaboration support or developing new collaboration support.

Appendix

Table 14 Pool of six synonymous prompts