Improving learning and writing outcomes: Influence of cognitive and behavioral group awareness tools in wikis

Abstract

Group awareness (GA) tools can facilitate learning processes and outcomes by visualizing different social attributes, such as cognitive and behavioral information about group members. To assist learning and writing in social media, combining various types of awareness information may foster learning processes due to challenges, which are difficult to address by one type of GA information alone. The systematic investigation of GA tool combinations is largely unexplored with GA information often being examined separately or intermixed. To reveal both positive and negative (interaction) effects of providing different types of GA information, we conducted a 2 × 2 between-subjects experiment with N = 158 participants. Learners were provided with a wiki learning environment and, except for the control condition, different types of GA tools involving cognitive (knowledge bars) and/or behavioral (participation bars) GA information. GA tool effects were considered at wiki selection, discussion, and article levels. Eye-tracking was used for investigating the attentional effect of the GA visualizations. The results show that both types of GA information have effects on individuals’ selection preference, more strongly with the goal to learn new content than to support other wiki collaborators, which were introduced as within goal scenarios. Also, participants provided with behavioral GA support were more engaged in wiki contributions. However, only the combination of cognitive and behavioral GA information, rather than their separate visualization, had a positive effect on resulting article quality. This highlights the need for a holistic perspective when developing GA tools to improve wiki processes and outcomes.

Introduction

Wikis can support collaborative learning and writing in many ways. They enable users to work with each other on joint knowledge artifacts without time and place constraints (Chen et al., 2015). Furthermore, their usage involves both valuable cognitive and behavioral processes. Since wiki articles are often created collaboratively, a large knowledge base is depicted. Therefore, wiki content can itself serve as an extensive learning database. Although the potential for using wikis in learning scenarios is enormous, not every wiki has a primary learning focus. It is also possible to obtain wiki information solely for a specific problem or task. Moreover, it is necessary to encourage wiki users to actively produce content. This is not only important for collection purposes but can also have positive effects on cognitive processes, especially when change intentions are discussed in detail in the underlying wiki discussion forum (Heimbuch et al., 2018). Nevertheless, using such platforms also poses challenges (Hadwin et al., 2018), such as the facilitation of selecting meaningful content that is conducive to learning (cognitive challenge) and the stimulation of active content production/contributions (behavioral challenge). To overcome such challenges, we consider group awareness tools (GATs), which collect, transform, and present information about group members (Bodemer et al., 2018). GATs can provide learners with different types of group awareness (GA) information, with cognitive and behavioral GATs precisely addressing the two challenges mentioned above. Although there are already some promising findings for both types of GA information in relation to different learning processes and outcomes (e.g., Janssen et al., 2011; Sangin et al., 2011), the types are often examined separately or in combination without disentangling their specific functions (Bodemer et al., 2018; Janssen et al., 2011). Only few studies have investigated the differences and overlaps between different types of GA information as well as the effectiveness of GAT combinations (e.g., Lin et al., 2015). We aim to shed light on their interplay in a systematic way to better support learning processes in wikis and similar computer-supported collaborative learning (CSCL) platforms.

By using an experimental 2 × 2 between-subjects study and taking wiki environments as an example, this paper considers the linkage of cognitive and behavioral GA information and how the combination of both can be used to enhance single GAT outcomes. Additionally, we illuminate attention processes on a descriptive level by using eye-tracking to measure the extent of visual attraction of the designed GA support functions and other elements of the wiki environment. Our findings will help to better understand GA support and involved processes in CSCL for improving the instructional design of existing technologies.

Potentials of wiki-based CSCL

One essential characteristic of social media platforms, such as wikis, is that they promote user-generated content and social interaction among their users (Dabner, 2012). The popularity of such participatory technologies can be explained by using socio-constructivist approaches, which assume that individuals learn best when they have constructed knowledge through social interaction (Cole, 2009), something that is particularly promoted by wikis. This construction of knowledge is possible on the wiki article page as well as on the underlying discussion page. Thus, wikis offer considerable potential for valuable processes, both behavioral (content creation) and cognitive (individual learning/collaborative knowledge construction), to occur (Daspit & D’Souza, 2012).

Content creation is possible on several levels in wikis. Learners can create articles collaboratively, and they can interact with each other on the underlying discussion pages (Reinhardt, 2019). At the article level, writing can be seen as a distinctive set of monitored thinking processes, orchestrated by the user and based on self-generated goals. These include the planning, translating, and reviewing of texts (Flower & Hayes, 1981), which is considered a fundamental method to support learning growth (Tynjälä et al., 2001). At the discussion level, writing offers the opportunity to exchange views on what has been written and to clarify disagreements, making it possible for content-related controversies to occur. Such controversies are based on the exchange of different points of view on a specific topic. These might induce beneficial socio-cognitive conflicts between learners’ cognitive structures and the social system or wiki, leading to meaningful cognitive reorganization, restructuring, and consensus-finding processes (Bell, 2004).

According to Piaget (1977), defending or balancing different opinions is relevant because disequilibrium or cognitive conflicts have positive effects on learning growth. Although this theory was originally intended to describe individual learning processes, Piaget’s assumptions can also be applied to social interactions and collaborative knowledge construction within a wiki (Cress & Kimmerle, 2008). If wiki users encounter information that contrasts to their original assumptions, such a beneficial state of conflict can be triggered. Individual learning then takes place through internal assimilation or accommodation processes – more specifically, knowledge is transferred from the wiki into the user’s cognitive system by purely adding new information or changing existing knowledge. However, the user can also resolve this conflict status by external assimilation and accommodation processes. External assimilation refers to the simple addition of new information to the wiki, whereas external accommodation describes activities of rearranging entire wiki sections. Both internalization and externalization of wiki information leads to a co-evolution of the cognitive (individual learning) and social (collaborative knowledge construction) systems (Cress & Kimmerle, 2008). This highlights that beneficial cognitive and behavioral processes are closely intertwined in wiki-based learning.

Challenges of wiki-based CSCL

Even if wikis have huge potential for learners, this has not always been fully exploited. In the following subsections, we introduce the cognitive and behavioral challenges that learners face in their wiki’s intersubjective space.

Cognitive challenge

Although social interaction is conducive to learning, not all types of interaction have the same beneficial effects (Heimbuch & Bodemer, 2017). Therefore, the drive to select or focus on “relevant” and content-related discussions is important. Social interaction needs to be meaningful and highly qualitative, in the sense of wiki actors dealing with content that can lead to the aforementioned beneficial socio-cognitive conflicts (Ollesch et al., 2019). This could be content about which of the user has less knowledge than other collaborators or has a different point of view. Dealing with meaningful content can lead to relevant internalization or externalization processes, which is not always the case due to the huge amount of information offered by social media (Buder et al., 2015). It has already been shown that discussions in this context are frequently not deeply elaborated, instead simply skimming “over the surface” (Zhang & Zhang, 2010). Most wiki discussion pages (e.g., in MediaWiki) are empty pages lacking any pre-structure (Heimbuch & Bodemer, 2017), which makes it difficult to dive deeply into existing discussions without further structuring. This indicates that it makes sense to highlight cognitively relevant contributions to support the information search and navigation in discussion forums. Although selection processes can be described as a kind of behavioral process, the cognitive challenge is primarily concerned with decision-making prior to the actual action.

Behavioral challenge

In addition to meeting the cognitive challenge, all collaborators should be willing to actively engage or interact with each other in task fulfillment and communication processes (Ollesch et al., 2019). In wikis, the behavioral challenge involves both contributing to filling the article with content and creating this content collaboratively and with adequate discussion. For this purpose, the discussion forum is suitable for exchanging opinions on the article topic before actual editing (Heimbuch et al., 2018). However, there is not always the willingness to exchange opinions in such environments (Kimmerle & Cress, 2008; Kreijns et al., 2013). One reason is that the individual contributions in wikis are not directly visible, because the version history as well as the discussion page are hidden behind the actual article. As soon as individuals work toward a common goal and their individual performance becomes anonymous or not openly visible to all, their participation quantity is often reduced (Kimmerle & Cress, 2008), which can lead to social loafing and free riding (see Janssen et al., 2007). Moreover, social exchange arguably poses a social dilemma because it takes time and effort for individuals to share their knowledge, and they do not immediately benefit from this action. However, if no one contributes, the whole group is more likely to suffer (Cress & Kimmerle, 2007). This shows that it is important to overcome the behavioral challenge at both the article and discussion page levels. Our study focuses on supporting the exchange in wiki discussion forums before article editing. It has already been shown that social interaction before actual article editing can be beneficial for both behavioral and cognitive outcomes (Heimbuch et al., 2018), leading to articles with fewer mistakes and enhanced learning potentials.

Group awareness support

To stimulate social interactions in general (addressing the behavioral challenge) and in a meaningful way (addressing the cognitive challenge), wikis and similar platforms depend on some degree of support (Kreijns et al., 2013; Stahl & Hakkarainen, 2020). Here, we consider GATs as one solution to overcome the challenges mentioned above. These were highlighted by Rosé et al. (2019) as a helpful scaffolding method for collaborative learning. GA information is known to provide implicit guidance, which assists and improves self-regulated collaborative learning without giving explicit instructions regarding what to do (Miller & Hadwin, 2015). GA includes the knowledge about other group members’ knowledge, opinions, activities, or feelings. While this information is often highly inferable by learners in face-to-face situations, it is often not salient enough in computer-mediated learning scenarios (Wendt & Rockinson-Szapkiw, 2015), which highlights the need to support GA technically. With the help of GATs, the perception of GA can be supported. GATs provide learners with information they can use to form a group, to participate in effective collaborative learning procedures (Bodemer et al., 2018), to monitor their learning and collaboration behavior, and to change this behavior if necessary (Butler & Winne, 1995). GATs differ in the GA information they contain (Phielix et al., 2011). In the following, we discuss the effects of cognitive and behavioral GA information on cognitive and behavioral processes as well as on individual and collaborative outcomes.

Cognitive GA information: different levels of knowledge

Cognitive GATs are tools that provide learners with awareness information about the knowledge, interest, or opinions of group members and help to focus on what content is meaningful, thereby mainly stimulating qualitative cognitive processes (addressing the cognitive challenge; Bodemer et al., 2018; Lin et al., 2015). They allow an individual’s knowledge or opinions to be compared with those of other group members, thus promoting better grounding and partner modeling processes (Bodemer, 2011; Sangin et al., 2011). Especially regarding knowledge, it has been shown that the expertise of others is often over- or underestimated without any kind of support and that there is a need to support learners in the construction of accurate partner models (Dehler et al., 2011). An example of a cognitive GAT is the knowledge awareness tool of Sangin et al. (2011), which provides learning partners with information about each other’s knowledge differences (in the form of bar charts) based on a pretest score. It has been shown that this tool allows for a better assessment of the learning partners’ knowledge (Sangin et al., 2011).

Selection preference based on learning and supporting goals.

Studies in the social media field have already shown that the provision of cognitive GA information in the form of rating visualizations can facilitate navigation in large online forums and help to find high-quality contributions for learning purposes (e.g., Buder et al., 2015). Moreover, learners who are provided with controversy awareness markers in wikis direct their attention toward the selection of unresolved controversies for supporting purposes, leading to the potential for socio-cognitive conflicts to occur (Heimbuch & Bodemer, 2017). Although there can be a variety of intentions for actions in wiki learning environments, we count the aforementioned learning and supporting purposes as the main goals that drive learners in such environments. Even though not yet examined in the social media context and for knowledge awareness, we assume that there are also positive effects on discussion thread selections in wiki environments when GATs visualize different levels of involved user knowledge. When the main goal is to learn new content, learners potentially select wiki discussion threads that visualize a high level of group knowledge rather than a medium or low level, because of the higher chance for finding helpful contributions. However, when the main goal is to support other collaborators, they potentially tend to select wiki threads that visualize low group knowledge among the participants rather than a medium or high level.

Engagement in behavioral contributions.

Cognitive GA information stimulates not only cognitive processes but also behavioral processes like the formulation of messages (Krauss & Fussell, 1991). On the one hand, there is the possibility of explaining and discussing information that is not known by the learning partner. On the other hand, there is the chance of closing knowledge gaps with the help of experts (Dehler et al., 2011). It has been shown that the visualization of partner deficits can motivate learners to contribute more in terms of explanations (addressing the behavioral challenge; Dehler et al., 2011), whereas experts may also strategically withhold information (Ray et al., 2013). Therefore, cognitive GA information can trigger calculated supporting behavior, although this has yet to be confirmed across different contexts.

Outcome variables.

According to Bodemer et al. (2018), cognitive GATs show positive effects on both internalization and externalization tasks as outcome variables. These range from objective measures, such as individual knowledge gains (e.g., Gijlers & de Jong, 2009) and collaborative wiki article quality (Heimbuch et al., 2018), to subjective measures like partner knowledge assessment (e.g., Sangin et al., 2011). However, desirable effects on outcome variables cannot be shown across all studies in this field, especially in relation to the findings on learning outcomes (Bodemer et al., 2018).

Behavioral GA information: different levels of participation

Even if wiki participants perceive that an action is expected, there are often no clear standards of what this effort should look like. Behavioral GATs provide such awareness information by visualizing the activities of the group and its members (Kimmerle & Cress, 2008), e.g., the number of contributions to online discussions (Bodemer et al., 2018). When group members are aware of one another’s productivity rates during learning, they are able to assess how much content is desirable, which mainly stimulates quantitative behavioral processes (addressing the behavioral challenge; Lin et al., 2015). This is even the case when people get bogus feedback (given information without real persons involved) about teammates’ high or low contribution rates (Cress & Kimmerle, 2007). A prominent example of a behavioral GAT is Janssen et al.’s (2011) participation tool, which represents each group member as a circle whose size and distance from a group center varies. The larger the circle, the more has been contributed. Results show that learners who used the tool for a longer period were more involved in the discussions and the work on the group project.

Selection preference based on learning and supporting goals.

Although there is still little research on how the visualization of behavioral GA information influences selection decisions (addressing the cognitive challenge; Ollesch et al., 2019), analogous to cognitive GA information, it might be concluded that when the main goal is to learn new content, learners potentially select wiki threads that visualize a high level of group participation rather than a medium or low level due to a potential higher amount of available learning material. However, when the main goal is to support other collaborators, they rather select wiki threads visualizing low group participation rather than a medium or low level due to the lack of available contributions regarding a specific learning topic.

Engagement in behavioral contributions.

Being mutually aware of contribution rates stimulates feedback and motivational processes because learners want to avoid negative social evaluations by the group (Janssen et al., 2011). In particular, descriptive social norms play a crucial role in the effectiveness of behavioral GA information on contribution engagement because they motivate actions by specifying what most people do in a particular situation (Reno et al., 1993). It is assumed that actual participation is determined by such norms and that people adapt their behavior by observing others (Ajzen, 2015), an assumption that can be transferred to the observation of behavioral GAT information about other people. Crucial to the effectiveness of behavioral GA information in information-exchange situations is not only the knowledge about others’ behaviors but also the identifiability of one’s own effort, because all people have a need to present themselves in a certain way (e.g., as a very productive person; Heo et al., 2019; Kimmerle & Cress, 2008). According to Cress and Kimmerle (2007), the self is especially strategically presented in online CSCL settings, as people have more control than in offline settings. People use the technological characteristics of such environments with the goal of managing the impressions they give to other people (Walther, 2007, as cited in Kimmerle & Cress, 2008). This highlights that cognitive processes are also involved in the effects of behavioral GA information.

Outcome variables.

According to Bodemer et al. (2018), behavioral GATs have mixed results. Behavioral GATs mostly (not exclusively) show positive effects on externalization outcomes, which range from objective measures, such as the quality of collaborative writing products (e.g., Liu et al., 2018), to subjective productivity assessments (e.g., Phielix et al., 2011). However, results on the quality of collaborative group performances are not consistent across studies (e.g., Janssen et al., 2011; Lin et al., 2019). In addition, internalization outcomes, such as learning performance, are often not explicitly addressed in most papers on behavioral GA support or rarely present (Bodemer et al., 2018).

Personal characteristics influencing GAT effects

Although GA support has been proven to be beneficial, it is important to focus not only on the objective information given by a tool because learners may act differently, even with identical technology support (Tchounikine, 2019). Personal characteristics are currently rarely considered in the GAT context and could provide the basis for personalized GA support approaches. It may be that individuals, depending on their personal characteristics, use one type of information more heavily than the other type during the collaboration. Here, we introduce possible influencing variables, since it cannot be assumed that different GATs have the same effect on all types of learners.

One potential influence on the processing of both types of GA information is social comparison, which helps individuals to compare their own opinions and abilities with those of others in order to maintain an adequate and stable self-image (Festinger, 1954). Once learners work collaboratively on a task, they can become aware of the skills and activities of their learning partners. This promotes processes of social comparison and helps them to gain information about themselves and their own performance (Neugebauer et al., 2016). GATs can further foster this by visualizing awareness information about others (Bodemer et al., 2018), such as their knowledge level (cognitive GA information) or participation amount (behavioral GA information). However, social comparison is not used by everyone to the same extent, with some relying more on it than others. Gibbons and Buunk (1999) termed this predisposition social comparison orientation. People with a higher tendency toward social comparison are more interested in information that allows comparison with interaction partners, enhancing GAT effects (Kimmerle & Cress, 2009; Neugebauer et al., 2016; Ollesch et al., 2020).

Besides the tendency toward social comparisons, the construct need for cognition (NFC) is a potentially relevant personality trait affecting information processing in wiki environments, especially with regard to cognitive GA information. One prerequisite for effective cognitive GA support is that learners need to be willing to engage in valuable cognitive processing activities, as described by the NFC (Cacioppo & Petty, 1982). NFC is considered a stable individual difference in people’s tendency to engage in and enjoy effortful cognitive activity (Cacioppo et al., 1996). Cohen et al. (1955) defined the NFC construct as “a need to structure relevant situations in meaningful, integrated ways. It is a need to understand and make reasonable the experiential world” (p. 291). It can thus be assumed that NFC determines the perception of a learning environment and the setting of learning goals. Cognitive GA information helps people with different amounts of NFC to distinguish between content with different cognitive requirements. Based on this, learners with a high NFC can elaborate presented GA information and wiki content more thoroughly, whereas learners with low NFC might avoid effortful thinking. It can be assumed that people with low NFC try to avoid cognitively demanding tasks, whereas people with high NFC consistently engage in and enjoy cognitively challenging activities (Petty et al., 2009) or deep learning activities in general (Cazan & Indreica, 2014). However, both selecting wiki discussion threads involving people with high knowledge and low knowledge could be considered challenging, so the direction of these effects is not clear here. On the one hand, choosing threads with GA information visualizing high group knowledge means that more cognitive effort is needed for potentially highly qualitative group discussions, which is in line with high NFC (Ollesch et al., 2020). On the other hand, people with high NFC should not be inhibited by content that implies cognitively challenging issues and therefore potentially prefer threads with GA information that visualizes a lower group knowledge level in comparison to people with lower NFC (see Cacioppo et al., 1996).

Potential interplay of cognitive and behavioral GATs

Both cognitive and behavioral GATs have shown promising results on both cognitive and behavioral processes and outcomes in isolation. However, the separate provision of these types of GA information does not always lead to an improvement regarding cognitive (e.g., learning performance) and behavioral outcomes (e.g., wiki article quality), even though these are crucial dependent variables when it comes to measuring the success of GAT support in CSCL settings. A possible explanation for this is that both types of information are more likely to unfold their full effect when combined. Single GA indicators may not be able to address all the challenges that learners face in high-level CSCL settings to the fullest extent. While behavioral GA information mainly has the potential to heighten social interaction in general (addressing the behavioral challenge and quantitative behavioral processes), cognitive GA information has predominantly positive effects on content-related decisions and interactions (addressing the cognitive challenge and qualitative cognitive processes), which we concretize in the following.

Behavioral GA information could be beneficial for strengthening the effects of cognitive GA information because it is known to trigger intensified social interaction, which cannot be taken for granted but is seen as a fundamental prerequisite to knowledge construction by socio-constructivist approaches (Bento & Schuster, 2003; Scardamalia & Bereiter, 1994). For high-order thinking skills, active participation or externalization of knowledge is indispensable (Daspit & D’Souza, 2012; Galikyan & Admiraal, 2019). It has already been shown empirically that collaborative writing processes, including active interaction patterns, generate higher-quality texts, can have a positive effect on learning performances (Liou & Lee, 2011; Liu et al., 2018), and help to develop better writing and language skills (Oh, 2014; Storch, 2011). The importance of such behavioral processes in knowledge construction illustrates the potential for using behavioral GA information to trigger behavioral engagement as a supplement to cognitive GA information.

Although behavioral processes in general can stimulate cognitive processes, simple participation does not automatically guarantee the emergence of valuable and highly qualitative cognition as assumed by Piaget (1977) or by the co-evolution model (Cress & Kimmerle, 2008). Rather, interaction frequently refers only to emotional instead of cognitive exchange (Garrison et al., 2000). However, group collaboration requires both behavioral and deep cognitive engagement. Instructional design should therefore also strengthen valuable cognitive processes, such as by using cognitive GA information (Daspit & D’Souza, 2012; Ma et al., 2020). This is very promising because specific individual and group knowledge levels, highlighted by cognitive GA information, might influence the direction of social interaction and help to steer goal setting for wiki writing processes in a certain desirable direction (see Dehler et al., 2011). This might involve discussions with interaction partners who have a higher level of knowledge that is challenging but still within the user’s zone of proximal development (see Vygotsky, 1978). The importance of cognitive processes in content and discourse orientation illustrates the potential for using cognitive GA information to trigger cognitive engagement as a supplement for behavioral GA information.

Each form of GA information potentially benefits from the interaction with the other one, since the previously outlined learning processes are essential but cannot be triggered equally by single behavioral or cognitive GA information. Therefore, in studies without or with only slight GAT effects, providing one type of GA information might not have been enough to make learners aware of how much and what kind of content is desirable. We argue that even in studies where such effects have been shown, these could potentially have been enhanced by GAT combinations. We assume that cognitive and behavioral GA information could complement each other positively when presented simultaneously to fulfill the aforementioned wiki potentials and enhance expected outcomes. This is in line with the research of Phielix et al. (2011), who considered and examined the effects of combined GA information, but using a full-tool versus no-tool examination strategy as well as a small group context. For the next step of understanding the behavioral and cognitive functions of GATs and of empowering wiki-based CSCL with GATs, it is essential to systematically disassemble and compare separate and combined effects of cognitive and behavioral GA information. This will help to identify positive, as well as potentially negative, (interaction) effects (see also the review by Ghadirian et al., 2016). Accordingly, hypotheses based on the previous theoretical considerations are derived and presented in the next section.

Hypotheses

Hypothesized single effects of GATs on selection decision (cognitive challenge)

With regard to the cognitive challenge and earlier empirical results, cognitive GA information and especially group knowledge awareness leads to improved topic selection in different contexts (e.g., Dehler et al., 2011; Heimbuch & Bodemer, 2017). We also expect that different levels of group participation influence wiki selection, depending on the individual’s intentions. Above, we defined two selection goals that should be considered as the main drivers for actions in a wiki learning environment: the goal to learn new content and the goal to support other collaborators. Based on these two selection goals, we assume that people with GA support are more focused on selecting threads with a clearer need for action – involving high and low group knowledge and participation levels – rather than medium levels. The following hypotheses are proposed:

H1.1: With the goal to learn new content, individuals who are supported by cognitive GA information select wiki threads presenting high group knowledge, whereas with the goal to support other collaborators, wiki threads with low group knowledge are preferred.

H1.2: With the goal to learn new content, individuals who are supported by behavioral GA information select wiki threads presenting high group participation, whereas with the goal to support other collaborators, wiki threads with low group participation are preferred.

As outlined in the section on personality influences, a high NFC can potentially be satisfied in group constellations involving both very high and low group knowledge (see Cazan & Indreica, 2014; Cacioppo et al., 1996). Therefore, for H2, we focus on potential moderation effects related to high and low group knowledge awareness presentations, this time without distinguishing between specific selection goals. We derived the following two-sided hypothesis on the influence of NFC on the effects of cognitive GA information:

H2: The effect of cognitive GA information on the individuals’ selection of wiki threads presenting (a) high and (b) low group knowledge levels is moderated by NFC.

Hypothesized single effects of GATs on contributing (behavioral challenge)

With regard to the behavioral challenge that has not yet been examined in the social media or wiki context, research suggests that information about the participation amount of other wiki users leads to an alignment of discussion contributions when these are salient on the individual and group level (e.g., Kimmerle & Cress, 2008). We also assume certain effects of cognitive GA information on behavioral engagement, depending on the individual’s self-perception as more knowledgeable, which can be beneficial (Dehler et al., 2011) but potentially also non-beneficial (Ray et al., 2013) for behavioural outcomes, leading to the following hypotheses:

H3.1: Individuals who are supported by behavioral GA information match their own number of written discussion contributions to a presented behavioral group mean in comparison to individuals without behavioral GA support.

H3.2: With cognitive GA support, there is a correlation between the individual’s presented knowledge and the individual’s number of written discussion contributions.

Moreover, it has already been shown that a high need to engage in social comparison can enhance GAT effects on behavioral engagement of learners (Kimmerle & Cress, 2009; Neugebauer et al., 2016). Therefore, we propose:

H4: The effects of (a) cognitive GA information and (b) behavioral GA information on the individual’s number of discussion contributions are reinforced by the tendency toward social comparison.

Hypothesized interaction effects of GATs on learning and wiki article quality

As stated above, behavioral and cognitive processes can support each other in collaborative learning settings, which suggests that the interaction of behavioral and cognitive GA guidance, potentially triggering the same processes, should be crucial for increasing learning outcomes and wiki group products, in comparison to providing only one type of support. This goes beyond the simple addition of cognitive and behavioral GA information. We are mainly interested in the interaction effects of both types of information, since our assumption is that they unfold their effects disproportionately or fully if they are presented in combination:

H5: Individuals who are supported by cognitive and behavioral GA information achieve a better learning performance about the wiki topic than individuals who are supported by only one type of GA information.

H6: Individuals who are supported by cognitive and behavioral GA information achieve a higher wiki article quality than individuals who are supported by only one type of GA information.

Exploratory investigations

This section explains the variables collected in the exploratory investigations. Regarding H5 and H6, it could be argued that the joint processing of cognitive and behavioral GA information may be mentally exhausting (see Janssen et al., 2011) and thus cancels out the assumed positive interaction effects of GAT combinations in H5 and H6, especially when they contain subjectively unnecessary information (Janssen & Kirschner, 2020). However, we assume that the positive interaction effects clearly outweigh the potential negative interaction effects based on the presented empirical and theoretical relevance of cognitive and behavioral GA information. To check this, we controlled for the cognitive variable mental effort to exclude this as a confounding variable. Moreover, despite focusing mainly on cognitive and behavioral outcomes in this research, we are also interested in whether cognitive and behavioral GAT combinations have a potential positive interaction effect on emotional outcomes, such as the well-being of the group, because emotions are part of cognitive and behavioral processes and relevant for intended technology use (Polo et al., 2016).

Methods

Research design

We used a 2 × 2 (cognitive GA information: present vs. absent; behavioral GA information: present vs. absent) factorial between-subjects design to test the hypotheses. There were four experimental conditions: a group without GA support; two groups with either cognitive or behavioral GA support; and a group receiving both. The study was approved by the ethics committee.

Participants

N = 158 German participants (116 females, rest males; M age = 22.89, SD = 4.90, 17–59 years) took part in the study, which lasted 1 to 1.5 h. Most participants (93%) were students in study programs unrelated to the topic, with the rest being in employment. Some participants were randomly assigned to the control condition (n = 38), others were assigned to the cognitive (n = 41) and behavioral GA (n = 41) conditions, while the rest were assigned to the combination condition (n = 38).

Learning environment

Learners were provided with a self-designed wiki environment. We initiated an objective work-in-progress wiki article on energy sources. Participants were also provided with a wiki discussion forum, including additional arguments for and against different types of energy (e.g., solar energy, wind energy, etc.), given by a bogus group. In each thread of the discussion forum, two bogus collaborators were involved: one person in favor of the respective energy form and another person against the respective energy form. GA information was presented as horizontal bars in the discussion forum in the header (individual and whole group level) as well as next to the 12 discussion threads (subgroup level). Figure 1 shows the wiki discussion forum for the combination condition, which was provided with cognitive GA information (yellow knowledge bars) and behavioral GA information (blue participation bars). To eliminate sequence effects, the order of individual and group knowledge/participation bars was horizontally randomized across participants. We used “standard colors”, which are not especially eye-catching (Few, 2006). The wiki article contained no GA visualizations.

Fig. 1

Extract from the discussion forum with individual and whole group GAT levels in the header and subgroup GAT levels next to the discussion threads (partially translated in English)

Presentation of cognitive and behavioral GA information on individual level

Concerning individual level awareness information, the participants’ individual GA information was presented in the header. This “You”-row was initially without an entry but was measured and adjusted accurately during the course of the study, leading to naturally occurring differences between participants.

The individual knowledge bar was adjusted after taking part in a knowledge pretest. In order to make the pretest neither too difficult nor too easy, a pilot study was conducted in advance, which comprised N = 55 participants (37 females, 16 males, 2 diverse; M age = 28.42, SD = 8.40, 20–62 years). The aim was to select suitable items of medium difficulty from a predefined set of 25 single-choice items on energy sources with four answer options each. Most participants (55%) were students, the others were in employment. As criteria for the item selection, the items should result in a score of ≈ 5 and should cover a wide range of the energy topics. This resulted in 10 items for which 10 points could be achieved, with a mean test score of M = 5.29 (SD = 1.57). Hence, there was a 10-step knowledge scale. Those items were mainly fact-based – that is, they asked for facts, such as “What is the efficiency of hydropower?”. An example for an adjusted individual knowledge bar with a final score of 7 points (= 7 correct answers) can be seen in the header’s “You”-row in Fig. 1.

The individual participation counter was adapted dynamically during the collaboration. It went up by one every time a new discussion contribution was written. An example for an adjusted individual participation bar after one discussion contribution can be seen in the header’s “You”-row in Fig. 1.

Presentation of cognitive and behavioral GA information on group level

Even if the participants were not informed about it, the different levels of cognitive and behavioral GA information at whole and subgroup level were experimentally varied before the experiment started. For reasons of complexity, a distinction was only made between discussion threads visualizing low (counter on 2), medium (counter on 5), and high (counter on 8) subgroup average knowledge and/or participation. Concerning group level awareness information, the knowledge bar in the header next to "Group" represented the average overall group knowledge of the whole group involved in the wiki environment/discussion forum, based on the total score of the aforementioned knowledge pretest. The participation bar in the header next to "Group" represented the average group contribution frequency of the whole group in the discussion forum in an aggregated form. Below, next to the respective discussion threads, the same visualizations were shown in a non-aggregated form. These GA visualizations referred to subgroups of the whole group and indicated the extent of participation and knowledge of the respective group members involved in threads like “Windenergie” (wind energy), in this case with low subgroup knowledge and medium subgroup participation (see Fig. 1).

Regarding the different levels of knowledge, discussion threads differed in terms of the bogus contributors’ own expertise assessment and the number of sources they used. In threads with low subgroup knowledge (small knowledge bar), collaborators were novices and did not use any sources (e.g., “[…] I am not an expert, but for me, the potential negative effects on people and the environment speak against the use of wind energy. […]”). In threads visualizing medium subgroup knowledge (medium knowledge bar), collaborators were very well-read or interested in the area and used only one source each (e.g., […] “I have read quite a bit on the subject of biofuels. According to Source x, biofuels have so far proven to be very inefficient and also represent for only about 10 percent of mankind an important source of energy. […]”). In threads with high subgroup knowledge (large knowledge bar), both collaborators indicated that they worked in a relevant area of the energy industry and used two sources each (e.g., […] I work for a large energy company and think we should emphasize the good availability of silicon. It is the preferred material for crystalline solar cells according to Source a. According to Source b, the earth's crust consists of about 25.8 percent silicon […]”). None of the threads contained false or irrelevant information. Participants were told that this knowledge variation at the group level was based on the total score of the 10-items pretest, which they also took part in before the wiki collaboration.

Regarding the different levels of participation, discussion threads differed in terms of the number of contributions by the bogus collaborators. There were threads with low average subgroup participation (2 discussion contributions), medium average subgroup participation (5 discussion contributions), and high average subgroup participation (8 discussion contributions). In order to keep the visualization of knowledge and participation GA information equal, a scale of 0-10 was chosen here as well. Participants were informed that the group member who contributed most had made 10 contributions (which was actually possible, derived from pilot testing).

Editing phases

The editing consisted of three phases, which all participants went through, regardless of the respective assigned condition. GAT effects were examined systematically at selection, discussion, and article levels (see Fig. 2). In the selection phase, participants were provided with the discussion forum (see Fig. 1). Only the wiki topics and the GA information were visible, depending on the different conditions. The actual discussion content was not yet revealed. For participants in the cognitive GAT condition, their individual knowledge bar was adjusted in this phase based on the pretest. As assessment of the participants’ intention behind specific thread selections (see H1.1 and H1.2) was difficult later in the study, the selection tendencies were examined in detail during the selection phase. This phase consisted of two selection tasks (STs), queried in succession, in which selection tendencies were systematically examined for learning and supporting goals. All participants were asked to choose the three preferred threads (out of 12) they would most likely select based on the two subsequent scenarios. In ST1, they were asked to engage in using the wiki discussion forum with the goal to learn new content. In ST2, they were asked to engage in using the discussion forum with the goal to support other collaborators in solving their controversies.

Fig. 2

Procedure of the editing phases for all participants (within steps)

The second 20-min discussion phase was about interacting with the discussion page. In this phase, participants were free only to read the content and learn from it or to write their own contributions to the discussion forum to solve bogus collaborators’ controversies. They were not instructed to support others or share their knowledge. Discussion contributions were restricted to contain a word count of 30–100 words in order to be sent. For participants in the behavioral GA condition, their individual participation bar was updated in this phase. In the last 10-min phase, the wiki article should be supplemented with arguments from the discussion forum. This article phase contained only the wiki article rather than any kind of GA information. Kimmerle et al. (2017) recently identified different stages of collaborative activities during text development, starting with individual knowledge sharing, leading to a mutual restructuring phase, and finally the expression of a shared opinion. Since our work is primarily concerned with the contribution that the individual makes to the group and no actual reaction occurs, our research can be located in the knowledge-sharing phase.

Measures

To answer H1.1 and H1.2 (concerning selection preferences), we measured the proportion of selected threads presenting high, medium, and low group knowledge, as well as presenting high, medium, and low group participation, relative to the total number of threads in both STs (goal to learn new content and goal to support other collaborators). This resulted in 12 variables (six variables each ST).

To measure NFC (H2a and H2b), the NFC subscale of the German Rational-Experiential Inventory (Epstein et al., 1996) by Keller et al. (2000) was used (α = 0.87; 14 items measured on a seven-point Likert scale, ranging from 1 = “completely incorrect” to 7 = “completely true”). An example item is: “I would rather solve a task that requires intelligence, is difficult and significant than a task that is somehow important but does not require much thought.” Two mean values for thread preferences were determined from STs 1 (goal to learn new content) and 2 (goal to support other collaborators): one for the selection preference of high group knowledge, the other for the selection preference of low group knowledge across both STs.

To answer H3.1 and H3.2 (concerning the number of discussion posts), the number of total discussion contributions was counted.

The tendency toward social comparison (H4a und H4b, concerning personal influences) was surveyed with the German version of the Iowa-Netherlands Comparison Orientation Measure (COM; Jonas & Huguet, 2008) (α = 0.84; 11 items answered on a seven-point Likert scale, ranging from -3 = “I don’t agree at all” to 3 = “I totally agree”). An example item is: “I always pay a lot of attention to how I do things compared to others.”

To answer H5 (concerning learning-related variables), we used a posttest consisting of 24 single-choice items: 10 fact-based items were identical to the pretest, to which we added 14 argument-based items. The latter were mainly related to pro and contra arguments regarding different energy types, derived from the discussion forum, such as “What are the arguments against using biofuels?”. A total sum score was calculated for the whole posttest with the number of correctly answered questions and the learning gain for the 10 identical fact-based items of the pretest and posttest.

To answer H6 (concerning article quality), we analyzed the word count, the argument count, and the balance of new pro and contra arguments in the article. Since the task was to add new content from the discussion forum to the article, it was assumed that more content or a higher article productivity also implied higher quality. Arguments were considered as new if they contained new and correct information not yet included in the article, such as “An argument against the use of water for energy production is that it would involve interfering with nature, such as by constructing reservoirs, which would affect humans and animals.”. Eight overall categories were defined as being analogous to the respective topics contained in the text: fossil energy, nuclear energy, renewable energy, wind energy, hydro energy, solar energy, biofuels, and geothermal energy. The following subcategories were created for each topic: contra argument, pro argument, and additional information. The latter was almost never the case, as it was the participants’ task to complement the article with pro and contra arguments from the discussion forum. One coder encoded all records, and 10% of records were initially counter-encoded by a second coder with an almost perfect interrater reliability of Cohen’s κ = 0.83. After a discussion among the researchers, the rest were further finalized by the first coder. To calculate the argument balance, two categories were created. Code 1 was assigned when there was at least one pro and one contra argument for a topic; otherwise, a code of 0 was assigned. The sum score of all codes was then divided by all thread topics contained in the article to evaluate whether people wrote balanced articles on average.

Mental effort was measured with a German translation of Paas’s (1992) one-item scale (“In solving or studying the preceding problem I invested”; 1 = “very, very low mental effort” to 9 = “very, very high mental effort”). Well-being was measured with the German version of Lang and Bachinger’s (2017) Well-being scale (α = 0.89; 14 items answered on a five-point Likert scale, ranging from 1 = “none of the time” to 5 = “all the time”). An example item is: “I’ve been feeling good about myself.” The scores for COM, NFC, and well-being were calculated from the corresponding questions in the questionnaire. In the case of mental effort, we calculated an overall mean score from three measuring points (α = 0.76).

Procedure

Before the study began, participants received information about it and signed a consent form. During the study, participants were provided with a scenario about working together with other people on a wiki article on energy sources, which would be evaluated and also serve as a test preparation. Participants who were in the different GA conditions additionally received an explanation of the function of the knowledge and/or participation bars, but without being explicitly told how to deal with such GA information. The main procedure was as follows:

(1) After an initial five-minute wiki article reading phase, (2) all participants completed the aforementioned 10-items pretest. (3) Then, the actual editing commenced, consisting of the three phases (see Fig. 2). (3.1) In the selection phase, all participants went through the two STs (with a goal to learn new content and a goal to support other collaborators). For participants supported with cognitive GA information (cognitive GA and combination conditions), their individual knowledge bar was adjusted at this point based on the score they achieved. (3.2) In the discussion phase, some participants (behavioral GA and combination conditions) were provided with an adaptive individual participation bar. In this phase, there was no specific focus on whether cognitive (reading) or behavioral activities (contributing) were preferred. (3.3) In the article phase, the wiki article on energy sources was edited. Mental effort (Paas, 1992) was measured directly after each editing phase. After the editing phases, all respondents filled out the well-being scale (Lang & Bachinger, 2017), the posttest, the COM (Jonas & Huguet, 2008), and the NFC scale (Keller et al., 2000), before being debriefed.

Eye-tracking investigation

We were interested in the participants’ visual attention on different elements in our learning environment, because this provides an insight into participants’ consideration of our GA visualizations and further underlying attentional processes. Visual attention is understood as the process of selecting and concentrating on stimuli of interest, and hence the processing of characteristics to which attention is paid more deeply (Goldstein, 2008). This is relevant as people never perceive the whole scene or learning environment; rather, only a fraction of the available visual information is processed. The actual processing of visual information only takes place when there is a persistent fixation – in other words, when the gaze remains in a certain position for a quarter or half a second. Saccades represent the eye movements or “jumps” between fixations (Manhartsberger & Zellhofer, 2005). Since humans have only a limited processing capacity, attention processes are directed to subjectively important aspects, something we wanted to disclose for our setting.

To investigate the impact of cognitive and behavioral GA visualizations on visual attention, a further N = 20 participants (11 females, M age = 22.45, SD = 3.59, 18–30 years) were recruited (five participants for each condition). The number was small because of the limited eye-tracking equipment. Moreover, via visual observations, we found that participants’ behavior was inhibited by the eye-tracking procedure. They received further eye-tracking-related instructions, which potentially interfered with their natural interaction behavior. The distribution of attention in the learning environment was recorded and examined via Tobii Pro Spectrum. Participants were positioned at a distance of about 70 cm from the screen and were instructed to move as little as possible during editing. To increase recording accuracy, eye movements were calibrated with a nine-point calibration technique before the editing phases. The aim of this was to adjust the eye tracker to the participants’ eyes. Lighting conditions were kept as similar as possible for each participant.

Results

For H1.1 and H1.2, we used ordinal regression analyses because of the ordinal selection variables. For H2a/H2b and H4a/H4b, we used the PROCESS macro v3.4 for SPSS to run moderation analyses. Extreme outliers (three interquartile distances away from the upper/lower quartile) were removed in all regression-based procedures. We mainly calculated 2 × 2 ANOVAs (H5, H6, and exploratory calculations on mental effort and well-being) to check for significant interaction terms (involving the factors cognitive GA and behavioral GA information). For H3.1, we were particularly interested in the behavioral GA information effects. In most cases, a normal distribution was not given for the dependent variables of the ANOVAs (Shapiro–Wilk test; p < 0.05), and bootstrapping (10,000 samples) was applied. To test H3.2, a bivariate correlation was applied. A significance level of α = 0.05 was defined for all procedures. Tests for directional hypotheses were one-sided.

Impact on selection (hypotheses 1 and 2)

In H1.1 and H1.2, we assumed that the availability of GA information has an impact on the preference of threads visualizing different levels of GA information, depending on the selection goal (variables derived from ST1 and ST2). To check if there was a difference between persons with respective cognitive or behavioral GA support and those without GA support, we calculated ordinal regression analyses with the factors cognitive or behavioral GA information (single factor models). To test the hypotheses, we report the following selection effects: cognitive GAT effects on the selection of high/medium/low group knowledge wiki threads in comparison to no cognitive GA support (H1.1), and behavioral GAT effects on the selection of high/medium/low group participation wiki threads in comparison to no behavioral GA support (H1.2). In total, we conducted 12 regression analyses (support conditions were used as a baseline here), because there were two STs (goal to learn new content and goal to support other collaborators) and six dependent variables. The χ2 tests for the whole model fits of all regression analyses (step 1) are reported in Table 1.

Table 1 χ2 tests related to differences in selection preference of threads involving low, medium, and high group knowledge (k.) and participation (p.)

In the following (step 2), we provide only the parameter estimates (b) for ordinal regression analyses showing a significant χ2 test (see Table 1) to determine the direction of effects. In H1.1, we assumed that the selection frequency of wiki threads presenting different group knowledge levels is influenced by the provision of cognitive GA information. Ordinal regression analysis shows that when the goal was to learn new content (ST1), learners supported with cognitive GA information were more likely to select threads with GA visualizing high group knowledge (Wald’s χ2(1) = 26.56, p < 0.001, b = 1.71, Exp(B) = 5.54) and less likely to select threads with GA information visualizing low group knowledge (Wald’s χ2(1) = 13.13, p < 0.001, b = -1.29, Exp(B) = 0.28) in comparison to learners without cognitive GA support. They were also less likely to choose threads with medium group knowledge in comparison to participants without cognitive GA support (Wald’s χ2(1) = 14.75, p < 0.001, b = -1.28, Exp(B) = 0.28). However, when the goal was to support other collaborators (ST2), the provision of cognitive GA information did not make a significant difference regarding the selection of threads with GA visualizing high group knowledge or with GA information visualizing low group knowledge in comparison to learners without cognitive GA support (see Table 1). On the other hand, there was a significant effect on the number of selected threads with GA visualizing medium group knowledge. Here, participants with GA support were less likely to choose threads with medium group knowledge in comparison to participants without cognitive GA support (Wald’s χ2(1) = 7.20, p = 0.007, b = -0.96, Exp(B) = 0.38). In summary, it can be stated that the provision of cognitive GA information had an effect on selection (clearer in ST1, goal to learn new content). Participants with cognitive GA support were more focused on “relevant” or high GAT levels with a learning goal (ST1), as stated in H1.1.

In H1.2, we assumed that the selection frequency of wiki threads with different group participation levels is influenced by the provision of behavioral GA information. Ordinal regression analysis shows that when the goal was to learn new content (ST1), the provision of behavioral GA information made a significant difference regarding the selection of threads with GA visualizing high group participation, in favor of the behavioral GA support condition (Wald’s χ2(1) = 6.34, p = 0.012, b = 0.81, Exp(B) = 2.24), as well as threads with GA information visualizing low group participation, here in favor of the control condition (Wald’s χ2(1) = 5.26, p = 0.022, b = -0.71, Exp(B) = 0.49). However, there was no effect on threads with GA visualizing medium group participation (see Table 1). When the goal was to support other collaborators (ST2), the provision of behavioral GA information made a significant difference regarding the preference of medium group participation threads, in favor of the control condition (Wald’s χ2(1) = 12.40, p < 0.001, b = -1.10, Exp(B) = 0.33). There were no significant differences regarding low and high group participation threads (see Table 1). Although the provision of behavioral GA information did not produce such clear effects as the provision of cognitive GA information, there were interesting effects, partially supporting H1.2.

To better understand the influence of NFC on selected threads visualizing high (H2a) and low (H2b) group knowledge, two moderator analyses were conducted. In our moderator analysis, NFC represented the moderator, cognitive GA information was the predictor, and (a) selection preference of high group knowledge threads and (b) selection preference of low group knowledge threads were the respective criteria for the two analyses. The calculations revealed no significant interaction effects for NFC and the cognitive GA factor on selection preference of high group knowledge threads (β = -0.01, t(141) = -0.09, p = 0.930) and low group knowledge threads (β = 0.06, t(145) = 0.78, p = 0.433), thus refuting H2a and H2b.

Impact on number of discussion posts (hypotheses 3 and 4)

Concerning H3.1, a 2 × 2 ANOVA shows the expected positive main effect for the factor behavioral GA information (F(1, 154) = 64.50, p < 0.001, η²_p = 0.30; further single/interaction GAT effects were absent; see Table 6 in the Appendix). People with behavioral GA support wrote approximately two more contributions to the discussion forum (without being instructed to do so) than people without behavioral GA support (no support: M = 1.74, SD = 1.75; behavioral GA: M = 3.63, SD = 1.22).

Concerning H3.2, we applied a two-sided bivariate correlation to examine if there was a relationship between the presented individual knowledge score and the number of written discussion posts. Because the focus here was on the effects of cognitive GA information, only people who received knowledge bars were included in the calculations. We found a small positive correlation (r(79) = 0.25, p = 0.029), supporting H3.2. Participants wrote more discussion posts when the cognitive GAT presented a higher individual knowledge score.

To better understand the influence of the tendency toward social comparison processes on the effect of the factors of cognitive GA information (H4a) and behavioral GA information (H4b) on the number of discussion contributions, we conducted two moderator analyses. In these moderator analyses, the tendency toward social comparison processes represented the moderator, the cognitive or behavioral GA information factors were the predictor, and the number of discussion contributions was the criterion. However, there was no significant interaction effect between cognitive GA support and tendency to comparison (β = 0.03, t(153) = 0.33, p = 0.373). Also, the interaction of behavioral GA support with tendency toward social comparison did not explain a significant proportion of the number of discussion contributions (β = 0.07, t(153) = 1.02, p = 0.154). H4a and H4b are not supported.

Interaction effects on learning (hypothesis 5)

Before we checked H5 or the interaction effects on learning outcomes, we examined whether there were differences in the first pretest score. The descriptive pretest scores (10 points for 10 correct answers) are as follows: no support – M = 4.66 (SD = 1.84); cognitive GA support – M = 4.73 (SD = 1.61); behavioral GA support – M = 4.85 (SD = 1.74); and combined support – M = 5.16 (SD = 1.22). A 2 × 2 ANOVA shows that the cognitive GA information factor (F(1, 154) = 0.54, p = 0.465, η²_p = 0.003), the behavioral GA information factor (F(1, 154) = 1.45, p = 0.230, η²_p = 0.01), and their interaction (F(1, 154) = 0.20, p = 0.656, η²_p = 0.001) did not become significant; hence, potential effects of prior knowledge could be excluded.

For the main calculations of H5, we were interested in the interaction effects of cognitive and behavioral GA information regarding the learning performance variables. For descriptive values, see Table 2. Here, the means show that a slight superiority of combined GA support is recognized.

Table 2 Descriptive statistics (M (SD)) of the learning variables

Although the descriptive values are always slightly in favor of the combination condition, neither the interaction effects for the whole posttest score (F(1, 154) = 1.84, p = 0.178, η²_p = 0.01) nor the learning gain (F(1, 154) = 0.02, p = 0.903, η²_p < 0.001) became significant. Further single GAT effects were also absent (see Table 6 in the Appendix).

A repeated-measure analysis shows that there was a general learning gain (improvement of fact-based items between pretest to posttest) across all conditions (F(1, 157) = 65.00, p < 0.001, η²_p = 0.29) from M = 4.85 (SD = 1.62) points to M = 6.04 (SD = 1.77) points. The interaction with our created wiki environment thus generally led to an improvement in learning, even when it was independent of GA manipulation.

Interaction effects on article quality (hypothesis 6)

As assumed in H6, a 2 × 2 ANOVA reveals that there were positive interaction effects on (a) article word count (F(1, 154) = 4.17, p = 0.043, η²_p = 0.03), (b) argument count (F(1, 154) = 18.31, p < 0.001, η²_p = 0.11), and (c) argument balance (F(1, 154) = 5.07, p = 0.026, η²_p = 0.03), however, the latter was low in all conditions (possible maximum value = 1). For interaction plots, see Fig. 3. Further single GAT effects (see Table 6 in the Appendix) are not interpreted because of the significant interaction terms.

Fig. 3

Interaction plot (M (SD)) of article word count, argument count, and argument balance

To show that each form of GA information benefits when the other one is added, we calculated two post-hoc simple effect analyses for each dependent variable, using cognitive or behavioral GA information as a predictor. The results reveal the visual trends and show that the single presence of knowledge or participation awareness information did not make a significant difference on (a) word count, (b) argument count, and (c) argument balance, in comparison to the control condition. Only in combination are significant positive effects in evidence (see Table 3). Thus, H6 is supported.

Table 3 Post-hoc simple effect analyses for article quality

Exploratory analyses

Due to the missing positive interaction effects for the learning variables, it can be assumed that an increased mental effort or decreased well-being played a role here (see “Exploratory investigations”).

No significant interaction effect was found for the 2 × 2 ANOVA concerning the overall mental effort (F(1, 154) = 0.79, p = 0.376, η²_p = 0.01). However, the 2 × 2 ANOVA relating to well-being revealed a significant interaction effect (F(1, 154) = 5.87, p = 0.017, η²_p = 0.04); see Fig. 4. Further single GAT effects were absent for both dependent variables (see Table 6 in the Appendix).

Fig. 4

Interaction plot (M (SD)) of well-being

Post-hoc simple effect analyses show that the single presence of cognitive or behavioral GA information made a significant positive difference on well-being. However, in combination these positive effects were not significant, neither with knowledge nor participation awareness information as a predictor (see Table 4).

Table 4 Post-hoc simple effect analyses for well-being

Eye-tracking results

To examine visual attention in our wiki environment, we chose the relative duration time as a measurement metric, because none of the editing phases contained an automatic forwarding. After the respective time frames, participants were asked to go to the next page. They proceeded independently, so the absolute duration times were slightly different for each participant. To ascertain which elements attracted the most attention, the following areas of interests (AOIs) were defined for the selection phase (including both STs; goal to learn new content and goal to support other collaborators) and discussion phase: topics, cognitive GA bars, and behavioral GA bars. Header bars (individual and whole group level) and wiki thread bars (subgroup level) were included together; see Fig. 1 as a reference. The article phase was not considered as it contained no GA information. The folded text field in the discussion phase (see Fig. 1, right side) was also not defined as an AOI in the first step or for the duration time. Most of the focus on it was during the discussion phase, which was not included in the selection phase, and thus including it would have made the further duration times uninterpretable.

Independently of all conditions and editing phases, the thread topics received the most attention out of the three defined AOIs (see Table 5). In the control condition, this was 100% because there was no GA support at all. Moreover, both types of GA information received considerably more attention in the selection phase, especially in combination, than in the discussion phase. Furthermore, (individual and group) knowledge bars received more attention than (individual and group) participation bars in the selection phase (see Table 5). In the discussion phase, participation bars got more attention than knowledge bars in separate GA support conditions, whereas it was the other way around in combination.

Table 5 Duration time on area of interests (in %)

In order to derive further insights into attention processing, the share of the total saccade amount between all AOIs was considered (see Table 7 in the Appendix). This time, the text field was included as a further AOI. Saccade results indicated that there was more switching between the wiki thread topics and the text field (especially in the discussion phase), whereas GA information was given relatively little attention, thus supporting the results of the duration times. However, in the selection phase attention was more distributed with combined GA information, with more saccades between participation and knowledge bars.

Discussion

The aim of the reported study was to uncover potential individual effects of providing learners with cognitive and behavioral GA information, as well as investigating the interaction of both types of GA information in a wiki context.

Main results

The results of our study reveal that both cognitive (H1.1) and behavioral (H1.2) GA information have certain effects on selection. Especially with the goal to learn new content (ST1), threads involving much group knowledge as well as group participation are preferred when there is GA support. H1 is, therefore, partly supported. Related to the number of written discussion threads (H3.1), behavioral GA information leads to a stronger alignment of engagement. Moreover, people for whom the GAT displayed more individual knowledge were more motivated to support other people in resolving content-related conflicts (H3.2). H3 is, therefore, fully supported. With regard to the assumed effects of the influencing variables (H2a/b and H4a/b), no significant effects could be found. Therefore, H2 and H4 are not supported. With respect to the assumed positive interaction effects on learning performance (H5) of behavioral and cognitive GAT combinations, we were not able to show significant results. Thus, H5 is not supported. However, there were positive interaction effects for article productivity (word count, argument count) and argument balance (H6), which were our defined metrics for article quality. Participants from the combination condition scored best here. H6 is, therefore, fully supported.

Single effects on cognitive challenge

With regard to selection decisions or the cognitive challenge, both H1.1 and H1.2 are partially supported. When the intention was to use the wiki threads as a learning material (ST1), people with cognitive GA support preferred threads involving high levels of group knowledge. This is in line with the study of Dehler et al. (2011), who found that learners are more likely to ask for explanations if their learning partners know more than they do. Selection effects of behavioral GA information have not been systematically investigated in prior studies, but our research shows that there were indeed behavioral GAT effects for ST1 (goal to learn new content) in favor of high group participation threads. These are not unexpected, because high participation or the availability of many contributions can mean that there is much content to learn. In terms of supporting others (ST2), however, the effects are not clear for either type of GA information. This may be due to the fact that we only presented GA information without any action recommendations to the learners. Thus, some participants might have associated low group knowledge with a need to support, whereas other participants saw high group knowledge as an incentive to engage in and support higher-quality discussions. Similarly, low group participation may indicate a need for support, but a high level of participation might also have signaled heated discussions or a high potential for conflicts, which increased the need for support. Since GA information only provides implicit incentives (Janssen & Bodemer, 2013), it can be interpreted in different ways. Therefore, it would be interesting to examine how participants behave when telling them the expected use of GA information, such as by enhancing GA usage with scripting (see Tsovaltzi et al., 2014). To sum up, our results show that cognitive GATs in wikis have the potential to overcome the cognitive challenge in wiki-based learning by helping to focus on meaningful content (Ollesch et al., 2019). Moreover, behavioral GA support can have some valuable effects here, which is a new insight.

For H2a/b, we had assumed that NFC would influence the effect of cognitive GA information on selected threads presenting high and low group knowledge. However, this was not statistically supported. Think-aloud protocols could be useful for gaining a more precise idea of what kind of threads are considered more cognitively strenuous and why.

Single effects on behavioral challenge

With regard to discussion contributions or the behavioral challenge, both H3.1 and H3.2 were supported. Participants who saw individual and group participation bars wrote two more discussion contributions in 20 min than those without these participation bars. This is a large difference considering the short time slot, and it aligns with the findings of Kimmerle and Cress (2008) and Janssen et al. (2011). Therefore, behavioral GA information can be shown to contribute to overcoming the behavioral challenge in wiki-based learning. With regard to cognitive GA information, it was not clear in advance what effect this would have on discussion engagement, especially for persons with high individual knowledge. We were able to show that people with more knowledge presented also wrote more supporting discussion posts, so cognitive GA information did indeed address the behavioral challenge in wikis. However, for those with less individual knowledge presented, the trend is more likely to go in the opposite direction.

For H4a/b, we assumed that the tendency toward social comparison processes would increase the GAT effects (Neugebauer et al., 2016). This tendency was not significant for the behavioral GA condition, possibly because the behavioral GAT effects were generally already strong (see H3.1) so that individual differences did not make a big difference. Furthermore, real or synchronous interactions instead of bogus interactions might have increased the effects for both types of GA information.

Interaction effects of cognitive and behavioral GA information

Regarding the interaction effects on the learning-related variables (whole posttest and learning gain), the results for H5 were not significantly supported. There was a general increase in learning related to fact-based items from the pretest to the posttest. This means that the interaction with the wiki content per se was beneficial. However, we originally assumed that the effects for behavioral and cognitive GA information combinations would be stronger than found in this study (Ollesch et al., 2019). We expected that cognitive GA information would help indicate unequal knowledge levels within the group and thus trigger beneficial socio-cognitive conflicts (Bell, 2004), and that behavioral GA information would help verbalize and internalize new content more deeply (Garrison et al., 2000; Webb, 1991). It is possible that a learning superiority was not that clear because of the time limit of 20 min in the discussion phase, leading to the content being dealt with rather superficially. Since the instructions for the discussion phase were open, it would have been interesting to expand the engagement time and define the learning purpose more clearly, thereby possibly identifying stronger learning differences between those conditions.

With respect to article quality, our findings clearly show a positive interaction effect or that, on average, significantly more new words and arguments were added when both types of GA information were provided, which supports H6. Blumenstock (2008) suggested quantitative metrics like word count as good predictors of highly qualitative wiki articles in comparison to other more complicated methods. Additionally, participants in the combination condition wrote on average more balanced wiki articles – that is, they inserted both pro and contra arguments regarding one topic into the whole article – which also supports H6. This finding should be looked at more deeply in the future; however, it provides an initial indication that participants in the combination condition not only added more words and arguments, but also argued in a more balanced way, even though balance scores were in general rather low. Future learning environments and research should take this into account in order to support learners in their balance of argumentation style.

To sum up, the results on word count, argument count, and argument balance demonstrate that cognitive and behavioral GA support can indeed improve the contribution to group outcomes or wiki article quality by triggering participation and providing awareness about cognitively relevant content. It makes sense, therefore, to look at both GA components together (see Janssen & Bodemer, 2013), because respective activities go hand in hand: Behavioral activities can contribute to increasing cognitive competencies and vice versa.

Exploratory findings and eye-tracking observations

We did not find any significant differences in mental effort, so the provision of two types of GA support did not have a negative impact on cognitive resources. However, the combination of cognitive and behavioral GA information did not have a positive effect on well-being for the group, which contrasted with the situation for the single support conditions. Although we did not focus on emotional processes and outcomes, these might be considered more in relation to long-term usage intentions because social media use is significantly driven by emotional needs and interpersonal support. Current research suggests giving greater weight to emotional regulation support in CSCL settings, both at individual and group level (Järvelä et al., 2016, 2019). Therefore, future research should integrate and investigate emotional cues in the form of awareness tools (see Slakmon & Schwarz, 2019).

Eye-tracking observations show that the content elements (topics and text field) received more attention (duration time and saccades) than the GA information, which means that GA bars were more likely to be seen as an add-on. We had expected that GA information would have a higher visual impact, which might have increased its effects. However, as higher duration time and saccades for the GA information were found in the selection phase than in the discussion phase (especially for GAT combinations), it is clear that the GA information is more likely to be drawn to the selection than to the actual engagement. As duration time was higher for cognitive GA information than for behavioral GA information in the combination condition for the selection and discussion phases, this might mean that cognitive GA information was more important to participants. However, this could also be a sign of higher complexity. The cognitive GA bars were possibly more difficult to elaborate on. Whereas the participation bars simply represented the number of contributions, in the case of knowledge bars participants may have needed more elaborations on necessary actions with little or high individual and group knowledge.

The eye-tracking sample was small, so it is difficult to say whether the results can be fully generalized. However, they provide initial indications of potential improvement. We plan to change the GA design in future studies to make it more prominent or to explicitly suggest threads with specific GAT constellations when it comes to, for example, supporting others and best practices are not known (see Rosé & Ferschke, 2016). Also, implicit real-time gaze feedback would be conceivable, which is known to facilitate coordination processes and learning gains in CSCL (Hayashi, 2020). In this study, we deliberately did not specify a concrete goal for GA usage. In the context of online communities, it is conceivable to give guidelines or tips on how to deal with GA information on a visual level. This could also be accompanied by a stronger consideration of GA information, which, according to our eye-tracking observations, seems to take a back seat. Eye-tracking was very helpful for assessing how attention-grabbing our GA design is, so we recommend ensuring that support measures are prominently placed in the focal point of a user’s visual field to enlarge their effects. Moreover, adapting the learning environment accordingly is a priority (Wise & Schwarz, 2017). We believe that attention patterns could be used in the future to give adaptive (on-the-fly) feedback to students (Król & Król, 2019) when helpful support measures are not adequately considered.

Addressing limitations

Our study has some limitations. First, a wiki is a specific platform and not everyone has experience with handling wikis. However, advanced wiki-specific functionalities (markup language, etc.) were removed beforehand in order to simplify the handling as much as possible and to make the results generalizable.

The greatest limitation is most likely the laboratory design of the study, which focused more on internal validity and experimental control than on external validity. Laboratory settings have artificial features in comparison to real-world settings. For example, participants knew they were in an experiment, and the visualizations of the group’s cognitive and behavioral variables were generated artificially rather than by real group data, which makes it easier to manipulate the independent variables but reduces external validity. The overall topic could not be chosen by the user, which is often a critical defining characteristic of wikis like Wikipedia. Moreover, the group’s behavioral GA mean was initially always higher than the participants’ individual behavioral GA bar. This allowed for positive effects rather than negative effects to occur. If this had been the other way round, opposite effects are conceivable. According to Kimmerle and Cress (2009), this could be counteracted by using a different information presentation format. In our study, for example, a format that visualizes not only group means but also individual values of other people might have allowed for more accurate comparisons, even if group mean values were not necessarily high. Nevertheless, we believe that it is necessary to further investigate these results in more realistic settings and to investigate potential negative effects of low group means in subsequent work.

Despite these concerns, we believe that conducting our study in a laboratory setting was important because the complex mixture of uncontrolled factors in field studies can threaten their internal validity (Anderson & Bushman, 1997). The detailed and exact analysis of the isolated and pure effects of cognitive and behavioral GA information would not have been possible without this kind of manipulation (e.g., allowing for systematically different GAT levels). Field studies include confounding variables, for example introduced by possible cheating in knowledge tests or error variance due to fluctuations in technological conditions (Reips, 2000). Despite the highly controlled laboratory setting, it is worth noting that participants were not instructed to do anything specific in the discussion phase, such as sharing their knowledge; instead, they were free to read or write discussion posts, which particularly makes the behavioral results notable. Furthermore, real interaction over larger time spans might have even heightened the effects. Finally, the setting allowed us to examine the attentional impact of the GA information, which helps to optimize future design efforts.

Our laboratory study provides a basis for recognizing fundamental GA mechanisms that can be easily transferred to more realistic settings. This is supported by the fact that many properties of real wiki scenarios can be found in our laboratory study – for example, not knowing each other individually, selecting subtopics of interest, writing asynchronous discussions, and editing the wiki article – even if they were investigated in isolation in our research. In addition, the high number of different types of data collected in our study (from log-data to attentional metrics) also suggests an easy transfer to more realistic settings, since studies need to collect more diverse types of data to characterize a situation in all its complexity (Jeong et al., 2014). In environments such as Wikipedia, the editing phases are not divided in the way we divided them, but the design could be contextualized in a classroom setting, which is guided by a teacher outside the physical classroom who defines the overall topic and deliberately divides the collaboration into wiki discussion and editing phases. GATs may be helpful in such and similar learning situations to give real-time feedback about task progress, facilitating teacher orchestration (Martinez-Maldonado, 2019). One way of examining the effects in such contexts could be a randomized classroom quasi-experiment in which the experimental conditions are set up through online grouping (see Cho & Schunn, 2007).

Implications

Although most of our results show only small effect sizes, it can be stated that both cognitive and behavioral GA information have an effect on cognitive processes like selection decisions (cognitive challenge), as well as on behavioral processes like motivation to contribute in wiki-like environments (behavioral challenge). Thus, we believe it could be helpful to provide such GA information in real wiki learning or similar scenarios to make it easier for users to navigate the platform and plan certain actions.

In this paper, we have shown that cognitive GA information indeed addresses the cognitive challenge (Ollesch et al., 2019). Behavioral GA information also seems to offer direction to students. Furthermore, we have demonstrated that behavioral GA information, in particular, addresses the behavioral challenge (Kimmerle & Cress, 2008; Ollesch et al., 2019). Hence, this can be a very helpful support measure in wiki or similar scenarios when the goal is to get people to contribute.

In the case of cognitive GA information, effects regarding the behavioral challenge are not so straightforward. If the focus of a wiki learning activity is on contributing or supporting, less knowledgeable users may better overcome their inhibitions if the state of knowledge is not visualized. To avoid overwhelming participants with information in this study, we considered only the effects of GATs in the discussion forum. It would be interesting to look at the effects at the article level in further studies. Furthermore, we deliberately chose average GA values for the bogus group to give participants the possibility of outperforming or underperforming the group means. Therefore, it would be important to look at what the effects look like if GATs visualize group deficiencies or superiorities. Cognitive and behavioral GA information might have the opposite effect and should possibly even be faded out in order not to reduce the motivation of the participants. In addition, further user characteristics should be considered in follow-up studies, as the effects of NFC and the tendency toward social comparison could not be shown.

Finally, it can be stated that article quality can be increased by combining cognitive and behavioral GA information, which means that these GAT combinations could be evaluated in the field context. Based on the results on well-being, it has to be considered to what extent emotional processes or challenges in wikis can be further supported in the future, such as by asking for wiki users’ subjective feelings or by using sentiment analysis of discussion posts to make people aware of potentially unfriendly comments. Sentiment analysis includes the automatic recognition of the tonality of text, such as a tweet, an online customer review, or a comment in a forum (Liu, 2012). Although this has not yet been applied to the wiki context, it might in the future be used to automatically extract the friendliness level of wiki contributions based on constellations of words used.

Conclusion

The results show that providing users with cognitive and behavioral GA information, both when separately applied and in combination, can have positive effects on cognitive as well as behavioral learning processes and outcomes in social media. This is highly relevant for the practical application and support of wikis and other CSCL platforms. However, the results also highlight that cognitive and behavioral GAT combinations do not necessarily have a positive effect on emotional outcomes like the well-being of learners. Maintaining a positive group climate is one of the big challenges in CSCL, which underlines the necessity of also considering emotional processes in learning contexts. Such investigations serve to discover the extent to which emotional GA needs to be supported in wikis or similar platforms to create a sense of community. In general, more laboratory and field studies that systematically investigate other types of GAT combinations (e.g., emotional and cognitive GA information) are therefore needed to uncover other interaction effects. Further, disentangling behavioral, cognitive, and emotional processes and outcomes on different levels (support measures, challenges) seems to be essential for understanding and using the full potential of GA support in social media contexts.

Appendix

Table 6 Summary of all 2 × 2 ANOVAs

Table 7 Saccade metrics of the selection and discussion phase (% related to total number of saccades)