Patterns of action transitions in online collaborative problem solving: A network analysis approach

Abstract

In today’s digital society, computer-supported collaborative learning (CSCL) and collaborative problem solving (CPS) have received increasing attention. CPS studies have often emphasized outcomes such as skill levels of CPS, whereas the action transitions in the paths to solve the problems related to these outcomes have been scarcely studied. The patterns within action transitions are able to capture the mutual influence of actions conducted by pairs and demonstrate the productivity of students’ CPS. The purpose of the study presented in this paper is to examine Finnish sixth graders’ (N = 166) patterns of action transitions during CPS in a computer-based assessment environment in which the students worked in pairs. We also investigated the relation between patterns of action transitions and students’ social and cognitive skill levels related to CPS. The actions in the sequential processes of computer-based CPS tasks included using a mouse to drag objects and typing texts in chat windows. Applying social network analysis to the log file data generated from the assessment environment, we created transition networks using weighted directed networks (nodes for those actions conducted by paired students and directed links for the transitions between two actions when the first action is followed by the second action in sequence). To represent various patterns of action transitions in each transition network, we calculated the numbers of nodes (numbers of actions conducted), density (average frequency of transitions among actions), degree centralization (the dispersion of attempts given to different actions), reciprocity (the extent to which pairs revisit the previous one action immediately), and numbers of triadic patterns (numbers of different repeating formats within three actions). The results showed that pairs having at least one member with high social and high cognitive CPS skills conducted more actions and demonstrated a higher average frequency of action transitions with a higher tendency to conduct actions for different number of times, implying that they attempted more paths to solve the problem than the other pairs. This could be interpreted as the pairs having at least one student with high social and high cognitive CPS skills exhibiting more productive CPS than the other pairs. However, we did not find a significant difference across the pairs in terms of alternating sequences of two or three actions. Investigating the patterns of action transitions of the dyads in this study deepens our understanding of the mutual influence between the CPS actions occurring within dyads. Regarding pedagogical implication, our results offer empirical evidence recommending greater awareness of the students’ social and cognitive capacities in CPS when assigning them into pairs for computer-based CPS tasks. Further, this study contributes to the methodological development of process-oriented research in CSCL by integrating an analysis of action transition patterns with a skill-based assessment of CPS.

Introduction

In today’s knowledge-intensive and digitalized society, there is a critical need for learners to combine their expertise and ideas in various collaborative situations to solve problems together, both in individuals’ working lives and in a variety of learning communities. This issue has been addressed by research on computer-supported collaborative learning (CSCL). In CSCL, students share and construct knowledge with their group members through various interactions. In this way, we may consider collaboration as connections between diverse types of entities, such as students, actions, and the digital artifacts that the students create (van Aalst, 2009). Relations that are formulated within and/or between these entities are associated with effects on CSCL outcomes (Ouyang, 2021), embodying the philosophy of CSCL, which states that “relationships matter” (Dado & Bodemer, 2017, p.161).

The present study investigated sequential actions in CSCL, including, for instance, clicking buttons, dragging to move objects with a mouse, and typing texts in the chat window. Particularly, we studied action transitions to better understand the relations within the sequential progressions of actions in dyadic interaction in an online assessment environment. The progressions of actions that a pair of students enact contribute towards processes of understanding, planning, solving, and revising, which are universally applicable across tasks in computer-based assessment environments (Care et al., 2015). This conceptualization provides a solid foundation for studying the processes of online assessment tasks through the lens of action sequences in the broader context of students’ efforts towards collaboration. The actions from two paired participants are interdependent, and their contributions “mutually build upon each other” (Hesse et al., 2015, p. 38). Therefore, the transition patterns between actions in sequences might capture this mutual influence between students in a CSCL environment.

Today’s technologies for CSCL are becoming more advanced, and large amounts of computer-generated data are available, such as in the form of log files (Jeong et al., 2014). In addition to communication data (e.g., the contents of messages), log file data also record the sequences of actions that students conduct in CSCL environments. Contributing to inter-objective theories referencing networks of students, actions, and artifacts (Stahl & Hakkarainen, 2021), social network analyses have been widely applied to analyze the relations between entities in CSCL research (Dado & Bodemer, 2017). In the current study, applying social network analysis allowed us to capture the relations between the sequential actions contributed by each student in a dyad setting, here represented by patterns of action transitions.

Taking the form of short-term collaboration in CSCL (Reimann, 2009), collaborative problem solving (CPS) has been regarded as an important skill in the twenty-first century (Griffin et al., 2012; Ludvigsen et al., 2015; Rummel & Spada, 2005). CPS refers to collaboration as “a coordinated, synchronous activity that is the result of a continued attempt to construct and maintain a shared conception of a problem” (Roschelle & Teasley, 1995, p. 70). The social and cognitive skills of CPS have typically been studied in assessment research (Hesse et al., 2015). Administering CPS tasks designed for dyads in a computer-based assessment environment among sixth graders, we treated students’ social and cognitive skills of CPS as outcomes and worked to disentangle the relations between such outcomes and the patterns of action transitions contributed by pairs of students working together. This study deepens our understanding of the mutual influence of paired students’ actions related to their CPS skills in a computer-based CPS assessment environment. Moreover, our study contributes to the methodological development of process-oriented investigations in CSCL by integrating an analysis of action transition patterns with a skill-based assessment of CPS.

Background

Peer interactions and patterns of action transitions

CSCL inherently involves intensive interaction between peers, which can be very effective in promoting learning (for a meta-analysis, see Tenenbaum et al., 2020). Peer interactions in CSCL are often mutual and untutored, involving “the use of small groups of students working together to achieve common goals of learning” (Topping et al., 2017, p. 5). In CSCL, peer interactions occur through the contingent, and therefore are in inherently sequential order of operational actions (e.g., moving objects with a mouse) and written/oral communication (Stahl & Hakkarainen, 2021). A given operational action or written/oral utterance is typically a response to the previous operational action or discourse move and is “generally designed to provoke a response and to propel the discourse and inquiry forward” (Stahl & Hakkarainen, 2021, p. 37). Intrinsically, contributions of these operational actions and written/oral communication exhibit mutual influence (Baker et al., 2007). Mutual influence between actions operates through the students who produce them. The phenomenon of mutual influence between actions is a matter of degree of strength, and it often operates within the sequentiality of actions (Stahl & Hakkarainen, 2021). Therefore, the patterns of action transitions (i.e., the transition between two actions with the first action followed by the second action, Zhu et al., 2016) might contribute towards the degree of mutual influence between actions. In the current article, by “actions,” we include operational actions and actions of writing messages for communication, though we exclude the specific content of messages.

The interwoven progression of operational actions and written contributions leads to successful task solutions. However, the sequence of actions that will ultimately lead to a successful solution is not obvious to group members at the start of the task (Wieber et al., 2012), especially in a setting where individuals do not have identical resources in the online environment. Group members or pairs need to actively communicate with one another (e.g., type texts in the chat window) and to collectively attempt different operational actions (e.g., drag to move an object) to discover the path to the successful solution, often through trial-and-error. Accordingly, the action space affords high variability in possible action transitions. Based on U.S. data from PISA 2015, De Boeck & Scalise (2019) report that students who contribute more actions (e.g., click buttons, excluding actions of oral or written communication) are less successful. This may be because these PISA interactions are human–agent interactions (i.e., a student interacts with the computer-simulated agent rather than with another student; He et al., 2017), which results in limited opportunities for communication, which in turn may hinder demonstration of CPS skills when compared with the human–human approach (i.e., a student interacts with another student). In contrast to the findings of De Boeck & Scalise (2019), we assume that pairs conducting more actions in CPS tasks are likely to demonstrate better outcomes in the human–human approach because tasks in the human–human approach often require more actions, including those for the purpose of communication between partners. Another impetus for additional actions is that no structured assistance is offered, as it is for tasks implemented using the human–agent approach. Therefore, our study fills a research gap studying the relationship between the quality of students’ CPS outcomes and the number of actions conducted by paired students in the human–human approach, where the number of actions conducted may be very different than that in the prior work related to human–agent collaboration.

On the other hand, Zhu et al. (2016) propose and validate two different patterns of action transitions in a computer-based problem solving task for individual students: First, they discuss the average frequency of action transitions (e.g., click available buttons, excluding actions of oral or written communication). In computer-based assessment environments, students need to attempt different paths to solve a problem. The more frequently students change actions on average as they engage in more exploration, the longer the total resulting sequence of actions. Second, they discuss the dispersion of attempts given to different actions. This pattern captures the extent to which students frequently conduct certain actions. The more dispersion of attempts given to different actions is present, the more students tend to conduct actions for the same number of times.

In addition to studying the patterns of action transitions, Zhu et al. (2016) also relate patterns of action transitions to individual students’ problem solving outcomes. In particular, Zhu et al. (2016) investigate the action transitions in an online problem-solving task conducted by individual students whose efficiency scores associated with a scoring rubric are related to student outcomes in terms of their problem solving speed. In that case, speed is rated higher if students only take the necessary actions in order to reach a solution. The systematicity scores are also calculated that are related to how systematic students were at solving the problem. Here the systematicity score was rated higher if students follow the problem solving routine in the manual provided, meaning checking, fixing the problem, and testing the solution, respectively. Zhu et al. (2016) report that eighth graders with higher efficiency scores demonstrate a significantly lower average frequency of action transitions overall, whereas students with higher systematicity scores exhibit a more even distribution of probabilities across action sequences, indicating that they do not show a preference for certain action sequences over others.

These findings are understandable given the conditions under which the study has been conducted in Zhu et al. (2016). That is, students in that study engaged in individual problem solving guided by a manual that provided the needed knowledge for the computer-based assessment task. In the work reported in this paper, in contrast, we focus on dyads engaging in human–human collaboration, with no provided structured assistance related to the tasks provided for the participants (see details in Methods section). Consequently, we assumed that the frequency of action transitions, and dispersion of attempts, demonstrate the productivity of paired students’ CPS in computer-based assessment environments because these dimensions bound the space of possible measures of exploration pairs engage in while searching for a solution path. Compared with unproductive CPS, productive CPS is assumed in those dyads that conduct more actions with a higher frequency of action transitions on average and with a higher tendency to conduct actions for different number of times, a pattern that would stand in contrast to the findings in Zhu et al. (2016). In expanding from individual problem solving to problem solving in dyads, the present study fills a critical knowledge gap regarding how students’ CPS outcomes are related to their average frequency of shifts in action progressions and the dispersion of their attempts given to different actions, thus challenging the interpretation of past findings (Zhu et al., 2016).

Moreover, Zhu et al. (2016) study the extent to which individual students revisit the most recent one or two actions during online assessment tasks. Information regarding the way students revisit these very recent actions can provide sophisticated insights into action design and the common pitfalls in problem-solving tasks (Zhu et al., 2016). For instance, if a certain action transition is frequently repeated yet is not a part of the path to the solution, this indicates that there might be a potential design problem in the task or that there is a common misconception hindering effective problem solving. Zhu et al. (2016) report that students who represent a higher level of systematicity scores (i.e., outcomes of individual students’ problem solving) to a lesser extent revisit the most recent previous actions right away (the repetition of the same actions over and over was not taken into account here). In our setting of dyadic human‒human collaboration, we build on past insights by exploring the extent to which dyads revisit the most recent one or two actions during a computer-based CPS task as an extension of existing knowledge about individual problem solving to dyadic CPS.

CPS skills and group composition

CPS has gained increasing societal interest, especially because of its inclusion in large-scale international assessments and in the development of computer-supported learning assessments (Shute & Rahimi, 2017). In 2015, the OECD integrated a CPS framework (OECD, 2017) into their PISA 2015 computerized assessments, utilizing human–agent tasks. Another important initiative to evaluate CPS is the Assessment and Teaching of 21st Century Skills (ATC21S) project (Care et al., 2018) that developed human–human computer-based assessment tasks in pairs.

The theoretical framework of CPS in the present study relies on the one developed in the ATC21S project (Hesse et al., 2015). In this framework, a distinction is proposed between social and cognitive skills, which are further divided into a hierarchy of subskills (Hesse et al., 2015). This framework is not identical to the one in PISA 2015 that does not separate social skills from cognitive skills in CPS tasks. In Hesse et al.’s (2015) framework, social skills refer to the “collaborative” part of CPS that is often enacted through social interactions with the paired partner, including participation, perspective taking, and social regulation. Cognitive skills, on the other hand, refer to the “problem-solving” part of CPS, such as task regulation and knowledge building. The social and cognitive skills of CPS are inherently intertwined (Hesse et al., 2015). Establishing a mutual understanding requires members to socially engage in knowledge building and to constantly work to find common ground (Clark & Brennan, 1991), which calls for certain cognitive skills, such as collecting the elements of information related to the task. Small groups or pairs often experience major difficulties, particularly in establishing common frameworks of references, coming to a joint understanding, resolving differences of understanding, and negotiating individual and collective actions (Barron, 2000). Communication involving group members with high social skills is the key to resolving these challenges (Bause et al., 2018). Consequently, the skill composition of the group has been recognized as one contributor to the success of the collaborative learning process (Cen et al., 2016).

The benefits of collaborative learning have often been argued as emerging through the assistance exchanged between collaborating partners (Stahl et al., 2006). For instance, with participants aged from 18 to 68, Dowell et al. (2020) report that groups with more members taking social responsibility performed better than those groups with greater proportions of less socially engaged partners. In school settings, students with a lower skill level can benefit from a collaborative learning situation more than students with a higher skill level (Saner et al., 1994). There are claims that poor group composition is one of the main reasons for unproductive collaborative learning (Fiechtner & Davis, 1985; Graf & Bekele, 2006). The optimal sharing of resources (e.g., existing skills, learning materials) has been suggested as vital for making collaborative learning more effective (Isotani et al., 2009). However, the reality in authentic classroom settings is that collaborating groups are often formed on a voluntary basis. In some cases, the voluntary selection of group members can result in off-task behaviors and resistance to group work (Dillenbourg, 2002). Therefore, there is a need for a retrospective investigation into the relationship between a group’s composition in terms of heterogeneous skill levels and the productivity of its collaboration. Mapping onto the PISA 2015 framework, Herborn et al. (2017) report that seventh graders’ individual profiles of social and cognitive skills differed in CPS performance in the human–agent approach. That is, students with high social and high cognitive skills demonstrated significantly better performance, whereas those with low social and low cognitive skills exhibited poorer performance.

Based on Herborn et al. (2017), Andrews-Todd & Forsyth (2020) indicate that students with low social/low cognitive skill profiles exhibited the poorest performance in the human–human approach in triads. Moreover, having at least one high social and high cognitive member in a triadic group facilitates performance (Andrews-Todd & Forsyth, 2020). Herborn et al. (2017) and Andrews-Todd and Forsyth (2020), respectively, have uncovered a relationship between students’ performance related to individual skill profiles and the skill composition of groups with three members.

Taking all this into account, the present study fills a research gap in the field by examining how students’ learning outcomes are related to their productive/unproductive CPS in human–human dyads. The quality of the solution for a CPS task has been the core criterion of interest that is traditionally regarded as the learning outcome (Graesser et al., 2017). However, log file data recorded in online CPS environments on students’ problem solving processes provide rich information on how students conduct various actions to solve a problem instead of merely whether the problem is solved or not (Zhu et al., 2016). In our study, we utilized CPS skill levels as learning outcomes calculated from log file data in the processes emerging from a series of CPS tasks. In line with the findings of Herborn et al. (2017) and Andrews-Todd & Forsyth (2020), we hypothesized that in computer-based CPS assessment tasks, pairs having at least one member with high social and high cognitive skills might exhibit more productive CPS than other pair compositions. That is, dyads having at least one student with high social and high cognitive skills appear to conduct more actions with a higher frequency of action transitions on average and with a higher tendency to conduct actions for different number of times in CPS.

Social network analysis (SNA) to study processes

Alternative approaches exist for understanding different processes in computer-based learning environments, for instance, process-oriented content analysis (Isohätälä, 2020), process mining (Paans et al., 2019), and lag-sequential analysis (Malmberg et al., 2017). Other researchers have utilized Markov models and the item response theory framework (Shu et al., 2014) for analyzing process data. For modeling concurrent and sequential features in processes, Petri nets have also been utilized (Reisig, 1985). With their complex rules, it is claimed that Petri nets are not suitable for CSCL process data because “they are overly deterministic” (Reimann, 2009, p. 252). Therefore, heuristic methods that are more algorithmically complex and more readily to interpretate results need to be applied after Petri nets have been employed (Reimann, 2009).

In addition to the methods mentioned above, SNA encompasses an important array of methods for analyzing various processes in CSCL as it allows for the analysis of patterns of relationships between two nodes interacting with one another in a relation-based system (Wasserman & Faust, 1994). In CSCL settings, nodes could be any entities in the CSCL processes: humans (e.g., instructors, students), artifacts (e.g., posts in a discussion forum), or types of online learning behaviors (e.g., content analysis with codes representing knowledge-building behaviors), depending on the specifics of target research questions (Dado & Bodemer, 2017). A link between two entities might connect entities with identical types (e.g., a link between students reveals a student communicates with the other one) or entities of different types (e.g., a link between a student and an activity reveals that the student participates in the activity). Admittedly, the definitions of nodes and of relational links play a vital role in the interpretations of analytical outcomes (Fincham et al., 2018). Some previous SNA research in CSCL has primarily focused on humans as nodes and communication between humans as links in the network (e.g., networks regarding communication-based interactions between students; Ouyang, 2021; Saqr et al., 2020). Some CSCL studies have employed SNA to address the temporality of discourses by utilizing qualitatively extracted parts of the participants’ discourse as nodes and interactions among the discourse of individuals as links (e.g., Swiecki et al., 2020; Zhang et al., 2021). On the other hand, Zhu et al. (2016) apply transition networks to analyze the process data generated from an online environment among individual students. They define actions (i.e., click a button) as the nodes and sequential transitions of actions as links to examine patterns of action transitions in individual students’ problem solving processes. The analysis of the previous research discussed above has relied on identical SNA theories, even though their research questions are tremendously different.

Based on Zhu et al. (2016) and the fact that paired students’ sequential actions are intertwined and their contributions build one upon another (Hesse et al., 2015), we applied SNA methods to explore the patterns of action transitions in an online CPS assessment environment in a dyad setting with a human‒human approach. This fills a research gap in which there is scant research investigating paired students’ action transitions in computer-based CPS tasks. We created networks in which actions (e.g., drag to move an object, type messages) are nodes and the transitions of two actions are the links representing the order of the actions. We refer to these networks as transition networks (Zhu et al., 2016). Transition networks explicitly illustrate how students’ future actions are often influenced by their prior actions in CPS. Further, SNA measures are vital factors for assessing the formation and transition of entities (i.e., actions in our study) (Ouyang & Scharber, 2017), and SNA is often combined with other research methods to address the social and cognitive aspects of CSCL (Ouyang, 2021). Therefore, in addition to calculating SNA measures (see the Methods section) of transition networks as a way to represent patterns of action transitions, we also highlighted the distinct patterns of action transitions in CPS processes across differentiated pair compositions in terms of CPS social and cognitive skills. The process data from the present study are defined as a sequence of actions that the pairs conducted during CPS in an online assessment environment. Process data are typically represented by a sequence of actions, and each of these actions belongs to a finite pool of available actions (Zhu et al., 2016). In the ATC21S online assessment environment utilized in the present study, the action sequence was embedded with partial time information recorded by the environment in a way that the order of actions was relevant (i.e., the first action is followed by the second action in the sequence).

Objectives

The purpose of our study was to examine patterns of action transitions conducted by pairs of sixth-grade students in an online CPS assessment environment by utilizing an SNA approach. We also shed light on the relationship between the patterns of action transitions and students’ assessed skill levels of CPS (i.e., social and cognitive skills of CPS). Based on previous studies (e.g., Andrews-Todd & Forsyth, 2020), we hypothesize that pairs including at least one member with high social and high cognitive skills may conduct more actions with a higher frequency of action transitions on average and with a higher tendency to conduct actions for different number of times.

The following research questions were addressed:

• 1. How do individual students’ social and cognitive skill levels vary in CPS tasks? What kinds of pair compositions can be identified in terms of the social and cognitive skills of CPS?

• 2. How do patterns of action transitions differ between pairs comprising diverse social and cognitive skill levels of CPS?

Methods

Participants and procedure

With a convenience sampling method, data were collected in 2019 from 166 sixth-grade students (Mean_age = 12.60, SD_age = 0.33, female = 91, 54.82%) from 12 classes within 5 schools in a Finnish urban municipality from which we obtained permission to conduct research. Students participated voluntarily in our study with the option to terminate their participation at any time if they wished. All student participants and their guardians filled out a consent form. We utilized a computer-based assessment environment developed in the ATC21S project at the University of Melbourne. In our study, a series of four game-like tasks (i.e., Laughing Clown, Sunflower, Hot Chocolate, Olive Oil) was administered to the students. The tasks were based on the hypothetical-deductive approach that focuses on generic skills. The students were randomly assigned to work in pairs (female pairs: 27, 32.53%; male pairs: 19, 22.89%; mixed-gender pairs: 37, 44.58%); each student in a pair was assigned a role of “A” or “B” (Student A or Student B). Before the tasks started, the aims of the study and practical issues of using the assessment environment (e.g., how to log in the assessment environment) were introduced. Each student collaborated synchronously with their partner, each using a laptop, with each student being located in a different room. The participants were told that they would solve the problems together with their assigned partners and that they could talk with one another through typing texts in the chat window. They could also click buttons and drag to move objects on their computer screens. The participants worked through the assessment tasks in a fixed order (i.e., Laughing Clown, Sunflower, Hot Chocolate, Olive Oil), and they were not allowed to proceed to the next task without both members of the dyad having clicked the “Finish” buttons on their screens. Moreover, once they had finished a task, they were not allowed to return to it. The participants had the option to leave the tasks by not solving the problem, and no time limit was imposed.

CPS tasks

Laughing Clown is a symmetric task (i.e., both participants had the same resources and screen views) and is the simplest of the four tasks. In this task, a clown machine and 12 balls are shown to each paired partner, and students are required to first understand how a clown machine functions in general so they are able to discover whether their clown machines work in the same way. The Sunflower task is also a symmetric task, requiring dyads to mix two plant foods to maximize the height of the plant. The goal of Hot Chocolate is to maximize profits and sales in Europe by utilizing information related to recipes and markets. The Hot Chocolate task is asymmetric, in which dyads have different resources and screen views. For the details of the tasks, see Griffin & Care (2015).

The Olive Oil task

SNA on the action transitions in CPS was based on the data from the Olive Oil task only. We selected the Olive Oil task for the analysis of transition networks for two reasons. First, being content free with no requirement of prior knowledge and addressing the enhancement of inductive and deductive reasoning skills, the task follows the reasoning procedures required in the Tower of Hanoi problem popularized by mathematician Eduard Lucas in 1883 (Newell & Simon, 1972). The task requires dyads to collaboratively enact sequences of actions to achieve the goal by thinking of the “steps ahead of their current state and work out sub-tasks before acting” (Care et al., 2015, p. 93). From a methodological point of view, this sequential feature is well represented by transition networks (Zhu et al., 2016). Second, the task is structured asymmetrically, in which each student can access different resources (for details of the task, see Care et al., 2015). Such resource interdependence is typical for creating collaborative contexts (Johnson et al., 1998). Therefore, the paired students had to collaboratively work out what kinds of resources were available for them at the very beginning of the path to solve this task. More importantly, the asymmetric design of the task ensured the paired participants had to respectively conduct different actions, which meant the process to solve the problem consisted of actions conducted by both participants. This offers a methodological basis to utilize a transition network to represent the actions conducted by both students within a pair.

In the Olive Oil task, Student A had a three-liter jar, an olive oil tank, a transfer pipe, and a bucket, while Student B had a five-liter jar, a transfer pipe, and a bucket. The objective of the task was to fill Student B’s jar with four liters of olive oil. Both students could type texts in the chat window to talk with one another. The jars can be moved to carry olive oil between the oil tank and the transfer pipe (for Student A) and between the bucket and the transfer pipe (for Student B) by dragging these items with a mouse. Student B, in addition, had a button called “Accept transfer” on the screen for the confirmation to receive the oil from Student A. Consequently, before Student A could transfer oil to Student B, Student B would not be able to do anything to solve the problem except discussing it with Student A in the chat window. Figure 1 shows the resources in the problem space on the two screen views in the Olive Oil task.

Fig. 1

Asymmetrical screen views of Student A (at the top) and Student B (at the bottom) in the Olive Oil task (originally in Finnish with English translation)

Network formation for action transitions in the Olive Oil task

To study the patterns of action transitions, we created transition networks with the sequential process data, including 14,448 log file events conducted by 83 pairs in the Olive Oil task. The order of the events indicated the sequence of the actions (e.g., drag to move an object, type texts in the chat window) that the pairs conducted in the Olive Oil task. It is worth noting that the contents of the students’ messages in the chat window were excluded from the analysis.

We utilized weighted directed networks to represent action transitions; each network represents one pair of the students’ action transitions. The nodes in the network represent actions (e.g., drag to move an object, type in a chat window), and directed links indicate the transitions of the actions when the first action was followed by the second action. Every cell in an adjacent network matrix represents the frequency of action that changed from the action in the row to the action in the column. The nodes of “Start” and “Finish” in each transition network represent the beginning and end of the task. We distinguished participants’ operational actions (e.g., clicking a button or dragging an object with a mouse) from written communication (i.e., students’ typing texts in chat window) and further grouped operational actions into three categories based on the log file data that were automatically generated from the assessment environment (see Table 1): (1) two system actions corresponding to Student A’s proposal and Student B’s acceptance of the transferring oil, (2) four transfer actions that show the completion of transferring different amounts of oil, and (3) other operational actions (actions that are other than those in the above categories). All the operational actions and chats were automatically distinguished by the assessment environment (we merely categorized other operational actions into one group). The reason for separating system actions and transfer actions from other operational actions is that all system actions and all transfer actions are necessary for solving the problem. Table 2 presents the most efficient path for solving the Olive Oil task (for more details, see Care et al., 2016). As shown in Table 2, there are altogether 10 operational actions needed in the most efficient path to solve the task. These 10 operational actions need to be conducted in different amounts of times in a specific sequence showed in Table 2. In addition to the 10 operational actions illustrated in Table 2, chatting between Student A and Student B (i.e., the nodes of ChatA and ChatB) is also necessary to reach the most efficient path to solve the problem, and such written communication within a pair could occur at any time during the task for different numbers of times due to the asymmetric nature of the task. Therefore, there are altogether 12 actions (including 10 operational actions and 2 chat actions) in the most efficient path for solving the Olive Oil task.

Table 1 Categories of nodes in transition networks generated from the Olive Oil task

Table 2 The most efficient path to solve the Olive Oil task

Eighty-three transition networks were assembled corresponding to the actions conducted by 83 pairs of participants. Isolated nodes (i.e., those available actions that the student pairs did not conduct) were added to each network so that all the networks had an identical set of nodes. Thus, network measures (see the Network measures section) could be compared across the networks. We excluded self-loops (Wasserman & Faust, 1994) in all networks (i.e., repetitions of the same action over and over, for instance, type in chat window and click “send” then type again and click “send”) because we were interested in the transitions between different actions during the process. According to our definitions of nodes and edges in transition networks, self-loops were not used in our analysis and have not usually been considered in earlier analogous SNA theories and applications (Wasserman & Faust, 1994).

On the other hand, network visualizations can be useful to map the relations of nodes and to identify nodes that are more or less frequently connected (Poquet et al., 2021). In the context of transition networks, network visualizations provide useful information to identify the frequency of action transitions from a visual point of view. For instance, Fig. 2 shows the visualization of the transition network for all 83 pairs’ action transitions in the Olive Oil task. In addition to written communication, all the operational actions that were needed in the most efficient path to solve the problem in Table 2 were in the network. However, some essential actions (e.g., T3, A47) showed a lower frequency compared with other actions in the most efficient path. This implies that there are not many pairs that were able to recognize the importance of conducting these essential actions in the path to solve the problem, and these actions and related action transitions might represent key difficulties for the dyads.

Fig. 2

Visualization of the transition network conducted by 83 pairs in the Olive Oil task.

(Note: For meanings for the codes of the nodes, see Table 1 and Table 2. The size of the nodes is indicated by the indegree value of the node (indegree = number of incoming links multiplied by the weights). The parameter of indegree aims to identify the differences in the frequency of action transitions in the figure, especially for those actions necessary in the most efficient path but having been conducted less frequently (e.g., T3, A47) in comparison with the other actions. Edge sizes equal the weights of the edges divided by 150. The reason for this division is to avoid an unclear presentation and overlapped visualization of edges because the original weight values ranged from 1 to 668, with a mean of 25.42 in the network. Colors of the edges: Red = action transitions among actions in the most efficient path to solve the problem; Blue = action transitions among actions that are not in the most efficient path to solve the problem; Gray = action transitions between actions that are necessary and not necessary in the most efficient path to solve the problem. The positions of nodes are randomly assigned by the visualization software, meaning that the distance between the nodes does not have any particular significance.)

Network measures for action transitions in the Olive Oil task

Generated from the process data in the Olive Oil task, transition networks are directed and weighted to accommodate repeated action transitions and preserve the sequential nature of the data. To examine the patterns of action transitions, we calculated the set of fundamental SNA measures for the directed weighted transition networks.

The numbers of existing nodes, as a descriptive measure of a transition network, measure the number of actions that the pairs conducted during CPS. The higher the number of existing nodes, the more actions the pairs conducted.

Global measures. To address the global features of transition networks, we examined two commonly used network measures: weighted network density and degree centralization. It should be noted that in SNA these terms are not defined identically to the terms with similar names in general statistics.

Density in a directed weighted network refers to the proportion of the total number of existing edges over the maximum number of possible links. For a directed network with $g$ nodes and t links, the maximum number of links can be $g(g-1)$. It is notable that a link from node ${v}_{1}$ to ${\nu }_{2}$ is not identical to a link from node ${\nu }_{2}$ to node ${v}_{1}$. The corresponding network density is $\frac{t}{g(g-1)}$, which can be between 0 and 1. For a weighted directed network, there are $g(g-1)$ possible links, while all links are weighted by their values ${w}_{i}$. Therefore, the density for a weighted directed network (Wasserman & Faust, 1994) is as follows:

$${D}_{w}=\frac{\sum _{i=1}^{t}{w}_{i}}{g(g-1)}$$

Because the edge weights can be larger than 1, the weighted density value can be larger than 1. The density for a directed weighted network represents the theoretical average strength of the links. In the context of our process data, for one action sequence, the weighted density reveals the average frequency of transitions between two actions, as contingent in this case upon the fixed number of possible actions in the Olive Oil task.

The Freeman degree centralization (1979) is a global measure capturing the variability at the node level in a network. For a network $G$, the degree centralization is as follows:

$${C}_{D}=\frac{{\sum }_{i=1}^{g}\left[\underset{\nu \in V}{\text{max}}{C}_{D}\left(\nu \right)-{C}_{D}\left({\nu }_{i}\right)\right]}{\text{m}\text{a}\text{x}{\sum }_{i=1}^{g}\left[\underset{\nu \in V}{\text{max}}{C}_{D}\left(\nu \right)-{C}_{D}\left({\nu }_{i}\right)\right]}$$

where $\underset{\nu \in V}{\text{max}}{C}_{D}\left(\nu \right)$is the largest value of degree centrality ${C}_{D}\left({\nu }_{i}\right)$for any node in the network. The numerator is the sum of the difference between every node’s degree centrality and the largest value, whereas the denominator $\text{m}\text{a}\text{x}{\sum }_{i=1}^{g} [\underset{\nu \in V}{\text{max}}{C}_{D}(\nu )-{C}_{D}({\nu }_{i})]$is a normalized factor calculated as the largest possible sum of differences over all possible networks with the same number of nodes $g$. In this case, the possible maximum value of degree centralization exists in the network whose structure is a star (with one node connected to the rest of the nodes simultaneously, while the rest of the nodes are not connected to one another). The corresponding maximum is, theoretically, ${(g -1)}^{2}$ for a directed network. In contrast, the minimum value is 0, revealing that all nodes have an identical degree. The higher the value of the degree centralization, the more unequal the degree values are. In the action transition networks from our process data, degree centralization measures the dispersion of attempts given to different actions. A low value of degree centralization shows that a pair conducts actions for the same number of times, while a high value of degree centralization indicates that a pair prefers to implement certain actions instead of others.

Local dyadic and triadic measures. In addition to the global patterns in transition networks, we also examined local dyadic (two nodes) and triadic (three nodes) patterns because they constitute the basic blocks of the network structure and illustrate triadic dynamics at the local level. In the context of transition networks, these dyadic and triadic measures capture the local action transitions, which indicate the extent to which students revisited the previous action (i.e., dyadic pattern) and previous two actions (i.e., triadic patterns).

Reciprocity (Wasserman & Faust, 1994) captures the dyadic structures in networks, and it is the number of mutual links divided by the total number of existing links. For transition networks generated from process data, reciprocity captures the extent to which the participants revisited the previous action right away. In the context of transition networks in the current article, it is worth noting that the meaning of reciprocity is different from that of the self-loop. Reciprocity exhibits the extent to which paired participants reconduct a previous different action, whereas self-loop refers to the previous same action repeatedly conducted (we excluded self-loops in the analysis as mentioned above).

Triad census (Wasserman & Faust, 1994, p. 244) altogether has 16 link combinations made of three nodes. Figure 3 shows 16 possible triadic patterns that are named based on Holland & Leinhardt (1970) and Davis & Leinhardt (1972). The first digit in the names of the triads in Fig. 3 represents the number of reciprocal links in the triad; the second one reveals the number of nonreciprocal links in the triad; the third one shows the number of unconnected links in the triad; and a possible extra capitalized letter at the end represents the orientation of the triad in situations when the first three numbers are identical. The letter U means up; the letter D indicates down; the letter T reveals transitive; and the letter C represents cyclic. In networks generated from the process data, triadic structures show the extent to which the previous two actions tended to be revisited immediately. To obtain better insights into the transitive relationship among the three actions, we partitioned the 16 triadic patterns into five categories (see Table 3), here revised from Batagelj & Mrvar (2001) and Borgatti & Lopez-Kidwell (2014) in the context of transition networks that are generated from process data. Then, we calculated the numbers of 16 triadic patterns existing in each transition network for further analysis. All the global and local measures for transition networks we included (except the null triad that counts the number of isolated nodes) are confirmed to have significant predictive power related to the variability of students’ problem-solving outcomes in an online assessment environment (Zhu et al., 2016).

Fig. 3

Patterns of triad census for directed networks. (Note: Nodes = actions that the participants conducted; Links between nodes = the sequences of action transitions; Arrows at the end(s) of links = directions of sequences in action transitions. A tie with an arrow at only one end between two nodes indicates that the participant changed from conducting one action to another but not the other way; a tie with an arrow at both ends between two nodes means that the participant alternated between a pair of actions.)

Table 3 Categories of triad census in transition networks

Social and cognitive skills of CPS

The social and cognitive skill levels of CPS were identified based on the log file data of all four tasks. The log file data consisted of mouse events (e.g., clicking a button, dragging to move an object) and chat discussion (i.e., typing texts in the chat window) in the task environment. All the actions were recorded in order and time-stamped. The focus of the ATC21S assessment tasks is “the process and quality of problem solving” (Adams et al., 2015, p. 116) rather than the conventional design that relies on attaining a solution as the sole criterion using dichotomous scores. Adopting rubrics and partial credit approaches, students’ social and cognitive skill levels of CPS were scored through automation procedures based on their actions in the log file during the problem solving for all four CPS tasks.

The automation procedure began with the identification of task features matching the elements of Hesse et al.’s (2015) framework (i.e., participation, perspective taking, and social regulation represent social skills, while task regulation and knowledge building represent cognitive skills) from all the tasks administered. This was then followed by the generation of simple rules (see below) to collect data points that were able to represent these elements. The data points were extracted from log files generated by the students’ work in the assessment tasks, consisting of the documentation of each event (i.e., every action conducted by the participants). In particular, the actions observed in the log file data were used as indicators of social and cognitive skills, as defined in Hesse et al. (2015). Such indicative behaviors were then coded into rule-based indicators that could be extracted from the process log file data through an algorithmic procedure similar to the description in Zoanetti (2010), which reports how process data (e.g., action counts) can be interpreted as an indicator of a behavioral variable (e.g., learning from a mistake). These coded indicators were considered the primary data source for the scoring procedure. Each of the scoring algorithms took the coded dichotomous or polytomous indicators as the input and created a corresponding output, defined by the rule for the relevant indicator. For instance, the algorithm would count the occurrences of the event “chat” in the log file data if capturing the number of interactions in a task, and the output for this indicator would be a numerical value representing the frequency of the chat (for more details on the algorithms, see Adams et al., 2015). Then, the indicators were analyzed using Rasch modeling (Rasch, 1960) with two dimensions (i.e., social and cognitive skill levels). The modeling procedure set the average task indicator difficulty to 0, and the difficulty of an indicator was presented as an estimate describing the students’ skill level based on the four tasks. Consequently, the students’ skill levels were identified as higher if they conducted more actions whose corresponding theoretical indicators were more difficult to construct. The students’ social and cognitive skill levels were identified through weighted likelihood estimate scores (WLE; i.e., the estimates on item range of difficulty as enacted by the participant, which was given a measure of item difficulty). Table 4 shows the WLE distribution on the different skill levels of CPS. In practice, the scoring engine, which is managed by the University of Melbourne, automatically coded and scored the log file data, producing WLE scores and skill levels for further analysis and for producing reports for teacher and student use.

Table 4 Range of WLE scores in ATC21S portal corresponding to the social and cognitive skill levels of a bundle of four CPS tasks

Analysis strategy

The data analysis was conducted in R 4.0.2 (R Core Team, 2020). R packages sna (v2.6; Butts 2020), network (v1.16.1; Butts et al., 2020), and GGally (v2.0.0; Schloerke et al., 2020) were mainly applied to process, visualize, and analyze the transition networks with the process data from the Olive Oil task. In particular, we created a transition network for each pair of the participants’ action transitions. We calculated the number of existing nodes, density, Freeman degree centralization, reciprocity, and the numbers of the 16 patterns in the triad census (see Fig. 3) for each network. Analysis of variance, Welch’s tests, and post hoc tests (Tukey and Bonferroni) were applied to the network measures across social and cognitive skill levels of CPS that were computed based on the four CPS tasks in the ATC21S portal.

Results and discussion

Social and cognitive skill levels of CPS among individuals and pair compositions

Social and cognitive skills of CPS varied among the individual students in the four CPS tasks. The social skill levels of 153 (92.17%) participants were relatively high (i.e., at levels 4, 5, 6), whereas the cognitive skills of 120 (72.29%) participants were at the low levels (i.e., at levels 1, 2, 3). Table 5 shows the frequency of each social and cognitive skill level of CPS within the 166 individual participants.

Table 5 Frequency of social and cognitive skill levels within 166 individual participants generated from a bundle of four CPS tasks

We categorized all participants into four theoretically based individual CPS profiles (Andrews-Todd & Forsyth, 2020). In particular, levels 1, 2, and 3 in social and cognitive skills (see Table 4) were grouped into the low dimension of skills, while levels 4, 5, and 6 were classified as the high dimension. In our data, there were three CPS groups for 166 individual participants: high social and high cognitive skills (HH), high social and low cognitive skills (HL), and low social and low cognitive skills (LL). Because the four CPS assessment tasks were administered in pairs, five pair compositions of CPS skills were identified among the 83 pairs of participants for further analysis.

When the paired students exhibited the same levels of skills, we used “active” or “passive” to represent the high or low level of social skills, respectively, and used “high-performing” or “low-performing” to represent the high or low level of cognitive skills, respectively. “Compensated” was applied to demonstrate the pairs in which two students had different levels within the same skill (e.g., one member had high and the other had low social or cognitive skills). Accordingly, the five pair compositions of CPS skills were identified as active high-performing pairs (both participants had high social and high cognitive skills), active compensated pairs (both participants had high social skills while each member represented either a high or low level of cognitive skills), active low-performing pairs (both participants demonstrated high social and low cognitive skills), compensated low-performing pairs (each member represented either high or low social skill levels while both exhibiting low cognitive skills), and passive low-performing pairs (both participants showed low social and low cognitive skills). Almost half of the pairs (n = 40, 48.19%) were active low-performing pairs, whereas passive low-performing pairs accounted for the fewest pairs (n = 2, 2.41%). It is notable that there was a set of 32 pairs in which each pair included at least one member with high social and high cognitive skills (i.e., active high-performing and active compensated pairs), whereas there were 51 pairs that did not have one member with high social and high cognitive skills (i.e., active low-performing, compensated low-performing, and passive low-performing pairs). Figure 4 presents the frequency distribution of the five pair compositions within the 83 pairs of participants.

Fig. 4

Frequency distribution of the five pair compositions (N = 83 pairs). (Note: HH-HH = active high-performing pairs (n = 14, 16.87%); HL-HH = active compensated pairs (n = 18, 21.69%); HL-HL = active low-performing pairs (n = 40, 48.19%); LL-HL = compensated low-performing pairs (n = 9, 10.84%); LL-LL = passive low-performing pairs (n = 2, 2.41%))

Utilizing a computer-based environment with a human−human approach, Andrews-Todd & Forsyth (2020) cluster four types of individual students’ CPS profiles of social and cognitive levels in a triad setting: high social and high cognitive, high social and low cognitive, low social and high cognitive, as well as low social and low cognitive. Except for the low social and high cognitive profile, we found the same profiles of individual students’ CPS skills as those in Andrews-Todd & Forsyth (2020). The reason why no individual students were assessed as having low social and high cognitive skills might be related to the asymmetric design of the Olive Oil task that deliberately promoted social interactions between paired partners. If dyads did not actively communicate with their partners through chatting, they were not able to proceed further in the task, leading to low instead of high cognitive skills of CPS in the assessment environment. Beyond the individual level, we extended the knowledge of individual students’ CPS profiles to five pair compositions in a dyad setting.

Differences in patterns of action transitions among five pair compositions

In this section, we first present and discuss visualizations of transition networks from action sequences within different pair compositions in order to highlight the action sequences indicative of skillful pairs in the context of the Olive Oil task. Second, we offer the descriptive statistics of network measures in these transition networks. Finally, we present and discuss comparisons across the network measures between transition networks associated with pairs with different skills.

To illustrate the contrast in action transition processes across pair compositions, we visualized two transition networks, namely one conducted by an active high-performing pair and one by a passive low-performing pair; in both cases we chose the pairs who implemented the most actions among pairs that were similar in terms of composition of CPS skills. The comparison was based on the log file data of the Olive Oil task (see Figs. 5 and 6). The active high-performing pair consisted of two members with high social and high cognitive skills, whereas the passive low-performing pair had both members with low social and low cognitive skills. The active high-performing pair in Fig. 5 conducted more actions than the passive low-performing pair in Fig. 6 (i.e., there are more nodes in Fig. 5 than in Fig. 6). It can be seen that the active high-performing pair attempted a greater diversity of paths to solve the Olive Oil task. The active high-performing pair conducted 11 actions (see Fig. 5) out of the 12 necessary actions in the most efficient path (see 10 actions in Table 2 together with the nodes of ChatA and ChatB). In contrast, the passive low-performing pair implemented merely 4 actions (i.e., ChatA, ChatB, S1, and A7; see Fig. 6) that were in the most efficient path to the solution. On the other hand, the active high-performing pair repeated those actions that were not in the most efficient solution path less frequently than the passive low-performing pair (i.e., Fig. 6 has thicker blue edges revealing more frequent repetition between unnecessary actions in the most efficient path when compared with Fig. 5). Moreover, the active high-performing pair conducted a higher frequency of action transition when compared with the passive low-performing pair. Based on the above elaboration on the visualizations, Fig. 5 represents a relatively productive CPS, while Fig. 6 illustrates a relatively unproductive one. These visualizations of the transition networks show that the pair with high social and high cognitive skills of CPS appeared to conduct more productive CPS than the pair with lower skills. CSCL comprises processes of working collectively toward a solution of problems, especially when the path to the solution is unclear (Bause et al., 2018). Compared with unproductive CPS, productive CPS with more trials of different solution paths, and higher frequency of action transitions may better facilitate conducting the actions included in the most efficient path.

Fig. 5

Visualization of the transition network conducted by an active high-performing pair in the Olive Oil task. (Note: For meanings for the codes of the nodes, see Tables 1 and 2; Edge sizes equal to the weights of the edges divided by three. The reason for this division is to avoid an unclear presentation through visualization of overlapping edges; Colors of the edges: Red = action transitions among actions in the most efficient path to solve the problem; Blue = action transitions among actions not in the most efficient path to solve the problem; Gray = action transitions between actions necessary and not necessary in the most efficient path to solve the problem; The positions of nodes are randomly assigned, meaning that the distance between the nodes does not have particular meanings; this is decided randomly by the visualization software)

Fig. 6

Visualization of the transition network conducted by a passive low-performing pair in the Olive Oil task, here following the same symbols as in Fig. 5

In the transition networks generated from the process data in the Olive Oil task, the mean number of existing nodes was 27.8, meaning that 83 pairs conducted 27.8 actions on average during the task. The maximum number of actions that the pairs conducted was 58, while the minimum was 6, indicating a large variation between the pairs. The density of networks appeared to be sparse (M = 1.12%). This result implies that the participants were not likely to attempt every available action to work out the problem (some might have left the task before solving the problem). A low value in the mean of degree centralization (0.09) reveals that there appeared to be no particular focal action that the pairs implemented during the task. The low mean value of degree centralization is related to the asymmetric design of the Olive Oil task, leading to the solution path being unclear for paired students. Therefore, students had to communicate with their paired partners to attempt different paths to solve the problem. The mean of reciprocity was 0.24, indicating that the students revisited the previous one action immediately, at least to some extent. This implies that the students appeared to plan their solution paths with more than one action ahead. In terms of the triad census, the average number of null triads (i.e., three isolated nodes; see the “003” pattern in Fig. 3) turned out to be large (M = 71,881), meaning that there were 71,881 isolated “003” patterns on average. This result validates the low average value of density (1.12%) in the transition networks in which students did not appear to conduct many available actions. In turn, the average number of reciprocal triads (i.e., three nodes forming triadic reciprocity, see pattern “300” in Fig. 3) was small (M = 0.71). This shows that students barely revisited the previous two actions right away on average. That is, the student pairs were likely to attempt another path consisting of other actions to solve the problem rather than revisiting the previous two actions back and forth. Table 6 depicts the descriptive statistics of all network measures of the transition networks.

Table 6 Descriptive statistics of network measures in transition networks generated from the Olive Oil task

In the current study, we utilized students’ social and cognitive skill levels of CPS as the outcome measures to guide formation of pairs based on composition. Based on the results of the ANOVA and Welch’s tests comparing network measures of transition networks across five pair compositions of CPS skills (see Table 7), all network measures were significantly different across pair compositions, except for reciprocity and the number of reciprocal triads (i.e., three actions forming triadic reciprocity, “300” in Fig. 3). The most important result is that the transition networks of pairs having at least one member with high social and high cognitive skills (i.e., active high-performing and active compensated pairs) demonstrated significantly higher mean values regarding the number of actions, average frequency of action transitions, and the tendency to conduct actions for different number of times when compared with the other pair compositions.

Table 7 Results of one-way ANOVA and Welch’s tests on five pair compositions across network measures in transition networks of process in the Olive Oil task

First, pairs having at least one member with high social and high cognitive skills conducted more different actions than other pair compositions. This is further validated by our finding that the number of null triads (i.e., the number of three actions that were not conducted, e.g., “003” in Fig. 3) in the transition networks of active high-performing and active compensated pairs was significantly lower than that of other pairs (see Table 7). More concretely, the average number of actions that active high-performing pairs conducted was twice as much as that of compensated low-performing and passive low-performing pairs. Having at least one member with high social and high cognitive skills in a pair may facilitate the implementation of more actions to attempt different solution paths. In the Olive Oil task, chats and all operational actions in the most efficient path of solution (see Table 2) are not easy to conduct successfully in just one trial. Pairs have to discuss and attempt different paths to access the solution. Pairs having at least one member with high social and high cognitive skills were more likely to obtain the potential solution for the task, leading to productive CPS because they conducted more different actions than other pairs when attempting different paths for the solution.

On the other hand, our finding that pairs with the best skill levels (i.e., active high-performing pairs) conducted the most actions is not in line with the result in De Boeck & Scalise (2019), in which students who implemented more actions appeared to be less successful in PISA 2015. Compared with the human–human approach adopted in our study, the human–agent approach applied in De Boeck & Scalise (2019) provided students with limited opportunities for communication with the computer agent. Therefore, the path to solve the problem was designed to be relatively more structured so that the successful participants in the task did not necessarily need to conduct many actions to attempt different paths for the solution when compared with that in human–human approach tasks. Particularly, the Xandar task applied in De Boeck & Scalise (2019) has four parts (i.e., agreeing on a strategy, reaching a consensus, playing the game effectively, and assessing progress) that address a relatively clear path to solve the problem. For instance, in the first part of agreeing on a strategy, the student is expected to follow the rules of engagement provided, whereas in the fourth part, the agent poses a question about the progress. This structured form of assistance from the environment is useful for participants to figure out the solution path so that they do not need to conduct many actions to attempt different possible paths for the solution. In contrast, pairs representing the best skill levels were likely to conduct the most actions in the Olive Oil task with a human−human approach; this may be because of the asymmetric nature of the task, lack of structured assistance from the environment compared with the human–agent approach, and the restriction-free communication with partners. There could be substantial variations across computer-based CPS tasks in terms of the design of the tasks (e.g., whether the tasks are asymmetric or symmetric with a human–human or human–agent approach; how much and what kinds of the previous knowledge learned in the curriculum are required for the tasks; whether the communication is synchronous or not; whether the tasks are short-term, e.g., the tasks in our study, or long term that last for several weeks or months; or whether the tasks are for dyads, triads, or even more students in a group). To some extent, our results could be applied to those computer-based CPS tasks that are designed similarly to the Olive Oil task (i.e., synchronous short-term tasks that are designed with asymmetric human–human approach for dyads, not requiring knowledge obtained in the previous curriculum).

Second, active high-performing and active compensated pairs showed the highest density values in their transition networks of the Olive Oil task. This indicates that pairs having at least one member with high social and high cognitive skills appeared to change actions more frequently on average than other pairs. The Olive Oil task is asymmetric, emphasizing inductive and deductive reasoning skills. The solution path for the task is not apparent to participants at the beginning, so pairs must type messages for discussion to attempt different actions with the goal of obtaining the path for the solution. Therefore, pairs need to frequently change actions to access the solution path in the Olive Oil task. That is, the high frequency of changing actions in the Olive Oil task may exhibit more productive CPS. In contrast, utilizing a computer-based assessment environment for individual students’ problem solving, Zhu et al. (2016) report that eighth graders with higher efficiency scores exhibited a significantly lower frequency of action transitions on average. It is noteworthy that the actions referred to in Zhu et al. (2016) are all operational actions, excluding chats. The individual students in Zhu et al. (2016) were given basic information about how a hand water pump works, and then, they were asked to fix a pump that was not working properly. With prior knowledge provided and working alone, the students had better performance in Zhu et al. (2016) because they did not need to either attempt various actions to access the solution or communicate with partners during the task; this leads to less frequently changing actions on average. It appears that the results regarding relations between patterns of action transitions (i.e., number of conducted actions, frequency to change actions) and outcomes depend to a great extent on the task design and implementation.

Third, in our study, active high-performing and active compensated pairs had the highest mean values of degree centralization, while compensated low-performing pairs demonstrated the lowest mean value. This shows that pairs having at least one member with high social and high cognitive skill levels tended to attempt certain actions more frequently than other actions in the Olive Oil task. This result is not in line with that of Zhu et al. (2016) addressing individual students’ problem solving in an online assessment environment. Zhu et al. (2016) show that students with higher systematicity scores do not prefer to conduct certain focal actions instead of others, meaning that they conduct actions for almost the same number of times. The difference of findings in our study and in Zhu et al. (2016) may be because of the different designs of the tasks. With manual provided for students in Zhu et al. (2016), individual students merely need to learn the solution routine from the manual and apply such routine to solve the task. Consequently, individual students with higher outcomes do not need to try different solution paths by conducting certain focal actions more frequently (Zhu et al., 2016). However, in our study with human‒human approach excluding additional assistance provided such as the manual in Zhu et al. (2016), the students with better outcomes are likely to focus on conducting some actions more frequently than others when figuring out the solution path with their paired partners. That is, the extent to which pairs choose to conduct certain actions more frequently than other actions may relate to CPS productivity and the quality of their outcomes.

As discussed above, pairs comprising at least one student with high levels of social and cognitive skills appeared to exhibit more productive CPS than other pairs in the Olive Oil task. That is, in the transition networks generated from the Olive Oil task, student pairs having at least one member with high social and high cognitive skills (i.e., active high-performing and active compensated pairs) exhibited significantly higher mean values for the number of actions, average frequency of action transitions, and the tendency to conduct actions for different number of times compared with other pair compositions.

Our result that pairs involving at least one member with high social and high cognitive skills are more productive in computer-based CPS is in line with the finding of Andrews-Todd & Forsyth (2020) showing that having at least one member with high social and high cognitive skills in a three-person group facilitates performance. In addition, Herborn et al. (2017) report that in a human–agent setting, active high-performing collaborators (i.e., individuals with high social and high cognitive skills) exhibited the best performance on a number of indicators (e.g., knowledge acquisition, application, and reasoning of problem solving) in PISA 2015. Consequently, involving at least one member with high social and high cognitive skills is likely to produce better outcomes or be more productive in computer-based CPS than other individual profiles or pair compositions. Compared with other pair compositions, the active low-performing, compensated low-performing, and passive low-performing pairs (i.e., pairs that did not have one member with high social and high cognitive skills) showed less productive CPS, as indicated by the fewer actions conducted, lower frequency of changing actions on average, and greater tendency to conduct actions for the same number of times (i.e., less likely to repeat particular actions) in their transition networks. The relatively low level of cognitive skills that both members had in these three pair compositions may have hindered them in creating additional ideas and a shared understanding (Andrews-Todd & Forsyth, 2020). On the other hand, social interaction among members is considered a major contributor to productive CPS (Hao et al., 2015). Collaborative discussions allow students to enhance individual cognition by engaging in profound levels of knowledge restructure and revision (Liu & Hmelo-Silver, 2010). Computer-based CPS tasks require active levels of social interaction based on high cognitive skills; for instance, the ATC21S tasks that we administered appear to be sensitive to pair composition. Furthermore, the social and cognitive skills of CPS are “not mutually exclusive” (Pöysä-Tarhonen et al., 2018, p. 2); they are intertwined in a way that the outcomes of a CPS task are generally the results of the interactions of these skills (Liu et al., 2015). When there is at least one socially and cognitively more skillful member in the pair, this member is likely to efficiently interact with and cognitively assist the other member to facilitate their partner’s potential contributions.

Although the distinct pair compositions in terms of CPS skills presented diverse productivity in CPS, reciprocity and the number of reciprocal triads (i.e., triadic pattern “300” representing three actions that transit reciprocally in Fig. 3) were not significantly different across the five pair compositions. This result indicates that students were likely to revisit the previous one and two actions immediately to the same extent, regardless of their CPS skill levels. In the asymmetric reasoning task—Olive Oil—students needed to intensively communicate with their partners through writing in chat windows. Their partners were expected to respond to questions and requests right away, regardless of their skill levels. Accordingly, such immediate responses were demonstrated as reciprocity and as the number of reciprocal triads. Thus, reciprocity and the number of reciprocal triads in the transition networks showed insignificant differences across the five pair compositions of CPS skills.

There was no particular pair composition presenting the highest or lowest numbers of all triadic types in the connected triads (see Table 7). The main difference between the connected triads and reciprocal triads is that the former captures the action transitions of non-reciprocal three links within three nodes, whereas the latter represents the action transitions of reciprocal three links in three nodes in the transition networks (see Fig. 3; Table 3). Thereby, the result of no particular pair composition showing the highest or lowest numbers in all connected triads, in a way, validates the insignificant finding in the number of reciprocal triads across pair compositions because the number of nodes in the connected triads and in the reciprocal triad are the same. Significant differences in the numbers of brokerage triads and connected triads across pair compositions could provide useful information for designing tasks and looking for common pitfalls (Zhu et al., 2016) in online problem solving tasks. Analytic methods other than SNA may not offer such information. For example, if a certain action transition in a triadic pattern that is rather frequently repeated, however, does not belong to the path to the solution, this may indicate a potential design problem that needs to be improved in the task or that the participants are showing a common misconception of problem solving in these three actions.

Post hoc tests that compared the network measures in transition networks across the five pair compositions show that differences were found between active high-performing pairs and compensated low-performing pairs (number of nodes: p < 0.001; density: p < 0.001; centralization: p < 0.001; null triad: p < 0.001; numbers of dyadic triads: p < 0.01; numbers of brokerages triads 021D, 021U, 021 C, 111D (for visual structures, see Fig. 3): p < 0.05; number of connected triad 030 C: p < 0.05). Statistically significant differences in the numbers of other brokerage triads (111U, 201: p < 0.05) and connected triads (120D, 120U, 120 C, 210: p < 0.05) were found between active compensated pairs and compensated low-performing pairs. Moreover, multiple comparisons indicate that there was no significant difference in the network measures in the transition networks between the passive low-performing pairs and other pair compositions. This was unexpected because we assumed that passive low-performing pairs, in which both the participants had low levels of social and cognitive skills, might present significant differences from other pair compositions. This result might be because in the present sample, there were only two passive low-performing pairs. A small sample size of passive low-performing pairs might not be representative.

In an online assessment environment, we investigated transitional patterns of sequential actions (e.g., drag and move objects with a mouse, click buttons, type messages) and their relations with paired students’ CPS outcomes, which has not gained much attention in previous CSCL literature (Dado & Bodemer, 2017). We extended Zhu et al.’s (2016) study from individual to paired students, showing that paired students’ patterns of action transitions could also be studied through transition networks. Degree matters a great deal in the mutual influence between actions in CSCL (Stahl & Hakkarainen, 2021), and patterns of action transitions are able to capture such mutual influence to demonstrate the productivity of students’ CSCL. However, some patterns of action transitions (e.g., number of conducted actions, frequency to change actions on average) do not demonstrate identical relations with students’ outcomes in different tasks that are designed with either a human−human or human−agent approach. The relationship between patterns of action transitions and students’ outcomes appears to be task- and approach-oriented in CSCL. In particular, such relations depend on the approach used when designing the task (e.g., human−human or human−agent), what kinds of knowledge and skills are needed for solving the task, and the extent to which structured assistance is available for students before and during the task.

The tasks developed in the ATC21S project aimed at teaching and learning CPS skills. The outcomes represented students’ social and cognitive skill levels of CPS generated from the tasks designed to provide direct feedback to students and teachers in a formative manner. Teachers might be able to utilize this kind of information on students’ CPS skill levels to optimize students’ developmental progression in future CPS practices. Group composition is especially important in computer-based CPS tasks, in which the teachers’ facilitating role is often minimal (Borge et al., 2015). Based on our results on the pair composition of CPS skill levels and their relation to CPS productivity, teachers could assign one student who has high social and high cognitive skills into a pair for future CPS and other CSCL tasks although both members in a pair should ideally have high social and high cognitive skills. As a result, facilitated by the one with high social and high cognitive skills, the student pair may actively integrate each other’s opinions, obtaining and providing useful feedback to one another to enhance the comprehension of the problem solving tasks. This kind of facilitation by a more skillful learning partner may create opportunities but also challenges. On the one hand, there are positive opportunities, especially for a socially and/or cognitively less skillful student in the pair to observe and engage with a more skillful learning partner in a way that each student in the pair can have a relatively productive role in contributing to solving the tasks. On the other hand, a more skillful student can easily compensate for the performance of a less skillful partner. Yet, this may offer fewer opportunities for the latter one to practice the skills and strategies needed in CPS. To minimize this compensatory effect, teachers should offer additional learning opportunities, for instance, more practice and feedback for those students who need more assistance and facilitation in CPS and CSCL tasks. On the other hand, sophisticated information about students’ patterns of action transitions can provide teachers with knowledge regarding students’ productivity of CSCL so that teachers can obtain a comprehensive understanding of students’ CSCL performance. Moreover, triadic patterns of students’ transition networks could offer insights for practitioners and researchers to detect potential pitfalls in task design.

The limitations of the present study warrant some consideration. First, our study relied on a relatively small sample of Finnish sixth graders, and the frequency distribution of five pair compositions was not balanced (e.g., there were only two passive low-performing pairs). Future investigations could recruit a larger sample of students in different age cohorts and possibly also from different nations. Second, our research assessed students’ CPS processes and skills in a cross-sectional setting. Future studies could focus on longitudinal settings that probe the dynamic relationship between students’ CPS processes and skills to be developed over a longer time span. Third, the actions for transition networks generated from log files are usually not from a probability sample. Rather, the actions were not randomly selected to conduct. Thus, transition network data appear to differ from the data collected from conventional item response data in terms of the possibility of generalizing the results to the general population (Scott & Carrington, 2011). Fourth, we merely examined fundamental network measures in transition networks. To better understand the patterns of action transitions in transition networks, future investigations could explore more network measures for transition networks, for instance, betweenness, closeness, and average path length (Wasserman & Faust, 1994). Moreover, we did not analyze the contents of written communications (i.e., contents of students’ typing in chat windows) or students’ thinking processes during the tasks. Future research could combine a transition network analysis with other methods in more deeply analyzing CSCL processes. For instance, transition networks could assist in filtering data and identifying productive or unproductive episodes for more detailed micro-level analyses. With transition networks, the frequency of actions in the solution path could also be analyzed, followed by qualitative content analysis or computational linguistic analysis (Dowell et al., 2019) of the written communication related to those actions to reach a deeper understanding of what makes productive or unproductive CSCL.

Conclusions

CSCL is all about the peer interactions that occur in sequences of actions (including moving objects with a mouse and typing messages for communication; Stahl & Hakkarainen 2021). Within such sequences, the actions mutually influence each other (Baker et al., 2007), and such influences are viewed in the sequential nature. The mutual influence between actions is a matter of degree (Stahl & Hakkarainen, 2021), and patterns of action transitions are thus able to capture such mutual influence and demonstrate the productivity of students’ CSCL processes.

The purpose of the present study was to explore student pairs’ patterns of action transitions and their relation to students’ outcomes, namely, CPS skills, in an online CPS environment. We represented actions as nodes and sequences of actions as links to create transition networks representing action transitions. We also computed fundamental network measures (e.g., average frequency of changes among actions, the dispersion of attempts given to different actions) to study the patterns of action transitions exhibiting the productivity of the student pairs’ CPS. We extended existing knowledge of patterns of action transitions from an individual (Zhu et al., 2016) to dyad setting, demonstrating the productivity of paired students’ CPS. Based on our results, teachers and practitioners should carefully consider group composition when creating collaborative pairs or small groups (von Davier & Halpin, 2013). Our study further underlined that a pair having at least one member with high social and high cognitive skills of CPS is likely to exhibit productive CPS that is characterized by more actions, a higher average frequency of transition among actions with a higher tendency to conduct actions for different number of times when compared with unproductive CPS. It is notable that our results could be generalized to the contexts of synchronous short-term computer-based CPS tasks that are asymmetric requiring human–human collaboration in pairs, without knowledge obtained in the previous curriculum.

Our results contribute to developing pedagogical practices by providing empirical evidence of the importance of being aware of the students’ social and cognitive skill levels of CPS when assigning students into pairs for computer-based CPS tasks. At the time when CPS in a variety of digital environments has become part of the curriculum in many countries (Wise & Schwarz, 2017), we provided a way to study action transitions that demonstrates the productivity of CPS. Accordingly, teachers can benefit from inferences of procedural information from action transitions to obtain a holistic understanding of how the pair or small group collectively conducts a series of computer-based CPS tasks as a whole (Theiner & O’Connor, 2010). Methodologically, the patterns of action transitions in CSCL are challenging to study. Our study offers a novel approach to investigate paired students’ patterns of action transitions by integrating process-oriented research with skill-based assessments.