Introduction

Problem-based learning (PBL) is a student-centered collaborative learning approach that relies on engaging students in stimulating real-life situations (problems). The fundamental philosophy of PBL revolves around students working in small groups to tackle an ill-structured real-life problem with the help of a teacher (Albanese & Dast, 2014). The intention behind PBL has always been to stimulate students to take an active and deep approach to learning, enhance self-directedness, and embolden problem-solving skills (Dolmans et al., 2016; Yew et al., 2011). By doing so, PBL —arguably— has the potential to equip students with the skills needed for their future learning and professional continuous education (Car et al., 2019; Dolmans, 2019; Gewurtz et al., 2016). Whereas PBL originated at the Health Sciences Center at McMaster University over five decades ago, the approach has spread across several disciplines and educational levels (Servant-Miklos et al., 2019).

The large-scale adoption of PBL across universities and schools has resulted in a diverse range of implementations, traditions, and a wealth of research (Servant-Miklos et al., 2019). Such research has studied face-to-face and online PBL with methods that include surveys, interviews, qualitative, and quantitative analysis of interaction transcripts (Car et al., 2019; Dolmans et al., 2005; Servant-Miklos et al., 2019; Ünal, 2019; Yew & Goh, 2016). Nonetheless, existing research has mainly relied on counts of interactions or interaction types with less attention to temporality. As a learning process, PBL unfolds in time, where the sequence of students’ interactions could influence each other, stimulate, or curb productive collaboration (Sobocinski et al., 2017; Visschers-Pleijers et al., 2006). Given the importance of temporality and the paucity of existing temporal PBL research, our study takes a process-oriented approach to bridge such a gap. We take advantage of the latest advances in temporal analytic methods to explore the process of students’ interactions in online PBL. We study the sequential patterns of interactions and how they unfold over time, find which sequences are associated with better performance, and determine whether sequences are more predictive of a higher performance as compared to frequencies (counts of codes).

The following sections of the introduction provide a concise background on online PBL, the dynamic PBL process, and the temporal methods implemented in our paper. We then review the theoretical underpinnings of PBL, which grounds our research as well as the related literature. The introduction concludes with a formulation of the motivation for our research and the specific research questions addressed in the remainder of the paper.

Online PBL

Online PBL —also referred to as e-PBL, technology-supported, technology-enhanced, digital, or blended PBL (the context of our study)— is a term that describes a PBL process where the internet is used as a medium to run the PBL process solely or partially (Car et al., 2019). Interest has grown in online PBL with the advent of the internet, where a wide-scale adoption of technology has been observed across the educational domain (Valtonen et al., 2022) and PBL was not an exception (e.g., Oliver & Omari, 1999). Several scholars sought to capitalize on the potential of computers in education, building on flagship examples in the field of computer-supported collaborative learning (CSCL) that have been demonstrated to provide an environment for students in which to interact, exchange ideas, and collaboratively build knowledge in groups (Baker & Lund, 1997). CSCL offers students a platform where they can further develop their argumentation skills, improve their critical thinking and reflection (Lu et al., 2010; Tseng et al., 2008), as well as find, curate and share knowledge and learning resources. This is particularly valuable in complex tasks that require the integration of multiple resources, as is the case in PBL (Jeong & Hmelo-Silver, 2016). CSCL allows students to expand their thinking and enrich their understanding as they are exposed to the diverse perspectives of others (Lu et al., 2010).

Most scholars agree that collaborative learning, including online PBL interaction processes and regulation of collaboration, are time-dependent (Hadwin et al., 2017). As Molenaar and Chiu (2014, p. 138) stated, “the collaborative learning process is a sequence of events, in which earlier events influence subsequent events. By relating and engaging with one another, students co-construct ideas, elaborate, and integrate them with their own knowledge and understanding”. It is therefore imperative to use temporal methods to study the processes of interactions in online PBL. Among such methods, sequence mining —which we use in our study— is a method for the analysis of time-ordered data —for example, coded interactions. The method includes a rich repertoire of visualization, clustering, and statistical analysis. Such resources make sequence mining a popular technique in education across several domains. Consequently, sequence miming has been used to study students’ learning strategies (López-Pernas et al., 2021; Matcha et al., 2019), engagement (Saqr & López-Pernas, 2021), and behavior (Kinnebrew & Biswas, 2012). Differential sequence mining is often performed to find behavior patterns (i.e., clusters of sequences), and to study how such patterns explain students’ approach, strategies, or performance (Kinnebrew & Biswas, 2012; Matcha et al., 2019; Saqr et al., 2023).

Process mining is another related analytic method that we use to analyze the temporal changes of interaction patterns. The main objective of process mining is to discover, visualize, evaluate and enhance perception and understanding of a given process (van der Aalst, 2012, 2018). The process under investigation is commonly represented as a visual map created by calculating the frequency of activities and the transitions between them (van der Aalst, 2018). The ease and flexibility of process mining has made it a popular tool for researchers to use as they study, for example, the different aspects of the SRL process (Reimann, 2009), compare strategies among learners (high vs. low achievers) (López-Pernas & Saqr, 2021; Matcha et al., 2019; Peeters et al., 2020), or map the collaborative process (Reimann, 2009).

Theoretical grounding of PBL

The birth of PBL represented a major departure from traditional learning methods that were prevalent at the time. As an active approach to learning, PBL emphasizes students’ self-directedness, working in small groups as well as contextualized learning. Thereupon, several frameworks and learning theories have been used to explain learning in PBL. Since our study analyzes students’ co-constructed interactions within collaborative groups and how students regulate their learning, three theoretical frameworks are relevant to our study: constructivism, collaborative learning, and self-regulated learning (SRL) (Dolmans, 2019; Gewurtz et al., 2016; Hendry et al., 1999). As a constructivist approach to learning, students in PBL build knowledge through interactions with the world around them. Understanding is enhanced through interactions with peers as well as the environment, culture, and the context in which they learn (Lu et al., 2010; Marra et al., 2014; Savery & Duffy, 1995). The social (learning) environment is essential to the development of the learners' knowledge, wherein the learners can test their understanding of the studied phenomenon and interact with peers to enrich and extend their knowledge (Marra et al., 2014; Savery & Duffy, 1995).

PBL can be explained through the lens of self-regulated learning and, in particular, self- and socially-regulated learning (Hadwin et al., 2017). To engage in SRL, students have to use effective planning and goal-setting skills, become self-directed learners, and engage in their learning tasks in order to achieve their pre-set goals (Winne, 2005). PBL emphasizes students' agency and ability to direct their own learning as well as the need to use meta-cognitive skills to monitor and adjust their behavior in order to learn (Sandars & Cleary, 2011). As a socially regulated collaborative process, students have to work together, interdependently, on a shared task within a small group (Hadwin et al., 2017; Loyens et al., 2008). Social regulation involves joint planning and goal setting, role negotiation, contribution to the motivational, cognitive and metacognitive group processes as well as shared contribution to the monitoring and evaluation of group activities and adaptation (Hadwin et al., 2017; Malmberg et al., 2022).

Review of related literature

The relational aspects of interactions using social network analysis (SNA) have received significant attention in the PBL literature (e.g., Rienties et al., 2012; Saqr et al., 2020a). For instance, SNA research has investigated predictive analytics (e.g., Tempelaar et al., 2015), temporal relational patterns with temporal networks (Saqr & Nouri, 2020), as well as diffusion of knowledge in PBL contexts (Saqr & Viberg, 2020). Some SNA studies examined how students’ contributions relate to scaffolding, or how students’ motivation and participation are associated with performance (Giesbers et al., 2014). Notable also is the research that clustered PBL students according to their dispositions (Tempelaar et al., 2018), temporal engagement patterns (Saqr & Nouri, 2020), or SNA centralities (Saqr & Viberg, 2020).

Another line of research employed qualitative coding of students’ interactions to study the online PBL process. One example is the work by Ioannou et al. (2016), who analyzed the frequency of coded Facebook interactions such as discussions, questions, comments, and off-task interactions. Yet, such studies have mostly relied on counting and have not taken into account the full potential of the temporal nature of the time-stamped interactions. Among those few existing temporally-focused examples, Thompson et al. (2013) used Markov transition graphs for the analysis of collaboration between participants in a design task (PBL and case-based reasoning). Another study by the same authors (Lajoie et al., 2014) analyzed the patterns of interactions using Community of Inquiry (CoI) as a theoretical framework (Garrison et al., 1999). The authors found that student discourse exhibited a social as well as a cognitive presence that positively increased from one session to the next. In a subsequent study in a similar PBL context, Lajoie et al. (2015) examined how students adapt as a result of their monitoring. Their findings indicate that students followed the basic phases of self-regulation (forethought, execution, self-reflection) initially, whereas, by the subsequent session, learners were more adaptive, as the discussion addressed themes within each phase in a non-linear way. Moreover, positive social interactions (e.g., encouraging contributions from others or sharing information) encouraged cohesion within the group about the group’s next steps. Molenaar and Chiu (2014) studied how the sequential and relational patterns of students’ interactions influenced subsequent low cognition activities (e.g., reading) and high cognition activities (e.g., generating ideas) during collaborative learning in a primary school using structured and problematized scaffolds. They found that monitoring the learning process often resulted in subsequent low cognition activity and that planning can facilitate transitions between low and high cognition activity states. Moreover, positive relational activities (e.g., confirmation from others) led to subsequent low cognition states, and disagreement or denial led to subsequent high cognition states.

Whereas sequence and process mining —the methods used in our study— have been extensively used to study the temporal aspects of students’ behavior, often through the lens of SRL (Matcha et al., 2019), they have been rarely implemented in CSCL (Lämsä et al., 2021). Nonetheless, a few CSCL examples exist that primarily focus on comparing high and low achievers. Sobocinski et al. (2017) examined the temporal sequences of high and low performing students. The authors found that process models offered a distinction between high and low performing students compared to frequency-based analysis. Song et al. (2022) used process mining to map students’ collaborative problem-solving. The implemented process mining helped visualize students’ collaborative patterns, reveal the differences between high and low achievers as well as offer a potential guidance for scaffolding. Peeters et al. (2020) used process mining to study students’ interactions discussing an academic writing task and, in particular, how high achievers use text composition, argumentation and organization to initiate their interactions.

Research aims and questions

There is a large consensus on the central role of time in the learning process. In CSCL at large, and in online PBL in particular, temporality and temporal methods remain largely unexplored and underdeveloped. Therefore, it is timely that a process-oriented approach is applied to understand how students’ collaboration evolves (Chiu & Reimann, 2021; Häkkinen, 2013): How does online PBL unfold over time? When, and for how long does a group engage in collaborative discussions? What interactions trigger or curtail such a productive discussion? Which types of interactions trigger cognitive interactions? For example, does disagreement follow argumentation and, if it does, to what extent? Exploring such sequences, dependencies and probabilities would shed light on the process of online PBL (Chiu & Reimann, 2021). Similarly, little is known about which sequences of interactions if any might predict higher achievement. This study aims to bridge such a gap by implementing the latest advances in temporal learning analytics to analyze the sequential and temporal phases of online PBL across a large sample of qualitatively coded interactions. The research questions of this study are therefore as follows:

  • RQ1: What are frequent temporal sequential patterns of the online PBL process at the group level?

  • RQ2: Are there distinct patterns of PBL sequence episodes (individual interaction sessions) and, if so, what are their characteristics, and how do they unfold over time?

  • RQ3: To what extent are the frequencies of interaction types associated with better performance?

  • RQ4: Which temporal sequences of interaction types are associated with better performance and how do sequences compare to counts of interaction types in terms of explanatory power?

Context: A blended PBL curriculum

This study has been conducted within a blended PBL Head and neck course (four course offerings: two iterations across two campuses) at Qassim university. The course is offered to first-year medical students. The number of students in each course was 53, 53, 48 and 46 respectively. The mean age of the students was 18.2, 18.1, 18.3, and 18.1 respectively. The difference in age between the four courses was statistically insignificant and very small (F(1, 198) = 0.39, p = 0.535; Eta2 = 1.95e-03, 95% CI [0.00, 1.00]). A chi-square test evaluating the difference in gender proportion among the four different courses was statistically insignificant X2 = 1.9804, df = 3, p-value = 0.5765. The course was identical in all the offerings using the same PBL problems, same course design, and the same course organizer, and lasted for around six weeks. However, only four weeks had PBL sessions, each lasting a week (i.e., the first and last weeks of each course did not have PBL). The Head and neck course is integrated, meaning that it includes different subjects, including anatomy, physiology, histology, and pathology of the head, neck, and brain. The course includes lectures, seminars, practical sessions, and blended PBL. All lectures and other course modules are synchronized with the problem objectives. For instance, within the same week when students had a problem regarding the cerebrospinal fluid, they had lectures discussing the anatomy, physiology, and pathology of the cerebrospinal fluid. The PBL is a modification of the seven jump approach described in detail by Wood (2003), where the PBL interactions are designed to be online, embedded within the Moodle Learning Management System (LMS) discussion forums. For the PBL, students are divided into small groups (8–10 people) with an assigned tutor and are given an open or ill-structured problem each week. Students begin by reading the problem in a two-hour face-to-face session, clarifying the terms, and identifying the learning objectives. By the end of the first two hours, collaboration starts online for the entire week. The decision to move the collaborative interactions online was to harness the affordances of CSCL to facilitate information sharing, knowledge building and co-construction, improvement of critical thinking, and argumentation. Furthermore, asynchronous interactions give students more time to search for information, augment their understanding, and share their perspective with peers, as well as learn from peers’ perspectives. The written nature of the interactions allows students to re-read and more thoroughly process the complex points at their own pace. By the end of the week, students meet with the teacher to conclude and reflect on the process. As such, most of the PBL process (apart from initial reading and setting discussion goals) and all the interactions around the problem objectives are online by design. Since our captured data and analysis use the online PBL component of the blended PBL, we refer to it as online PBL for simplicity throughout the manuscript. See Fig. 1 for a graphical representation of the PBL process.

Fig. 1
figure 1

The blended PBL process where students start with setting goals in a short face-to-face session, and discuss the problem online throughout the whole week, and conclusions by the end in another short face-to-face session

Full size image

Students’ final course grades are used to measure their performance, and thus have two components: (1) the total exam grade, which is composed of a multiple-choice exam, essay questions and a practical assessment (80%), and (2) continuous assessment of participation in the learning activities, including seminars, class attendance, active class participation and PBL attendance online and offline (20%). The PBL collectively (participation in the online and offline activities) accounts for one fourth of the continuous assessment grade (i.e., 5% of the total grade).

Methods

Implementation

Students' posts from four consecutive offerings of the course were retrieved from the LMS, tabulated, and coded by two researchers. The coding was based on the widely used coding scheme by Visschers-Pleijers et al. (2006), which was specifically created to code the process of PBL interactions, as well as the coding scheme by Yew and Schmidt (2009), designed for the analysis of constructive collaborative interactions in PBL. The coding by Visschers-Pleijers et al. (2006) covers learning-related interactions (e.g., argumentation, questions, and disagreements) and procedural interactions related to organizing group task and off-task interactions. The coding scheme was extended based on the work of Yew and Schmidt (2009) to include information sharing, discussion, and argumentation interactions. The coding scheme was simplified by combining similar categories together, for example, “Opening question” and “Alternative question” into “Question”; “Argument” and “Counter argument” into “Argue”. Since the focus of this study is on the students, we combined teachers’ codes that were initially coded as “Guiding”, “Facilitating”, or “Giving feedback” as “Facilitating”, similar to Molenaar and Chiu (2014). Codes that reflect co-regulation include Organize, Question as well as starting discussions with Share or Discuss, which prompt others to engage, respond or debate. Codes related to shared regulation are Evaluation, Question, Agree or Disagree, where students evaluate their approach and discuss how it aligns with goals. It is necessary to re-iterate what Hadwin et al. (2017) emphasized: when only conversation is coded, the boundaries between coregulation and shared regulation may be blurred. The full coding scheme, with description of the codes, definitions, and abridged examples are detailed in Table 1. The two researchers initially met several times to agree on the coding scheme and the definition of each category. A total of 5% of the data posts were coded by both researchers. Then, the two researchers met, discussed differences, and reached a consensus. The Cohen’s Kappa for interrater reliability was 0.91.

Table 1 Coding scheme for the PBL forum data used in the study, with abridged examples
Full size table

Data analysis

Our study follows the conceptualization of CSCL as a time-dependent process that unfolds over time (Hadwin et al., 2017). While temporal aspects have been previously explored as durations and rates and have been studied under several frameworks (Cress et al., 2021; Häkkinen, 2013), our study embraces the view that order and sequence matter (Chiu & Reimann, 2021; Reimann, 2009; Ritter et al., 2007). In doing so, we take a process-oriented approach to the temporal data, in contrast to the “code and count” approach (Oshima & Hoppe, 2021; Reimann, 2009; Suthers, 2006). We concur with the view of (Chiu & Reimann, 2021) and their methodological approach, where the authors explained in great detail why sequence methods are needed and why a stochastic (as opposed to deterministic) view of sequences is more true to the dialogical analysis. This view enables the exploration of temporality and extends our understanding of the process of CSCL. Exploring such ordered sequences, accounting for the dependencies between sequences of events and calculating the probability that an event follows the other (e.g., disagree follows argue) requires appropriate methods (Chiu & Reimann, 2021). As such, the methods used in this study, namely, sequence mining, stochastic process mining, and regression models, follow the suggested methods of Chiu and Reimann (2021).

To answer RQ1 and RQ2, two analytical methods were selected: sequence and process mining (Chiu & Reimann, 2021; Häkkinen, 2013; Molenaar & Chiu, 2014). Sequence and process mining have been established in the literature to model the temporal patterns of learners’ behavior (Bogarín et al., 2018; Matcha et al., 2019; Saqr et al., 2023). Sequence mining is particularly useful in modeling sequences of events, visualizing the timeline of the unfolding of such events, and offers a rich toolset for statistical analysis and clustering (Kinnebrew & Biswas, 2012; López-Pernas & Saqr, 2021). Sequence mining was performed by chronologically ordering the coded forum contributions by their associated timestamps. The time-ordered, coded interactions were used to build a state sequence object (the sequence file format) using the TraMineR R package (Gabadinho et al., 2011). TraMineR is an open-source R package that has a rich repertoire of visualizations as well as statistical and modeling functions for sequence analysis. Building a sequence requires a time epoch, which is the unit of analysis, often referred to as a session or group of events that occur in close temporal proximity (Gabadinho et al., 2009). For the analysis and modelling of the temporal unfolding of interactions at the group level (RQ1), the time epoch was a whole week of interactions, which is the full duration a group takes from start to finish of a problem discussion. The case ID was the group ID, and therefore, every sequence object is a time-ordered sequence of interactions in a given group. The state sequence object was visualized using a distribution plot, which shows the proportion of each coded interaction that occurred at each time point and, therefore, is suitable for showing the temporal distribution of interactions across time. An index plot was also plotted for each sequence to show the sequence of coded interactions in each group (Gabadinho et al., 2011).

For the analysis and modelling of the temporal unfolding of interactions at the student level (RQ2), the time epoch was considered as the session, that is, a sequence of successive interactions by the same student with less than 20 min (the 85th percentile of the dataset) of inactive period between them (Fincham et al., 2019; Saqr & López-Pernas, 2021), and the case ID was the student ID. Differential sequence mining is often used in educational settings to identify similar patterns of behavior and, therefore, it was performed to identify similar patterns of interactions. That is, patterns of students’ sequences of interactions that are homogenous and similar to each other, yet different from other patterns (e.g., Fincham et al., 2019; Kinnebrew & Biswas, 2012). Differential sequence mining was performed using Agglomerative Hierarchical Clustering (AHC) based on similarities/dissimilarities using the Longest Common Prefix (LCP); in other words, clustering together sessions that have a similar start. LCP was chosen based on the best Average Silhouette Width (ASW) and R-squared (R2) (Gabadinho et al., 2009).

We complemented the aforementioned analysis with a stochastic process analysis that enables the visualization and estimation of transition probabilities between coded interactions. That is, for instance, the probability that a discussion is followed by an argument; which interactions follow an argument and what their associated transition probabilities are (Chiu & Reimann, 2021; Reimann, 2009). While several process mining algorithms exist (Bogarín et al., 2018), we have opted for a process mining approach based on Markov models implemented in the pMineR R package (Gatta et al., 2017a) as recommended in past work (Chiu & Reimann, 2021; Reimann, 2009). Markov models are particularly useful when assessing temporal processes and transitions (Gatta et al., 2017b). Compared to other algorithms that are based on frequencies, Markov models make stochastic decisions based on transition probabilities, in ways that are better suited to our research questions and conceptualization of the collaborative interactions as time dependent, sequential, and contingent (Hadwin et al., 2017; Molenaar & Chiu, 2014). We used a threshold of 0.1 such that transitions that accounted for less than 10% of the total count were omitted. In pMineR, the algorithm uses First Order Markov Modelling (FOMM) to train and visualize process models. The resulting model was visualized such that interactions are nodes, and the edges are transitions with arrows pointing in the direction of the transitions. The edges are labeled with the transition probabilities. We conducted this analysis both at the group level (RQ1), and at the student session level (RQ2).

To study the relationship between types of students’ interactions (counts or frequencies) and performance (RQ3), we used Multilevel Regression Models (MLMs). MLMs are superior to traditional regression models (e.g., ordinary least squares) when the data are nested or hierarchical (the data were nested in four different courses in our case) and can model this dependency across observations. As a preparatory step for the data to make it comparable across the four courses, the outcome variable was standardized (by subtracting the mean and dividing by the standard deviation). The rest of the variables were gaussianized using a non-paranormal transformation through a novel approach that fits the data to the normal distribution (Zhao et al., 2012). To estimate the MLM, we followed the method described by (Field et al., 2012; Roback & Legler, 2021), which starts by estimating the random intercept model or the unconditional means model, which contains no predictors but only the random effects (course in our case). A random intercept model is commonly estimated to investigate whether there is systematic variation across groups, that is, if there is a need for an MLM to account for the nestedness of the data. Also, random intercepts models were also used for comparison with the estimated models. The random intercept model Pseudo-R2 (fixed effects) was 0, Bayesian information criterion (BIC) was 1556.3, Akaike information criterion (AIC) was 1546.4, and the interclass correlation (ICC) was 0.59. Therefore, an MLM was deemed necessary to account for data clustering. Four MLM regression models were estimated using the maximum likelihood (ML) estimator. An ML estimator is recommended when comparing across models with different fixed effects (Field et al., 2012). The models were estimated using the R package lme4 (Bates et al., 2014). The four models consisted of a full model with the frequencies of all coded interactions and three models using different subsets of the variables: a cognitive subset where the frequency of codes indicating cognitive activities were included, a high-cognitive subset where the frequency of variables representing high-cognition activities (i.e., dealing with the reasoning, critiquing or generation of new ideas or knowledge) were included, and one last subset with the frequency of off-task activities (i.e., socializing and organizing). All models were evaluated with AIC, BIC, and ICC following the recommendation of Bates et al. (2014). The proportion of explained variance was calculated using Pseudo R2 (Nakagawa & Schielzeth, 2013). All variables were tested for multicollinearity using the Variance Inflation Factor (VIF). Only the variables “evaluate” and “agree” had a VIF of 5.77 and 6.63 respectively; still, they were within the acceptable range (O’brien, 2007). All models were also checked for homoscedasticity of residuals and for influential observations (Cook’s distance).

To study the relationship between students’ sequences of interactions and performance (RQ4), we used the temporal patterns of interaction (the clusters of sequences) in an MLM regression model as independent variables and the final grades as a dependent variable. The approach is similar to studies that investigated the association of such sequences with grades using other theoretical lenses (e.g., SRL) (Matcha et al., 2019). The model aims to identify which and to what extent temporal patterns of sequences are associated with better performance. Additionally, the model fit indices (R2 and BIC) are compared with RQ3 models to see which model (i.e., frequencies vs temporal sequences) is superior in explaining the final performance of students.

Please note that all regression models used individual-level data and therefore, all interpretation and inferences are made only at the individual level. This decision was made in order to avoid the interdependence of group-level data on the individual students forming the groups.

Results

Table 2 shows the descriptive statistics of the coded interactions, including mean (M), standard deviation (SD), and total interactions for each type. The total number of students was 204 (in 22 small groups). Four students did not complete the course (their data were removed). Thus, a total of 200 students were included in the study. The regression models included a mean number of interactions per student of 39.26 (SD = 37.60). The most frequent interaction was Discuss, which involved students using information or learning resources to trigger conversation and address shared information. This was followed by Argue, which required reasoning, justifying, or rationally trying to resolve differences of perspectives and issues in a critical discussion. Socialize was next, and somewhat less common were Share and Evaluate. On the other side of the spectrum, the least common interaction was Disagree, demonstrating that students rarely disagreed with one another. In the same vein, Question was also a rather infrequent interaction.

Table 2 Descriptive statistics of the coded interactions
Full size table

RQ1: The temporal dynamics of group interactions

The process mining of the online PBL process (Fig. 2A) suggests the significant types of interactions and how events may follow or possibly stimulate each other. As a co-regulated process, the interaction starts with a co-regulator (a scribe in PBL terms) who discusses the shared objectives and helps the group develop awareness of their task. Thus, a group usually starts with Organize, yet, organizing is distributed among the students, who discuss together the objectives of the problem and share the duties of group discussions, for example, “these are the objectives of our problem”. There is a loop (meaning a reply to Organize with another Organize interaction), for example, “I will take care of discussing the connection between bleeding and blood indices”. It seems that Organize dominates initially, yet it is rarely visited later except when it follows Disagree. As students conclude discussions of learning goals of their collaborative work, they start to enact the task by Discuss, contributing relevant self-formulated pieces of information to trigger or engage in discussions (16%) and Share, posting scientific facts and information that are mostly copied from an online source (15%). After the Organize phase, around 13% of the time, group members start arguing (Argue) the problem by providing information with reasoning and justification, for example, “A normocytic Anemia (normal blood cell size) is probably what we expect when bleeding is acute since the person is otherwise normal”.

Fig. 2
figure 2

Process map (A), index plot (B) and distribution plot (C) of sequences of interactions in PBL groups. In the index and distribution plot, the x-axis represents the timepoints in the sequence of interactions

Full size image

As students proceed, all groups who Share information continue to Discuss with no reasoning involved, describing the information they shared or extending it with their own knowledge. However, students who start the discussion with Argue trigger discussions (Discuss, 43%), Evaluate (14%), continue to Argue (12%), or Socialize, expressing gratitude, or appreciation of an argument, for example, “Thank you for the nice information, I really liked how you explained it”. In other words, initial discussions that start with Argue are more likely to trigger other types of productive discussions and students' appreciation. In the same vein, students are more likely to engage in an argument following Question (32%), Socialize (21%), Evaluate (18%), or Agree (17%). On the other hand, students who start with Share and Discuss do not progress to other types of high cognitive interactions and continue to Discuss (33%), Share (14%), or Socialize (21%). It is worth mentioning that students end their conversations also with discussions.

Questions were likely to trigger an Argue with reasoning (32%) as well as Discuss (26%), offering plain answers with no justification for example, “microcytic anemia results from chronic bleeding”. Disagree was likely to trigger Evaluate (14%), which then led to Agree (39%) or Socialize (26%) and shared feelings or task discussions.

Facilitate interactions by the teacher were largely disconnected from the overall process, which was largely autonomous. Teacher facilitation appears mostly un-triggered as the preceding events were very few, for example, Disagree or Question. As such, teachers gave feedback to students, guidance, or encouragement, which further triggered social acts, for example, expression of gratitude in 24% of the cases. The preceding events indicate that teachers intervened when questioned or attempted to resolve a disagreement. It is worth noting that Argue, Discuss, and Socialize acted as a common final pathways since most interactions ended there.

In the distribution plot (Fig. 2C), we see that Organize takes most of the first two time points (76.5% for t = 1, and 75.3% for t = 2), followed by Discuss (which increases from 11.8% on t = 3 to 22.3% on t = 4), Share (12.9% for t = 3, 40% for t = 4) and Argue (4.7% for t = 3, 18.8% for t = 4). Later and less frequently, Agree and Evaluate appear. The top side of Fig. 2C shows an increasing proportion of social gestures that go with all types of interactions. As time continues, group interactions are mainly centered around Discuss and Argue, with increasing Agree and Socialize, and a decrease in Share extending till the first 40-50th time point for almost all groups and for around 120th for some groups (see Fig. 2B where only some groups continue across the timeline). Put another way, while students initiate with sharing information, they quickly move to discussion, evaluation, and argumentation (higher cognitive levels), paved with a layer of socialization and agreement. We may say that this stage is roughly productive, and that co-regulation was —at least partially— effective. The timeline continues beyond the 120th time point where a pattern of Socialize, Agree, and Discuss dominates, with uneven Evaluate and decreasing Argue. That is, a more social context develops, with less challenging views and more agreement, which helps maintain the interactions in these group for longer sequences. As such, co-regulation derails, and the groups are less into higher cognitive interactions, but more about social bonding and sharing.

RQ2: The sequential patterns of students’ interactions

To explore the sequential and temporal patterns of students’ interactions in the online PBL, differential sequence mining was applied to cluster the sequences of interactions (groups of similar sequential patterns). Process mining was then applied to visualize and describe the resulting clusters. The best number of clusters according to ASW and R-squared was four. These clusters were labeled and are described below.

Cluster 1: Argumentation triggered

The Argumentation triggered cluster (n = 709) contains sessions that always begin with Argue: reasoning, justifying, or rationally trying to resolve differences of perspectives (Fig. 3). The process map in Fig. 3A offers insight into the overall transitions. It shows that Discuss, Question, Agree, and Socialize mostly transition to Argue. Disagree transitions to Evaluate (25%), and later to Agree (18%). The index plot (Fig. 3B) exhibits the diversity in the length of interactions, being only relatively dense in the early few time points. Lastly, the sequence distribution plot (Fig. 3C) shows that the initial Argue is often followed by Discuss (33.7% of the sessions) or Share (26.5%), Evaluate (15%) or Socialize (14%). Later, Discuss and Evaluate dominate with considerable presence of Argue and Socialize in the subsequent interactions with minimal Facilitate. In summary, the Argumentation triggered session is a mix of arguments and other high cognitive interactions for example, Evaluate, Question and Argue on a bedrock of discussion and socialization.

Fig. 3
figure 3

Process map (A), index plot (B) and distribution plot (C) of sessions in the Argumentation triggered cluster. In the index and distribution plot, the x-axis represents the timepoints in the sequence of interactions

Full size image

Cluster 2: Information triggered

Sessions in the Information triggered cluster (n = 423) commonly begin by Share, i.e., contributing objective information to the group forum (Fig. 4). The process map (Fig. 4A) shows the common transitions among interactions. Students are more likely to transition from Discuss to the end of the session (67%), to continue to Discuss (21%), or to Argue (13%). It is worth noting that Discuss was the event that most other events transitioned to. Agree led to Discuss in 100% of all the cases, and most transitions from Argue (75%) and Socialize (67%) were also to Discuss. The index plot (Fig. 4B) indicates that sessions were mostly short, where only 29.3% of sessions continued beyond two interactions. According to the distribution plot (Fig. 4C), the initial Share interaction triggers a Discuss of the provided information (98.8% of sessions). In summary, the sessions in the Information triggered cluster was short, less argumentative, and characterized by mostly low cognitive interactions.

Fig. 4
figure 4

Process map (A), index plot (B) and distribution plot (C) of sessions in the Information triggered cluster. In the index and distribution plot, the x-axis represents the timepoints in the sequence of interactions

Full size image

Cluster 3: Socially triggered

Sessions in the Socially triggered (n = 398) cluster, as the name hints, always begin by Socialize with peers expressing gratitude, feelings, acknowledgment of contribution or social conversations. The process map (Fig. 5A) shows that Socialize is commonly followed by Discuss (62%), Argue (18%), or continued Socialize interactions (20%). As such, social gestures trigger discussions and interactions, which are subsequently followed by further discussions, or arguments. Disagree was not common, yet always led to Facilitate, meaning that the teachers intervened to aid the PBL process and to resolve a disagreement. According to the index plot (Fig. 5B), sessions were rather short in length. The view from the sequence distribution plot (Fig. 5C) shows that, after the initial Socialize contribution, interactions are of varied nature, Discuss being the most frequent one followed by Socialize, Argue, and Agree. In summary, the Socially triggered sessions were short, dominated by discussion, further socializations, and infrequent arguments.

Fig. 5
figure 5

Process map (A), index plot (B) and distribution plot (C) of sessions in the Socially triggered cluster. In the index and distribution plot, the x-axis represents the timepoints in the sequence of interactions

Full size image

Cluster 4: Facilitated discussions

The sessions within the Facilitated discussions cluster (n = 883) have a considerable presence of teachers. The process map (Fig. 6A) shows that Facilitate was followed by continued Facilitate (68%), Socialize (17%), or by the end of the session (15%). The index plot (Fig. 6B) shows sessions with predominantly successive Facilitate actions reflected teachers who access the asynchronous forums to give feedback to several students in bulk. Yet, as the distribution plot (Fig. 6C) shows, Facilitate was not the predominant starting point but rather Evaluate (38%), Discuss (25%), Organize (23%), and then Facilitate (14%); indicating that teachers helped during Organize, Evaluate, and Discuss. In summary, the Facilitated discussions cluster was dominated by teacher-facilitated actions that led students to evaluate, discuss or organize their group work.

Fig. 6
figure 6

Process map (A), index plot (B) and distribution plot (C) of sessions in the Facilitated discussions cluster. In the index and distribution plot, the x-axis represents the timepoints in the sequence of interactions

Full size image

Sequence of sessions

The transitions between the sessions were rather diverse, with Information triggered being the most common one to start with (39%), followed by Facilitated discussions (28%), Argumentation triggered (22%), and Socially triggered sessions, which were the least common starting point (11%). The transitions between clusters had no noticeable pattern. In other words, students use different patterns of interactions and freely transition between such types of interactions (Fig. 7).

Fig. 7
figure 7

Process map of students’ sequence of sessions

Full size image

RQ3 and RQ4: Association with performance

To study which types of interactions —or interaction sequential patterns— are associated with performance, we developed two types of model: a set of models based on the frequency of each type of interaction (with four groups of variables), and a model based on the frequency with which a student is engaged in a sequential pattern of interactions (cluster).

RQ3: Interaction types

Four predictive MLM regression models were developed with interaction types: a full model with all the variables and three models using different subsets of variables: a cognitive subset where only codes indicating cognitive activities were included, a high-cognitive subset where only variables representing high-cognition activities (i.e., dealing with the reasoning, critiquing or generation of new ideas or knowledge) were included, and a last subset with off-task activities (i.e., socializing and organizing). The statistics of all the models are shown in Table 3.

Table 3 Statistics of the MLMs for predicting performance from students’ interactions
Full size table

The full MLM regression model (MAll) included all codes: Argue, Agree, Disagree, Evaluate, Socialize, Organize, Question, Share and Discuss and was estimated to predict the grades using the maximum likelihood estimator. The model included course as a random effect, since data were nested within courses. The model's total explanatory power was substantial (conditional R2 = 0.71), and the part related to the fixed effects alone (marginal R2) was of 0.12. None of the variables had a statistically significant association with the final grades.

Another model was estimated in the same way as MAll with all cognitive interactions (Mcog): Argue, Agree, Disagree, Evaluate, Question, Share, and Discuss. The Mcog model's total explanatory power was substantial and slightly less than MAll (conditional R2 = 0.70) and the part related to the fixed effects alone (marginal R2) was of 0.13, which is slightly higher than the MAll model. The variables that were statistically significant were Argue (beta = 0.36, 95% CI [8.54e-03, 0.72], t(190) = 2.02, p = 0.045) and Discuss (beta = 0.22, 95% CI [0.03, 0.42], t(190) = 2.27, p = 0.024), both with a positive association with grades.

A third model (Mhigh) was estimated with only high cognition variables (Argue, Evaluate, and Question) as predictors, and course as a random effect. The Mhigh model’s total explanatory power was lower than the Mcog and MAll (conditional R2 = 0.65), and the part related to the fixed effects alone (marginal R2) was 0.10, which is lower than the two previous models. The effect of Argue was statistically significant and positive (beta = 0.65, 95% CI [0.36, 0.94], t(194) = 4.39, p < 0.001), and so was that of Evaluate (beta = 0.52, 95% CI [0.24, 0.80], t(194) = 3.62, p < 0.001).

The last model (Moff) was estimated with the off-task variables (Socialize and Organize) and course as random effects. The Moff model’s total explanatory power was also substantial (conditional R2 = 0.69), and the part related to the fixed effects alone (marginal R2) was the lowest of all models (marginal R2 = 0.06). Only the effect of Socialize was statistically significant and positive (beta = 0.55, 95% CI [0.36, 0.74], t(195) = 5.76, p < 0.001).

The interclass correlation (ICC) of the four models was relatively high (0.72, 0.71, 0.67, 0.73) indicating that students were highly clustered: the contextual variation (group interactivity and course implementation) explains a large proportion of the variation in the outcome.

RQ4: Sequential patterns of interaction

We ran another MLM regression to predict the grades with the frequency of times a student has engaged in a certain temporal pattern or clustered sequence (e.g., facilitated discussions, information triggered, socially triggered, and argumentation triggered). The model included the course as a random effect to account for contextual diversity. The results revealed that students who engaged in more frequent clusters of the Information triggered, Argumentation triggered, or Socially triggered sessions were more likely to have higher grades when all variables are held constant. Only the frequency of Facilitated discussions was statistically non-significant.

The model's total explanatory power was substantial and higher than all previous models (conditional R2 = 0.74), and the part related to the fixed effects (marginal R2) is of 0.11. The association of Information triggered was statistically significant and positive (beta = 1.72, 95% CI [0.66, 2.77], t(193) = 3.21, p = 0.002). The association of the Socially triggered cluster was statistically significant and positive (beta = 1.11, 95% CI [0.53, 1.68], t(193) = 3.79, p < 0.001). The association of Argumentation triggered was statistically significant and positive (beta = 1.35, 95% CI [0.78, 1.93], t(193) = 4.67, p < 0.001). The statistics of the model are shown in Table 4.

Table 4 Statistics of the MLMs for predicting performance from students’ sequential patterns of interactions
Full size table

Discussion

The aim of this study was to understand the temporal dynamics of the online PBL process using the latest advances in temporal analytics. Our study applied sequence and process mining both at the group level —to understand the group dynamics across the whole problem discussions— and at the student level —to understand the students’ contribution dynamics across different episodes. We followed such analysis by examining the association of interaction types and sequences with student performance. Our coding and analysis included all interaction types: social (where students express gratitude and feelings or build relations), organization and regulation (where students divide their tasks and co-regulate), as well as cognitive interactions. In doing so, we consider all types of the collaborative regulatory process, an aspect that has started to receive attention in the literature (Hadwin et al., 2017; Kreijns et al., 2013; Molenaar & Chiu, 2014).

The analysis of the group interactions reflects to a larger extent —at least in sequence, yet less so in terms of depth— the scripted PBL process, as it starts with organization, agreement on duties, and discussion of the problem using prior knowledge. The process is aided by effective initial co-regulation of duties and task objectives. Nevertheless, later phases were short and lacked enough depth or high cognitive interactions. After an initial productive stage —at least partially—, students had few disagreements or conflicts, which is a common feature previously described in online PBL (Aarnio et al., 2013; Molenaar & Chiu, 2014). Students asked few questions; when they did, the questions were likely to trigger high cognitive interactions, for example, argumentations, evaluations, and discussions (Chin & Osborne, 2008). Due to the fact that our data captured the online section only, we cannot rule out that some of the interactions or arguments occurred through other channels or face-to-face. Most notably, argumentation, discussions, and socializing formed a common cyclic pathway, where most interaction types started and ended with either. High cognitive interactions kindled high cognitive interactions among others, and low cognitive interactions kindled similarly low cognitive interactions. Put another way, initial co-regulation helped initiate an effective collaboration process, which derailed with the passage of time into a more socially oriented process with sharing, socializing and less conflict.

The analysis of students' episodes seems to confirm the aforementioned group-level findings. We see that argumentation, discussions, and socializing were the de-facto triggers of all interactions. A cyclical pattern ensued between these three episodes, only and less frequently interrupted by the evaluation, questions, and agreement or the scarce lack thereof (disagreement). Therefore, our study does not corroborate the findings of Lajoie et al. (2014, 2015), who reported an increasing cognitive presence and more adaptive self-regulation between the first and second PBL sessions. A possible explanation is that the authors studied a limited sample with only two sessions, while our study involved a larger sample across several episodes. In fact, a systematic literature review by Dolmans et al. (2016) found evidence (with a medium effect size) that PBL enhances learning strategies within longitudinal studies, but not in short-term studies.

Further examination of the sequence analysis emphasizes the importance of social interactions as a continuous backdrop to the stream of interactions. Social interactions served as a “fabric” or a “bridge” over which other types of interactions were acknowledged or appreciated and, later, acted as a catalyst for other cognitive interactions to develop (Vygotsky & Cole, 1978). The prevalence, the reciprocity, and the centrality of social interactions, helped create a space of trust, a sense of relatedness and mutual understanding that motivated students to interact (Kreijns et al., 2013; Molenaar & Chiu, 2014; Wegerif, 1998). Such findings corroborate Lajoie et al.’s (2015) initial observation that social interactions promote cohesive PBL group dynamics. The social dimension of collaboration has been acknowledged in several research frameworks, yet the importance thereof has been commonly overlooked —in the analysis— in favor of a focus on cognitive interactions instead (Hadwin et al., 2017; Wegerif, 1998). Recently, more emphasis has been placed in unraveling how non-cognitive interactions help kindle collaboration (e.g., Saqr et al., 2020b). The MLM regression in our study further confirmed that engagement in social interaction (in the off-task model), or the frequency of engaging in socially triggered episodes of interactions have both been associated with better performance in a statistically significant way. Such results emphasize the importance of social skills, and the need for more emphasis on viewing collaboration as a holistic process that is not only “cognitive” (Hadwin et al., 2017).

The MLM regression models have also shown that the temporal sequences of interaction episodes (clusters) —compared to the frequency of interactions— were consistently positively associated with students’ achievement in a statistically significant way and offered higher explanatory power (R2) as well. All the three non-facilitated clusters, including the socially triggered cluster, were associated with better performance. Such findings contribute to the body of evidence that the temporality of interactions may be more indicative of productive interactions rather than simple counts or frequency (Molenaar & Chiu, 2014). As our data shows, students initiating the interactions were more likely to perform better, even more than those who engage in argumentative high cognitive interactions.

Contributing to the teacher-facilitated sessions was the only interaction episode that was not associated with better performance. The process mining shows that teachers’ interactions —while not few in count— were largely disconnected from other cognitive interactions. Whereas teachers had frequent interactions, they had limited presence (in time and process). The classical PBL process emphasizes the role of the teacher as a guide on the side or a facilitator, who should not heavy handedly guide the PBL process (Hmelo-Silver, 2004). Our results suggest that if high cognitive interactions, clinical reasoning, and critical thinking are the target of the PBL process, online teacher scaffolds may need to stimulate such higher cognitive interactions. In other words, mirroring the same tutoring face-to-face strategy in the online CSCL should not be the goal (Donnelly, 2013; Schmidt & Moust, 1995). It is also worth mentioning that the interclass correlation was high. Interclass correlation represents the unexplained variance that is unmeasured, or due to clustering or contextual variability. Such findings reflect the importance of implementations of CSCL (Roback & Legler, 2021).

Self-directedness, a major tenet of PBL, seems to be exercised by our students unevenly, and mostly to provide resources, or explain with suboptimal critical depth. Recent evidence seem to corroborate this idea, where only a limited subset of students seems to have a collaborative trait across courses (Saqr & López-Pernas, 2022). We concur with Loyens et al. (2008), who concluded that, in order to address uncertainty and anxiety about learning goal achievement, we need to enhance guidance and “reform” how we enact, monitor and perhaps remediate self-directedness. Sandars and Cleary (2011) suggested that the application of self-regulation to problem-based learning can be a potential solution. As the authors note: gaps in self-regulation seem to be more pronounced in low achieving students who tend to over-estimate their skills, knowledge, and performance; consequently, “self” judgment or directedness are not enough. Proactively working to improve self-regulation, to offer feedback and to embolden effective learning strategies in low achieving students is therefore required (Sandars & Cleary, 2011). Such efforts are more relevant when it comes to online learning environments —online PBL in our case— where self-regulation is even more critical (Greene et al., 2011). In summary, online PBL may require different strategies beyond the classic face-to-face strategies, in particular, for students who lack proper self-regulation skills, or whose self-directedness may not be optimal.

While online PBL is essentially a collaborative learning process, there are differences that warrant discussion before generalizing between contexts. PBL and collaborative learning (CL) share several commonalities but also have some differences. A shared goal or a task exist both in PBL and CL. The task is often a problem from everyday life in PBL, but that is not always the case in CL. Additionally, in both PBL and CL, students work in small groups towards accomplishing the shared goal. Interdependence is an essential aspect of both PBL and CL. Nevertheless, the interdependence of the two techniques differs greatly (Davidson & Major, 2014). In CL, structured tasks, social and academic goals are used to encourage positive interdependence, and roles are not often used; while in PBL —especially in face-to-face implementation—, roles are often used along with structuring and rewards (Davidson & Major, 2014). Another essential difference between CL and PBL is that, in the latter, students have to discuss, debate, and argue about possible solutions for the problem.

Limitations and future directions

Our study has some limitations that warrant mentioning. Our study did not include the face-to-face section and therefore, some interactions were missed. Since students were able to meet physically, their interactions online maybe based —at least partially— on their existing relationships. Furthermore, of course some interactions may have happened outside the platform, and are therefore unaccounted for (e.g., Rienties et al., 2019; Tempelaar et al., 2020).

Coding qualitative data is a subjective process that potentially results in some inaccuracies. Yet, we hope that the large number of interactions helps mitigate this concern. Sequence mining is a technique for the analysis of time-ordered interactions. While our analysis on the group level accounts for the interdependence between students, it remains limited on the student level, as it overlooks the relational aspects. We tried to mitigate this concern by offering analysis on the session level (Fig. 7). The distribution plot tends to overestimate the interactions as time progresses; therefore, we included an index plot with every figure that offers a realistic view of the interactions. We also built our interpretation on the process maps and index plots to avoid this concern.

Finally, our results are limited by the context and the design of our PBL process, and interpretation should take such contextual factors into consideration. We have used sequence and process mining as our main methods, yet the temporal relational aspect deserves an in-depth study of how the relational aspects unfold in time. Such aspects may be realized using novel methods such as temporal network analysis (e.g., Saqr & Peeters, 2022).

Conclusions

At the group level, the analysis of online PBL has revealed that the self-directed and largely scripted PBL process is followed in an orderly sequence, yet often lacked enough depth. Teacher facilitation may have helped engagement but not depth or critical thinking. While self-directedness has been the tenet and “the article of faith” within PBL, it was less likely to emerge spontaneously and less frequently sustained across the duration of the task. The findings that high cognitive interactions (e.g., arguments, questions, and evaluations) kindled high cognitive interactions begs the need for problems and tutor guidance to be designed with the goal of facilitating high cognitive functions in mind. Put another way, how we design the online PBL process needs a “reform” to help students to be self-directed and sustain a productive collaborative process. A sequence of interactions was likely to emerge on a social backdrop, which may have helped form a cohesive group. Nevertheless, the longer chains of sequences were dominated by more agreement, sharing and social bonding and rare high cognitive interactions; in other words, the long chains of messages were short of cognitive depth.

On the student level, several types of clusters of sequences of interactions emerged. Here, order and triggering of interactions mattered (e.g., social, information sharing and argumentation). Starting or initiating interactions (even with low cognitive content) showed the highest association with performance as it stimulated others to build on the shared knowledge. The results of the regression models show that sequences of interactions explained variations of grades more than simple counts. Interestingly, the variability of contexts was large pointing to the complexities of context in CSCL as a learning environment and possibly that design and elements (artefacts, teachers, students, medium) vary more than we expected.

Our results have implication for the group formation, PBL problem designers and teachers facilitating group interactions. Groups would benefit from inclusion of students who can trigger argumentative interactions, encourage others to engage and help sustain a focused productive discussion. Online PBL need to be monitored based on quality not quantity or length of interactions. Teachers need to prioritize content over quantity of interactions, for instance, stimulate argumentative interactions rather than simple knowledge sharing. We offered an example of how the modern process and sequence mining methods could help unravel the dynamics of CSCL which was a realization of the methods suggested by Chiu and Reimann (2021). While social network graphs have been implemented in CSCL for decades to represent and support argumentation, we believe that our study provides an initial demonstration that sequence and process graphs could prove worthwhile in following the sequential unfolding of interactions.