Abstract
The importance of learning design in education is widely acknowledged in the literature. Should learners make effective use of opportunities provided in a learning design, especially in online environments, previous studies have shown that they need to have strong skills for self-regulated learning (SRL). The literature, which reports the use of learning analytics (LA), shows that SRL skills are best exhibited in choices of learning tactics that are reflective of metacognitive control and monitoring. However, in spite of high significance for evaluation of learning experience, the link between learning design and learning tactics has been under-explored. In order to fill this gap, this paper proposes a novel learning analytic method that combines three data analytic techniques, including a cluster analysis, a process mining technique, and an epistemic network analysis. The proposed method was applied to a dataset collected in a massive open online course (MOOC) on teaching in flipped classrooms which was offered on a Chinese MOOC platform to pre- and in-service teachers. The results showed that the application of the approach detected four learning tactics (Search oriented, Content and assessment oriented, Content oriented and Assessment oriented) which were used by MOOC learners. The analysis of tactics’ usage across learning sessions revealed that learners from different performance groups had different priorities. The study also showed that learning tactics shaped by instructional cues were embedded in different units of study in MOOC. The learners from a high-performance group showed a high level of regulation through strong alignment of the choices of learning tactics with tasks provided in the learning design. The paper also provides a discussion about implications of research and practice.
Similar content being viewed by others
Introduction
In the era of rapid knowledge growth, “learning how to learn” has been a prominent phrase in the discourse around educational policy and practice (Black et al. 2006). Self-regulated learning skills as important manifestation of the “how to learn” have been identified by much research, as a key factor of learning success (Maldonado-Mahauad et al. 2018; Bannert and Reimann 2012). For a highly open, flexible and autonomous learning context like a MOOC (Massive Open Online Course), self-regulation becomes even more important and critical. Many MOOCs are designed to be self-paced and hold the assumption that learners can self-regulate their own learning (Milligan and Littlejohn 2015). However, when learners were left to study on their own, they always experience difficulties in adequately regulating their learning (Milligan et al. 2013; Azevedo et al. 2010). Therefore, improving self-regulated learning skills is an important task for both MOOC learners and MOOC designers.
Learning tactics and strategies are key elements of self-regulated learning, as activity patterns indicate how self-regulated learners monitor the learning process (Winne et al. 2002). Learning strategies are techniques used by learners to support robust learning (Dunlosky 2013; Rachal et al. 2007). A learning tactic is a micro and fine-grained level concept that can be understood as a sequence or pattern of actions learners perform in a given task within a learning session (Hadwin et al. 2007; Matcha et al. 2019; Jovanović et al. 2017). A sequence of tactics is a strategy (Winne et al. 2002); that is, a learning strategy is how learners use tactics (Derry and Murphy 1986), and strategies are structurally more complex than tactics (Winne et al. 2002). For example in a MOOC context, learners may try assessment before watching a video as a “fast learning tactic” when they encounter familiar content; or they may watch a video before assessment as a “step-by-step learning tactic” when they encounter challenging content. This change of tactic according to learning materials may indicate that learners show a certain level of learning strategy usage and self-regulation. Much literature does not deliberately distinguish between the concepts of strategy and tactic, and often mix or combine them. In this paper, this distinction is necessary to point out, that is, we stress the focus of learning tactics on the level of learning session. The terminology used in this study is shown in Fig. 1, including: trace data, learning actions, learning sessions, learning tactics, and learning strategies.
The relationships between trace data, learning sessions, learning tactics, and learning strategies
As much educational research and practices show that learning tactic is a dynamic concept that can be supported, changed or trained in a deliberately designed learning context (Oxford 1989; Winne and Nesbit 2009). For example, many programs have been developed (Dansereau et al. 1975; McKeachie et al. 1985; Segal et al. 2014) with the aim of helping learners develop their skills in using learning tactics. Even in the state of non-deliberate training, learners are able to gradually build their repertoire of learning tactics and strategies, as “by-products of learning” (Derry and Murphy 1986). Just like the previous example in the MOOC context, learners may gradually start selecting a specific tactic according to the learning content week by week. For example, they can use “fast learning tactic” when they encounter familiar content or use “step-by-step learning tactic” when they encounter challenging content. This gradually formed tactic is a by-product of learning this MOOC. Researching on the choices of learning tactics that learners make during learning is one of the emerging themes in SRL and learning analytics (Roll and Winne 2015).
There are many factors that influence a learner’s choices of learning tactics, including the learner’s initial knowledge (Willoughby et al. 1994; Winne and Hadwin 1998) or experience (Romero et al. 2007) about the task, knowledge about cognition and regulation of cognition (Brown et al. 1982), goal orientation for a task (Pintrich 2000; Howell and Watson 2007) and other motivation elements (du Boulay and Luckin 2016). However, there is also a very important factor that should not be ignored: the learning design of a course or task where learning happens. Learning design or instructional cues are emphasised in the models of self-regulated learning such as the COPES model developed by Winne and Hadwin (Winne and Hadwin 1998). In the COPES model, instructional cues as an important element of task conditions, influence both the tactics that learners apply and cognitive conditions and standards that shape learners’ decision-making (Winne and Hadwin 1998). Instructional conditions also play an important part in shaping the choices made by the learners (Greene and Azevedo 2007). For example, MOOC designers can use a detailed overview of the coming week to help learners form a better judgement about their familiarity with the expectations for the coming week and make better choices of learning tactics (i.e., “fast learning” or “step-by-step”).
A seminal theoretical framework proposed by Rigney (1978) explained the formation approaches of learning tactics with the emphasis on learning design, in which the formation of learning tactics can be understood from two dimensions: “control of orienting task” (student assigned or instructional system assigned) and “explicitness of cognitive strategies” (detached or embedded) (Rigney 1978). The role of learning design in the formation of learning tactics posited by Rigney has extensively been used in research and practice (Morrison and Guenther 2000; Barba 1993). However, few studies have quantitatively studied the links bridging between learning design and learning tactics. According to a recent systematic review of the literature on learning analytics for learning design (Mangaroska and Giannakos 2018), research is missing to measure what learning design decisions affect learning patterns and how can meaningful learning patterns be mapped back to tasks envisioned in a learning design. Especially, there is a lacuna in research that aims to analyse digital trace data in connection to specific course design meta-data (Mangaroska and Giannakos 2018) while taking into account the temporality of learning process (Knight et al. 2017; Bakharia et al. 2016). In this study, we aim to fill this gap by identifying learning tactics and their links with learning design elements. Therefore, we proposed our research questions as follows:
RQ1: Can the analysis of navigation log data from a MOOC context reveal learning tactics that are meaningful in terms of pedagogical intents that are behind learning designs?
RQ2: To what extent learners from different performance levels differ (if so) in the way they choose learning tactics to carry out pedagogical intents that are embedded in learning design?
The notch in the previous literature is largely associated with the lack of research methods and analytic techniques that can enable the study of the links between learning design and learning tactics. The novel contributions of this paper are methodological and empirical. Firstly, it is meaningful to apply a process mining technique to identify tactics usage patterns across learning sessions. In the past, research mainly focused on tactics usage within a learning session. Secondly, this study used a combination of different analytic methods (i.e., clustering, process mining, and epistemic network analytics) to answer the RQs; this combination of the methods offered a new methodological approach. In addition, this work detected empirical meaningful tactics, and also revealed the links between learning design and learning tactics. Practically, this work is also valuable for instructors and designers, and brings awareness and consideration of how their design influences or helps MOOC learners completing the learning tasks. The remainder of the paper starts with a section dedicated to Jim Greer’s work following the Background section which reviews relevant literature. This is followed by the Method section which includes details about the MOOC context, data collection, processing and analysis. The paper then reports the Results and offers a detailed Discussion section that outlines how the results address our research questions. The paper is concluded with the summary of the main results and directions for the future research.
Contributions of Professor Jim Greer to the Field of SRL and LA
We are honoured to contribute to this festschrift dedicated to celebrating the research and life of Jim Greer. In this section, we take the opportunity to outline main influences of Jim’s research on the growing field of learning analytics and specific relevance of his work on the study reported in this paper. We also take the opportunity to acknowledge Jim’s significant mentoring role for many researchers, including the last author of this paper.
Jim was a true research leader in artificial intelligence in education. He is well-known for his pioneering research in student modelling and his role in the development of intelligent tutoring systems (McCalla and Greer 1994; Huang et al. 1991). A specifically important contribution that Jim led was the work on inspectable student models that allowed students to see and possibly control information a learning system would store and use to assist in the learning (Zapata-Rivera and Greer 2004). This contribution has led to conceptualisation of open learner models (Dimitrova et al. 2001; Bull and Kay 2010), and the idea now cuts across several sister fields such as artificial intelligence in education, educational data mining, learning analytics, technology-enhanced learning, and learning sciences. What is important about Jim’s work on inspectable student models is the emphasis on the critical role of interdisciplinarity in artificial intelligence in education. Not only did Jim work on computational aspects of student models with techniques such as Bayesian networks (Zapata-Rivera and Greer 2004), but he also recognised the importance of user interfaces, interaction design, and psychology to enhance learning and user experience (Brooks et al. 2014; Brooks et al. 2014; Brooks et al. 2007) or to change learner behaviour(Orji et al. 2019). This interdisciplinary emphasis rooted in Jim’s pioneering work can be recognised in contemporary trends that highlighted human-centred design as a driving factor for success and impact of learning analytics and artificial intelligence in education (Buckingham Shum et al. 2019; Holstein et al. 2019).
Jim led some of the early work in the use of data mining and machine learning in educational technologies that created foundations for the fields of learning analytics and educational data mining. Some of his initial work in this area was dedicated to the analysis of video recordings of lectures in higher education. One part of that research program involved recognition of significant moments, which was acknowledged as a need for sophisticated approaches to feature engineering in order to train supervised machine learning algorithms (Brooks et al. 2009). While the field of learning analytics has progressed much during the last decade, we have seen insufficient work on video analysis. Given the growing importance of video in education, automated approaches to the analysis of video content require much closer attention in learning analytics. Another part of his research was focused on the analysis of patterns of student interaction with recordings of video lectures with the use of unsupervised machine learning techniques (Brooks et al. 2014). This latter part of his research on video analytics is particularly relevant for the study conducted in this paper that looks at the automatic detection of learning tactics with the use of unsupervised methods. Conceptually, the research direction of the current study is fully consistent with the vision of Jim and his colleagues about the need to automatically detect the ways how learners interact with information in order to improve and personalise their experience, environments in which learning happens, and the quality of learning designs and curricula (Greer et al. 2016). The current study is also related to Jim’s work on student modelling that aims to support students’ metacognition and learning strategies (Mandinach and Greer 1992).
Jim’s work in learning analytics has been inspirational. Although Jim is primarily known for his leadership in artificial intelligence in education, Jim inspired a whole new generation of researchers in learning analytics. The impact of Jim’s work and mentorship was one of the main influences on the decision of the last author of this paper to dedicate his academic career to learning analytics. Dragan still vividly remembers the days when he joined the LORNET research network funded by Canada’s NSERC. Jim co-led Theme 3 (Active and Adaptive Learning Objects) together with Gord McCalla, while Dragan was a postdoctoral fellow in Theme 1 (Interoperability). LORNET was conceived at the time when the Semantic Web was gaining much research attention (Winter et al. 2005; Dicheva et al. 2009) and educational metadata (Brooks et al. 2006). However, after initial interactions with the talented team Jim and Gord led in the Advanced Research in Intelligent Systems (ARIES) at the University of Saskatchewan, they decided to collaborate on the provision of teacher-focused feedback based on the analysis of data about student interactions. This collaboration resulted in the tool called LOCO-Analyst, a tool that has been recognised as a pioneering contribution to the field of learning analytics (Jovanovic et al. 2008). The key idea behind LOCO-Analyst was to provide teachers and course designers with insights into the efficacy of their learning designs and courses. The study presented in this paper is precisely connected to the ideas behind LOCO-Analyst to provide insights into the ways how learning analytics promote the use of learning tactics. This is done by proposing a new learning analytics approach that combines three data analytic techniques — an unsupervised machine learning technique, a process mining technique, and an epistemic network analysis.
Background
Links Between Learning Design and Learning Tactics
Research has emphasised the influence of learning design on the choices of learning tactics applied by learners (Winne and Hadwin 1998; Winne 2006; Diseth 2007; Biggs 1993). Learners’ success, to a large extent, is dependent on how well the adopted learning tactics matched the pedagogical intents behind tasks in a learning design (Pask and Scott 1972; Lust et al. 2013). The capacity of learners to choose and adapt their learning tactics according to the requirements of a learning setting is deemed to be a key self-regulatory skill (Winne 2006).
The links between learning design and learning tactics are influenced by several factors. One of the factors is learners’ perception of the learning design. For example, learners’ perception of the workload affects the choice of learning tactics. Learners show a tendency to demonstrate the use of surface learning strategies if they perceive the workload as heavy or proximity deadline (Diseth 2007). However, if learners perceive the workload and the challenge level of a learning task to be moderate, they tend to demonstrate the use of deep learning strategies (Postareff et al. 2015). Clear teaching objectives, clear evaluation criteria, and sufficient supporting resources in the course design have been found to be positively related to the use of deep learning strategies (Entwistle and Ramsden 2015).
Assessment tasks play an important role in the selection and application of learning strategies (Scouller 1998; Struyven et al. 2006). Existing research has found that assessment approaches such as homework or essays might promote learners’ deep learning strategies and are more effective than exams for students in higher education (e.g., multiple-choice quizzes) (Scouller 1998). Other approaches such as formative (Gijbels and Dochy 2006) and portfolio-based assessment in a core degree course (Baeten et al. 2008), or peer assessment in a teacher training context (Struyven et al. 2006) have been studied in terms of their associations with students’ learning strategies. However, existing research has revealed that formative, portfolio-based, and peer assessments may also lead to more surface learning because of increased workload (Struyven et al. 2006).
Feedback and scaffolding embedded in learning design are also shown to have strong links with learning strategies. For example, students’ tool use is not solely driven by motivation; rather, it is shaped by instructional conditions and experience with the tool use (Gašević et al. 2017). However, not every feedback influences the application of strategies. There are also empirical studies showing that providing only suggestions or feedback on learning strategies is not sufficient to significantly improve the use of learners’ learning strategies (Kizilcec et al. 2016; Davis et al. 2016).
The links between learning design and learning strategies are also strongly influenced by differences in cognition, meta-cognition, and motivation (Winne and Hadwin 1998; Greene and Azevedo 2007; Chen et al. 2017). In the MOOC context, learners join a course with very diverse learning objectives and motivations. These various motivations may lead to the use of different learning tactics and learning strategies, even within the same learning design (Chen et al. 2017). Therefore, we defined our first research question (RQ1) as shown at the end of the Introduction section.
Different performance groups may also show different adherence to predefined learning paths. For example, learners who passed a MOOC showed more “on track” learning behaviour and less “binge-watching” strategies compared to those who did not complete/pass the MOOC in the study reported (Davis et al. 2016). However, the difference in learning tactics across performance groups are under-investigated, especially lacks the perspective of learning design (instead of the demographic characteristics) in the analysis. Thus, this led to our second research question (RQ2) as shown at the end of Introduction section.
Detecting Learning Tactics from Trace Data
Many methods have been developed and utilised to detect learning tactics and strategies, including observations, think-aloud, self-report questionnaires and more recently digital trace data.
Initially, self-reports such as questionnaires and think-aloud protocols have been used to capture learning tactics in experimental and quasi-experimental studies (Pintrich and Maehr 2002; Meyers et al. 1990; Garcia and Pintrich 1996; Winne et al. 2002). A series of widely cited scales emerged in the last century, such as the Strategy Inventory for Language Learning (SILL) (Oxford and Burry-Stock 1995) and the Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich et al. 1991). These scales have been migrated to online environments and continuously being optimised (Jansen et al. 2017). However, scholars in learning analytics have pointed out the limitations of such traditional methods as questionnaires. For example, the reliability of self-reported data is frequently questioned in research on self-regulated learning (Winne and Perry 2000). More importantly, the concept of learning tactics and strategies emphasises that researchers should analyse learning in a timely manner (Schunk 1989), and the temporal nature of learning should not be ignored (Knight et al. 2017). However, the traditional self-report methods such as questionnaires or interviews can not often capture the temporality of learning processes.
Two common kinds of temporal features are often considered in learning analytics: the passage of time (e.g., how much time learners spend on a learning task, or how they allocate time during a learning session) and the temporal order (e.g., how events or states are sequentially organised during or across learning sessions) (Chen et al. 2018). To address the temporality of learning, the use of data analytic methods for extracting learning tactics from trace data has attracted considerable attention in the literature (Gasevic et al. 2017). Trace data captured by an online learning environment can unobtrusively measure cognition and metacognition of learners (Winne 2011) more accurately than an incomplete or biased self-report about learning events in general (Gasevic et al. 2017; Zhou and Winne 2012). Following the temporal analysis approach of trace data, many learning tactics (e.g., assessment-oriented ) (Ahmad Uzir et al. 2019; Matcha et al. 2019) and strategies (e.g., active agile strategy or summative gamblers strategy) (Saint et al. 2018) have been detected and discussed in the literature. Combinations of data analytic techniques have also been proposed to detect learning strategies and related factors using process, sequence and network analytic techniques (Matcha et al. 2019), or using timeline-based visualisation to detect inconsistencies between learning designs and student behaviours (Nguyen et al. 2018). The literature presented thus far has demonstrated that theoretically meaningful learning tactics can be observed from trace data.
The following subsection reviews existing learning analytic approaches that are commonly used for detection of learning tactics and strategies from trace data.
Related Methods and Findings
Cluster Analysis
As a family of data mining techniques, cluster analysis aims at identifying and characterising the structure or features in data. Cluster analysis techniques are unsupervised machine learning techniques and they aim to identify clusters and the degree of similarity between them. Hence, data with similar features/patterns are grouped in the same cluster. In the MOOC context, researchers used clustering techniques to segment learners into groups according to different study patterns and learning tactics. For instance, Ahmad Uzir et al. identified clusters, representing learning tactics, based on time management modes used by learners in online activities of a flipped classroom, and described four time-management tactics as “Mixed”, “Catching up”, “Preparing”, and “Ahead” (Ahmad Uzir et al. 2019). Poquet and Dawson identified four types of forum contributors based on learners’ social network features and revealed the potential for unfolding social processes among a persistent cluster of learners in a MOOC setting (Oleksandra and Shane 2016). However, we noticed that although several features (e.g., number of resources viewed, the length of continuous study, and the frequency of participation in the test) had been tested, a full range of features not fully well been investigated. “Time spent” on different learning actions during a learning process is one of these key features. However, only a few empirical studies have examined the link between learning tactics with time spent online (Ahmad Uzir et al. 2019). To fill this gap, this paper reported the findings of a study that performed a cluster analysis to detect learning tactics based on time spent on each learning action in order to unveil theoretically meaningful tactics used by MOOC learners, as part of the attempt to address research question one.
Sequence and Process Mining
Learning as a temporal process has sequential properties, and therefore, much work published in the literature reported the use of sequence mining techniques for detection of learning patterns from trace data. For example, Jovanović and her colleagues (Jovanović et al. 2017) used the optimal matching of state sequences to detect learning strategies and found a significant association between the detected learning strategies and academic performance in a flipped classroom context. The idea of comparing the sequences of learning and teaching activities has also been recognised in many studies. Davis and his colleagues (Davis et al. 2016) investigated into learners’ sequential movement over time through the activities offered in a MOOC, which enabled the MOOC teachers to evaluate and adapt the learning designs. Process mining techniques have also been used by several scholars to investigate the temporality of learning. For instance, Saint et al. used a process mining technique to analyse the “eventised log data” and reveal the “micro-level SRL processes” of students in online activities of a flipped classroom (Saint et al. 2018). A comparison of process, sequence and network analytic approaches has been performed recently by Matcha et al. (2019), indicating the tactics detected by a sequence analytic approach were differed from those identified by the other two methods. The process and network analytic approaches had more than 66% of similarity in the detected tactics. However, learning strategies detected by the three approaches were proved to be highly similar.
Epistemic Network Analytics
Epistemic network analysis (ENA) is a network analytical technique which bases on the theory of epistemic frames and is used to analyse trace data in individual and collaborative settings (Shaffer 2004). The theory of epistemic frames views expertise in complex domains as a network of connections among knowledge, skills, values, and decision-making processes (Shaffer et al. 2016). Associative connections are established through relative weighting. Statistical techniques are employed to compare the salient properties of networks generated in the context of the content of the network and traces of learning processes (Shaffer et al. 2016). Ahmad Uzir and colleagues applied ENA to study the connections between time management tactics and learning tactics and compare the connections across student groups who followed different learning strategies (Ahmad Uzir et al. 2019).
ENA can be combined with other methods like process mining, to analyse sequential and temporal patterns. For instance, Saint et al. found that a combination of ENA with a process mining technique to study self-regulated learning enabled to provide much richer insights into the process differences between learners on different performance levels than only one of these individual analytic techniques was applied (Saint et al. 2020). More importantly, ENA methods analysis can also visualise SRL pattern changes by the relative position of the weekly based network, and which showed the movement of learning strategies as the course progresses (Saint et al. 2020). These findings were potentially indicating the formation of learning strategies and the influence of learning design on the formation.
Few studies have used ENA to analyse the connection between learning design and learning strategies. To our knowledge, the only study on a related topic that used ENA to analyse learning design is done by Whitelock-Wainwright and his colleagues who have examined the associations between learning activities across digital and physical spaces and compared the associations across different subject areas. The comparison between different subject areas (e.g. Arts and Pharmacy) has shown significantly different connections in the learning designs (Whitelock-Wainwright et al. 2020). The current study investigated the links between learning tactics and learning design issues, with the combination of clustering, process mining and epistemic network analysis.
Method
Context
Introduction about the MOOC and Learning Design
The study was conducted in the context of a massive open online course (MOOC) “Flipped Classroom” because the MOOC meet four criterion. First, the MOOC had a focused subject (flipped classroom) and targeted adult learners who normally have relatively stable learning strategy status rather than children or young learners (Malmberg et al. 2014). Second, the MOOC attracted a large number of learners, which is ideal for detecting learning patterns from rich trace data. Third, the MOOC had been updated after several rounds of offerings, and the learning design was stable for the analysis and comparison. Lastly, the development team of the MOOC paid significant attention to the learning design; to cater to the different pedagogical intentions, each unit of the MOOC use different learning resources and offered different learning pathways. It is very important for this MOOC to meet the last criteria, because in order to answer our research questions, the MOOC should be well designed and use suitable and diversified designs for the different topics. For example, if the MOOC used “lecture only” design (without relevant practical tasks and projects) throughout the whole course, learners would only watch videos and adopt no other tactics, not to mention, using tactics flexibly. The detailed introductions of this MOOC and its learning design are explained in the following text.
The “Flipped classroom” MOOC targeted in-service and pre-service teachers in China. The course aimed to help the teachers understand the concept and approach of the flipped classroom and help the teachers apply this pedagogy model into their daily teaching practice. This MOOC was developed by a Chinese university in 2014 and offered to public through a Chinese MOOC platform called iCourse 163. It was opened for four consecutive years and had a total of 16 rounds, attracting more than 100,000 learners to register and participate. In the first year of the course (first three rounds), the course organisers supplemented some learning resources, adjusted the learning pathways, and revised the assessment criteria. Starting from the fourth offering of the MOOC in 2015, the MOOC followed a relatively mature and stable curriculum structure. Therefore, this study focused on the learners and their learning trace data from the 4th to the 16th offerings. The total number of registration was 97,475, and the overall pass rate of the course was 6.48%.
Each offering of the MOOC was seven weeks long and each covered one unit (i.e., seven units in total). In each week, learners were expected to spend around 4 hours to watch videos, participate in the discussion forum, read the materials, complete quizzes, and perform peer reviews (no unit quiz and peer review were offered for unit 0 and unit 6). In terms of the assessment, the unit quizzes were accumulated and accounted for a total of 25% of the final grade, peer review accounted for a total of 35%, forum discussion accounted for 20%, and the last 20% was the final exam (schedule in week 7). The MOOC was not self-paced. In each week of the course offering, the teaching team would release the learning resources related to the unit scheduled in that week.
The learning design of this MOOC is summarised in Table 1. Unit 0 described the learning objectives of the MOOC and included the preparatory learning activities (e.g., get familiar with the platform) and tips on how to study effectively in the MOOC. Unit 1 aimed to introduce the learners to the concept of flipped classroom and related educational theories. Unit 1 followed a traditional lecture-based design and used lecture videos to help learners build their understanding of what flipped classroom is. Unit 2 aimed to introduce the self-directed “pre-class” part of flipped classroom, and the theory behind flipped classroom (e.g., Bloom’s taxonomy or ARCS motivation model). Unit 2 followed a design grounded in case-based learning and aimed at reviewing practical cases of the flipped classroom by encouraging the learners to connect relevant learning theory with their own teaching practice. At the end of the week, learners were asked to write their own teaching plan; the grades of the teaching plans (marked through peer review) were accumulated to the final score. Unit 3 aimed to guide the learners to produce teaching videos for the flipped classroom; this unit followed a “learning by doing” or project-based design. This unit offered the learners with a step-by-step process which guided them to produce a complete video and master the video production process. At the end of this unit, learners were required to submit a video and complete reviews of their peers’ videos. Unit 4 focused on how to organise “in-class activities” for a flipped classroom. Unit 4 followed a similar case-based design to that of unit 2 but also included several reading activities instead of watching the videos. The theories were also implanted into the cases which were used to help learners understand the instructional principles and activities. Unit 5 aimed to cover practical issues of flipped classroom. This unit also followed a problem-based design. Hence, learners were required to reflect on their understanding in the discussion forum and share their teaching experience. Unit 6 was an extra and voluntary module focused on sharing teaching experience and cases by the MOOC learners themselves. The unit was mostly a continuous accumulation of lecture videos and followed no on-propose design from the teaching team.
The reason for the diversity in the learning design across the units was due to the unique course development approach. A common development approach for most MOOCs is usually led by principal instructors and all materials in this MOOC followed the consistent teaching strategies. However, this MOOC was developed by a large teaching team which was divided into sub-groups, and each sub-group led the design and development of one unit following the most suitable approach they found for the learning outcomes. The principal instructor of the MOOC reviewed and ensured that the main idea of design is sound in each unit; the principal instructor also evaluated each unit’s quality. This is why the MOOC remained coherent and coordinated as a whole, but each unit maintained its own characteristics. It is precisely this variety and diversity of the learning design that made this MOOC unique and provided a suitable research context to address the research questions.
MOOC Learners and Performance Groups
The learners participated in this MOOC were mostly in-service and pre-service teachers (around 80% and 8%, respectively), and 61.54% of them were female. The average age was 36 years old (all based on the self-report survey). For these teacher-learners who were very busy in their daily teaching and family lives, it was challenging to persist in a flexible learning context like this MOOC. Although there were nearly 100,000 registrants for the MOOC, only about half of them entered the course and generated trace data about their learning activities. Similar to any MOOCs, there were a large number of learners who have left the course only after a short stay.
In this study, we considered only those who sustained their studying for more than three learning sessions and were active in the course for more than three weeks. The reason why we set this threshold is that the majority of MOOC learners only logged in to less than three sessions or only stayed in the course for a week or two. If the analysis included the data of those MOOC learners, the results of the analysis would be mainly showing the learning process in the initial stage of this MOOC and it would be difficult to reveal the links between learning tactics and the learning design of the entire MOOC. As a result, we obtained 8,788 learners (9.02% of the total number of 97,475 registrants) who produced more than four million interaction logs as recorded by the MOOC platform. The learners, included in the study, were divided into three performance groups based on the total score obtained after the course was completed. The three groups included the Low-performance group who failed this MOOC (scores were between 0 and 60 out of 100), the Medium-performance group who completed this MOOC with “pass” grades (scores at least 60 but lower than 80), and the High-performance group were those who obtained “good” grades (scores equal to or greater than 80). These cut-off points were used in this MOOC to grade learners and issue corresponding certificates.
Data Analysis
In order to address the research questions around learning tactics and learning design, we used course design data, student performance data, enrolment data, and trace data. To detect learning tactics in and across learning sessions, we first needed to define learning sessions in this study as they were basis for all our analyses.
Learning Sessions
When segmenting streams of events recorded in trace data into sessions, unreasonable long dwell times between two events are often used as markers for separating learning sessions (Gasevic et al. 2017). The literature suggests that there is a big difference between the dwell time adopted by different studies such as 30 minutes (Gasevic et al. 2017), 45 minutes (Liu et al. 2015), or even 2 hours (Kizilcec et al. 2017). Most of the literature does not discuss how this dwell time is determined, and the basis for determining this time is rarely discussed (Kovanovic et al. 2015). We posited that an ideal cut-off criterion should not be either too short (ensuring the learning session are reasonably continuous and contain rich behaviours) or too long (avoid unreasonably long sessions cross lunch, dinner or even the whole night). With this in mind, we tested all possible dwell times from 5 minutes to 120 minutes and found that the learning sessions could exhibit enough learning activity and relatively few unreasonable sessions with 45 minutes as the dwell time to segment sessions. Here, unreasonable sessions meant the continuous sessions that were too long and should obviously be split into two (e.g., one session event recorded in the morning only and the second one in the afternoon).
Therefore, in this study, we used 45 minutes as the dwell time to terminate one learning session, that any action with duration equals to or was longer than 45 minutes meant the end of a learning session. According to this, all streams of events in log data of learners in one MOOC offering were divided into learning sessions. We started from 4,664,214 unique events recorded in the log data for 8,788 learners. These events were split into 201,038 learning sessions.
Learning Actions
Research on tactic detection relies on the analysis of actions performed by learners as recorded by trace data. The initial step required for the tactic detection process was the labelling of learning actions in trace data. Different studies label raw trace data quite differently (e.g. (Maldonado-Mahauad et al. 2018; Davis et al. 2016; Sinha et al. 2014; Kizilcec et al. 2017)). Nonetheless, the key concept of action labelling is to give meaningful action names that corresponded well to relevant learning tasks (Kizilcec et al. 2017). With the aim to be consistent with the literature and by considering the learning context of this MOOC, we defined nine labels of learning actions, as presented in Table 2.
Data Analysis Methods
To answer the first research question (RQ1) identified in Section 5, we employed a clustering technique to first detect the meaningful learning tactics used by MOOC learners. After that, a process mining technique was used to examine the tactics used across learning sessions (RQ2). To investigate the connection between the learning design and learning tactics, ENA was used. The data analysis process is summarised in Fig. 2. Most terminology used in this paper such as learning sessions and learning tactics have been introduced or defined. Tactics Usage in Fig. 2 means how learners use different learning tactics across several learning sessions(e.g. focus on watching videos in one session, then mainly spent time on assessment in the following session).
The data analysis process used in the study
Clustering
Several clustering techniques have been proven to be effective to detect meaningful patterns of learning actions (Jovanović et al. 2017; Matcha et al. 2019; Maldonado-Mahauad et al. 2018). In this study, the Expectation-Maximization (EM) algorithm was applied to detect the pattern of learning actions indicative of the tactic used by the MOOC learners (i.e., to address to answer research question one). Previous studies had demonstrated that the EM algorithm was useful to detect theoretically-meaningful learning tactics (Matcha et al. 2019; Ahmad Uzir et al. 2019). In this study, the proportion of learning actions were identified in Table 2. The total time spent online and the total number of actions across all learning session were used as inputs for the EM clustering algorithm. The Bayesian Information Criterion (BIC) was computed to identify a suitable number of clusters. The sequence analysis (TraMineR R package) was then used to explore the detected tactics in terms of the distribution of learning actions, the frequency and the ordering of the sequences (Gabadinho et al. 2011). The sequence analysis helped us understand better each cluster and propose more suitable names for each tactic by following (Matcha et al. 2019; Ahmad Uzir et al. 2019).
In order to check the statistical significance between the discovered clusters, a one-way multivariate analysis of variance (MANOVA) was conducted with the learning tactics as the single independent variable and the nine actions as the dependent variables. The assumption of homogeneity of covariances which was tested was not accepted. Thus, Pillai’s trace statistic was used. As it is more robust to the assumption violations (Field 2013) together with the Bonferroni correction method. As a follow-up, a series of one-way ANOVAs with Bonferroni corrections for each of the dependent variables (nine learning actions) were also conducted.
Process Mining
To further explore the differences between performance groups and address research question two, we used the process mining technique implemented in the bupaR R-package. Janssenswillen and colleagues (2019) proposed the process model allowing the mining of patterns focusing on each student’s learning process from the beginning to the end of the course. Thus, in our use of process mining, individual students enrolled in the course were cases. Each case related to a coarse-grained concept of activity. In this case, activities were the tactics adopted by the students while progressing in their studies throughout the course. The process models included all cases and tactics that were involved in learning. In order to check the statistical significance between three performance groups regarding the tactic that was involved in learning, a series of one-way ANOVAs with Bonferroni corrections for the four learning tactics were conducted.
Epistemic Network Analysis
To deepen the discussion of the first research question and visualise the links between learning design and learning tactics, we explored the connection of the detected tactics and the MOOC units for each performance group by using the Epistemic Network Analysis (ENA) (Shaffer et al. 2016). To construct a network model, ENA requires the definition of a unit of analysis, an activity code, and a stanza, and the network associations are defined as the co-occurrence between codes within the bounds of a stanza (Shaffer et al. 2016; Saint et al. 2020). For this study, learning session was used as the unit of analysis; learning tactics and the corresponding MOOC units were used as the codes of analysis; and learners in each MOOC offering were used as the stanza of analysis. Based on these definitions, we created the ENA model by computing the co-occurrences of learning tactics and the corresponding MOOC units in each learning session of individual learners in each performance group. In this study, ENA allowed visualisation and statistical analysis of the networks of each performance group by plotting them in the same ENA space, i.e. the nodes were located at the same positions across the different networks. Therefore, the structure, the strength, and the differences in the network of each performance group could be observed.
The data and analysis scripts are made available here in this link.Footnote 1
Results
Learning Tactics Identified from Clustering
In response to research question one, four clusters were identified with the cluster analysis, based on the proportion of learning actions, the total time spent online and the total number of actions in each learning session. Each cluster represents similar patterns of actions that are indicative of the learning tactic used. Descriptive statistics of these four clusters, including proportion, session duration, and numbers of relevant actions are shown in Table 3. It is worth noting that actions such as “Help-seeking” and “Forum” do exist in very few learner’s learning, but there were still some learners who (less than 25%) exhibited this action. Therefore, actions such as “Help-seeking” still appeared in Figs. 3 and 5, but played a relatively small role.
Median time spent on different course activities and state distribution of tactic 1; Note: the x-axis represents x-th action of the entire learning session from the first action, and the y-axis represents the distribution of different actions on x-th action of the sessions. For example, the first bar shows the distribution of the first actions for all cluster 1 sessions, and the second bar shows the distribution of the second action for all cluster 1 sessions
Tactic 1: Search oriented
As shown in Table 3 and Fig. 3, the first cluster accounted for 19.52% of the total learning sessions. The median duration of learning sessions was 51.30 minutes. This cluster of learning sessions contained a higher median value of learning actions than other clusters, and the biggest percentage was “Search” out of the nine learning actions. Learners, when working on the sessions in this cluster, spent nearly 25% of their time to search for certain information or content. Even though the most prominent actions were “Search”, high frequencies of interaction indicative of “Content_access” (Yellow), “Content_revision” (Deep green) and “Assessment” (Light green) were also observed. The stacked bar chart in Fig. 3 shows that “Search” throughout the entire session which formed sequences with other actions such as “Content_access”, “Content_revision”. This indicates that the MOOC learners employed the Search Oriented tactic to deepen their understanding, answer their questions, or overcome difficulties they may experienced in learning.
Tactic 2: Content and Assessment Oriented
As shown in Table 3 and Fig. 4, the second cluster accounted for 31.86% of the total learning sessions. The median session duration was 57.08 minutes which was the longest time among the four clusters. Learners in this cluster spent 20.00% of their time on “Content_access”, 9.68% of their time on “Assessment”, and 15.38% on “Content_revision”. This learning tactic was used by learners to construct new knowledge or acquire new skills, evaluate their learning through assessment, and deepen their understanding by revision. As the most commonly used tactic (31.86% of all sessions), Content and assessment oriented showed a very integrated and comprehensive pattern of learning. Learners accessed most of the resource types from the course design, including the discussion forum (e.g., the scored discussions) and overview action (e.g., reading the announcements). This learning tactic also indicates learners following the learning design and balancing their learning actions to achieve the learning goals.
Median time spent on different course activities and state distribution of tactic 2
Tactic 3: Content Oriented
Compared to clusters one and two, cluster three was relatively simple and focused (see Fig. 5). This tactic was also commonly used by learners (30.29% of all sessions). The median duration was 31.43 minutes. The most prominent characteristic of this cluster was that learners spent 37.50% of the median time focusing on “Content_access” (Yellow) with another 21.43% of their time on “Content_revision” (Deep green). There was a clear pattern in this cluster that learners in these sessions were highly “Content oriented”, and did not spend much time on the assessment or discussions. In addition, as shown in the stacked bar chart in Fig. 5, learners in these sessions spent more time on “Content_access” and “Overview” at the beginning of study sessions, then moved on to “Content_revision” and “Forum” later in their study sessions. The sequence feature shows that learners used Tactic 3 to explore new content and then revise and discuss previously explored content in the forum.
Median time spent on different course activities and state distribution of tactic 3
Tactic 4: Assessment Oriented
As the cluster that had shortest learning sessions, cluster four had only 14.93 minutes as the median duration and accounted for 18.33% of the total learning sessions. In such a short time period, learners only focused on one main type of learning activity – “Assessment” (66.67%, as shown in Fig. 6). We name this cluster as “Assessment oriented”, as learning sessions in this cluster did not account for any time on learning new content, participating in the discussion or searching any relevant information; the sessions purely focused on assessment activities such as the unit quizzes, homework, and peer reviews.
Median time spent on different course activities and state distribution of tactic 4
A statistically significant MANOVA effect was obtained, Pillai’s Trace = 1.65, F(27, 603081) = 27326.18, p = 0.0000. The multivariate effect size was estimated at multivariate η2 = .55. The series of one-way ANOVAs with Bonferroni corrections showed that all nine ANOVA models (one for each of the nine learning actions) were statistically significant.
It is worth noting that Content and assessment oriented tactic is not a literal splicing and a simple combination of Content oriented tactic and Assessment oriented tactic, but a more complex learning tactic. As shown in the stacked bar chart in Fig. 4, several learning actions (i.e., “Content_access”, “Assessment” and “Content_revision”) are relatively and evenly distributed on the x-axis (representing action locations from the first action to the last action of a session) of all Tactic 2 sessions. This distribution indicates that learners who used Tactic 2 were navigating back and forth between different materials in this MOOC such as content and assessment. For example, learners could attempt the unit test for the first time (assessment action), then they could revisit some specific videos or re-read materials (content_revision action), and finally attempt the same unit test again (assessment action) to improve the score. This tactic is therefore differentiated from focusing on content (tactic 3) or on assessment (tactic 4) only.
All these four learning tactics detected above by the cluster analysis were conceptually important for MOOC learners to complete this course. However, different learners used these tactics differently. In the following subsection, we will show the process mining results across sessions and discuss the differences between performance groups.
Tactics Usage Between Different Performance Groups
The differences among the learners of each performance groups were explored in terms of the process of tactics applications, the idle time, and the adoption of the tactics. Out of all the 8,788 learners, 65.13% (5724) of them belonged to the High-performance group, 14.76%(1297) of them belonged to the Medium-performance group, and the remaining 13.03%(1767) belonged to the Low-performance group.
Process Difference
As shown in Fig. 7, three process models were created (using the Heuristic Miner as our process discovery algorithm (van der Aalst et al. 2004)) to analyse the process of tactic usage across learning sessions for the entire duration of the MOOC by learners from different performance groups. The majority of the learners in all three performance groups began their learning by using the Content oriented tactic (around 50%), then shifted to use the Content and assessment oriented tactic (around 30%) to evaluate their learning, with other tactics following. Another common process they shared was the continued use of one same tactic. For example, after using the assessment oriented tactic, there was an over 30% probability (see the transition with the arrow pointing back to the Assessment oriented tactic in Fig. 7) that the learners would continue to use this tactic in the next session. However, besides these commonalities, the process differences between groups were observed.
Comparison of the low-, medium-, and high-performance groups with process models extracted with process mining representing how learner groups changed their learning tactics across study sessions
Low-Performance Group
(Fig. 7a) most frequently used the Content oriented tactic. As the most dominant tactic for low-performance group learners, Content oriented tactic accounted for 40.25% of all tactic usage with a high probability of continuous use (42.9% Content oriented to Content oriented). However, the Assessment oriented and Search oriented tactics were much less used in the whole learning process. More importantly, the probability of moving from other tactics to this tactic was significantly higher than those of the other two performance groups. For instance, the process probability from the Content and assessment oriented tactic to the Content oriented tactic for the low-performance group was 35.74%, but the same process probabilities of the other two groups were 27.35% for the medium-performance group and 26.27% for the high-performance group. This indicates that for the low-performance group learners, the choices of tactics focused on how to efficiently study through the content, but by ignoring a need to search for further information and deep understanding. The assessment was less than the focus for the low-performing group which was associated with relatively low scores.
Medium-Performance Group
Compared to the low-performance group, the medium-performance group of learners (Fig. 7b) showed a balanced usage of all tactics. They frequently used the Content and assessment oriented tactic (29.71%) and the Assessment oriented tactic (22.02%). The percentages of transitions from Content and assessment oriented to Assessment oriented (18.9%) or from Search oriented to Assessment oriented (20.37%) were much higher than those of the other two performance groups. The patterns of tactic usage of the medium-performance group showed that the students in this group were considerably focused on assessment. This is an interesting finding that the high-performance group was not as prominent as the medium-performance group in the usage of the Assessment oriented tactic.
High-Performance Group
The most frequently used tactic by the high-performance group was the Content and assessment oriented tactic (33.12%), which was characterised by a relatively long, integrated and comprehensive pattern of learning. What is more remarkable is that this group of learners continuously used this tactic, which means the learners in this group mixed content access, revision and evaluation with assessments to achieve their learning goals. This pattern of tactic usage was closely related to the intent behind the learning design of the course. That is, not only were learners expected to access the content or participate in the assessment, but learners were expected to take assessments (quiz questions, homework, and peer reviews) as a way to improve or deepen their understanding. Another difference, in contrast to the other two performance groups, was the use of the Search oriented tactic, as shown in Fig. 7c, indicating that nearly 30% of the learners in the high-performance group started their learning with the Search oriented tactic. This shows learners start to search for certain learning content in the early stage of the course, which potentially reflects a clearer goal or more specific needs related to flipped classroom.
All four ANOVA models (one for each of the four learning tactics) were statistically significant, and a follow-up analysis for the pairwise comparison between performance groups were conducted (results in Table 4). As shown in Table 4, most pairwise comparisons showed statistically significant differences except two pairs.
Comparison of Idle Times
The learners from the low-performance group had longer idle times between learning sessions than the medium-performance and high-performance groups. This shows that the high-performance learners engaged more frequently in studying with the MOOC. What is even more interesting is how high-performance learners connected different tactics over time. As shown in Fig. 8, the low-performance group had a considerably longer gap between an assessment oriented session and another assessment oriented session (2.96 days on average) than the other two groups (1.95 days for medium-performers and 1.29 days for high-performers). The high-performance group had on average less than 1 day idle time. The learners in high-performance group continuously and comprehensively used the Content oriented tactic, the Content and assessment oriented tactic, the Search oriented tactic or alternated between these tactics. The shorter idle time, the more frequent usage of diverse tactics and flexible flow or interaction across multiple tactics were the key features of the learners in the high-performance group.
Process models indicating differences in idle time between the low-, medium, and high-performance groups of learners
Comparison of Tactic Adoption Patterns
As this MOOC was seven weeks long, we examined the tactic adoption for each week, and analysed the differences between performance groups. As shown in Fig. 9, the mean number of four tactics (with four different colours) was plotted as a weekly sequence from week 1 to week 7, indicating tactic adoption difference across the three groups. For the low-performance group, the learners used the Content oriented and Content and assessment oriented tactics most frequently. The learners did not clearly adjust their use of tactics as the course was progressing to accommodate to the changing study needs from week to week as reflected in the learning design of the course. For the medium-performance group, the learners changed the use of tactics by initially focusing on the Content and assessment oriented tactic between week 2 and week 5 and then on the Assessment oriented tactic in the last two weeks of the MOOC. This shows that the medium-performance group of learners was able to adopt different learning tactics based on the requirements of the different stages of courses, and thus, demonstrated the ability to regulate their learning.
Comparison in the frequency of the tactic usage between the low-,medium, and high-performers for each week of the course
The difference in the tactic adoption in each week was more pronounced in the high-performance group. The learners in this group used more the Content oriented and Search oriented tactics at the beginning of the course; then, they showed a wide usage of the Content and assessment oriented tactic between week 2 and week 5; and finally a relatively frequent usage of the Assessment oriented tactic towards the end of the course. This shows that the high-performance group of learners did regulate their learning tactics with more flexible adjustments based on the requirements of different course stages.
Links Between Learning Design and Learning Tactics
Dominant Tactics for Each Unit
In order to better investigate into the links between learning design and learning tactics, it was necessary to locate each learning session to the specific unit position and discuss whether different learning tactics will be applied in different MOOC units (corresponding to different learning designs). The links between learning designs and learning tactics were explored in terms of the dominant tactic used in each unit of the course and the network of the tactics’ application.
As shown in Fig. 10, the learners used the Content oriented tactic as the dominant tactic for unit 0, unit 1, unit 3 and unit 6. Rarely did the learners use the Assessment oriented tactic while working on units 0 (preparation unit) and 6 (teaching experience sharing unit) because the unit quizzes and homework were not featured in these two units (refer to Table 1). The Content and assessment oriented tactic, as an integrated and comprehensive tactic, was most frequently used in unit 2, unit 4 and unit 5. This combination of content access and using assessment to evaluate learning was also expected in these units according to the course learning design (Table 1). For example, unit 2 and unit 4 followed a case-based learning design to introduce teaching theories and principles, which required not only watching videos or reading articles but also participation in unit quizzes and homework (write a flipped teaching plan) to promote deep learning. The learners demonstrated a relatively higher percentage usage of the Assessment oriented tactic while working on unit 3 (20.90%) and unit 5 (27.30%). This is because in unit 3, the corresponding learning activities followed a project-based learning design. That is, after using the Content oriented tactic to go through a step-by-step tutorial for making teaching videos, the learners needed to focus on the assessment task in which they were required to produce a video. In unit 5, as the learners were approaching the end of the course, they began to work more in the assessment tasks. As such, the dominant tactics for each unit reflect the requirements of the course learning design, and thus, reveal the links between learning design and learning tactics.
Distribution of the tactic usage across the seven course units
Network Difference
The epistemic networks for each performance group are shown in Fig. 11, in which the nodes represent the seven course units and the four detected tactics. The thickness of the links represents the relationships between the tactics and the units. The thicker lines indicate the stronger connection which represents the higher co-occurrence of the application of tactics with course units. The ENA space was produced by using singular value decomposition (SVD) to reduce the dimensionality of the graph. The mean rotation (MR) in Fig. 11 was used to show the maximum difference between two groups of networks in order to better observe differences in tactics used for each associated topic. In particular, the x-axis explained 6.3% of the variability in the network, whereas the y-axis explained 8.0% of the variability in the network. The course units were located close on the x-axis, while the y-axis explained the differences between the course units. For instance, the units that students were required to complete at the beginning of the MOOC such as the preparation and concept topics were located on the left hand side of the y-axis. As the course progressed to more practice-related topics, the location of the corresponding course units were moved to the far edge of the y-axis. The x-axis differentiates between the tactics. Particularly, the Content oriented, Assessment oriented, and Search oriented tactics were located in the upper part of the graph while Content and assessment oriented tactic was located in the lower part of the graph.
The epistemic networks of the low-, medium, and high-performance groups. The networks are based on the co-occurrence links between the seven course units and four automatically detected tactics
The three performance groups showed some similarity in the links between different tactics and units with some notable differences. The low-performance group (Fig. 11a Network model for the low-performing group) showed relatively strong links in the left part of the network, which was illustrated by the connections of the Content oriented tactic with unit 0, unit 1 and unit 2. The medium-performance group (Fig. 11b Network model for the medium-performing group) showed relatively strong links in the right part of the network, which was demonstrated by the connections of the Assessment oriented tactic with unit 3, unit 4 and unit 5. The high-performance group (Fig. 11c Network model for the high-performing group) showed relatively balanced links across the entire network and relatively strong links in the lower left part of the network, which represented the connections of the Content and assessment oriented tactic with all units in the MOOC.
As shown in Fig. 12, when subtracting the networks of two performance groups, the colours of the lines represented the groups which had relatively stronger connections between each pair of the nodes. For instance, the link between the Content oriented tactic and unit 1 is red (low-performance group) and the link between the Search oriented tactic and unit 5 is blue (high-performance group). Compared to the low-performance group which focused on using one tactic only, the learners in the high-performance group showed stronger connections of the remaining three tactics with the major part of learning materials (unit 2-5). This indicates that the learners with high performance did adopt different and diverse tactics to accommodate the needs of units, which were based on different learning designs.
Pairwise comparisons of the performance groups on their epistemic networks, which represent co-occurrence connections between the course units and automatically extracted tactics
Differences between the medium- and high-performance groups corroborated the findings from the conducted with process mining. That is, the medium-performance group used the Assessment oriented tactics more frequently than the high-performance group. The high-performance group used more balanced and integrated tactics including the Search oriented and Content and assessment oriented tactics than the medium-performance group. We found differences between the medium- and high-performance groups in the use of learning tactics in units that were based on the case-based design (i.e., unit 2 and unit 4). To study in these units, the learners from the high-performance group used the Content and assessment oriented and Search oriented tactics more frequently (blue lines) than the medium-performers. Conversely, the learners from the medium-performance group used the Content oriented and Assessment oriented tactics more frequently (green lines) than the high-performers when working on the units that followed the case-based design (units 2 and 4). This difference shows that the higher performance learners were more likely to connect the exemplary cases and theory the cases were based on the use of relevant tactics for learning from the cases (e.g., drafting their own flipped classroom designs) and evaluation (e.g., peer reviewing designs of their peers based on a rubric). After reflecting on and evaluating their own learning through assessments, the high-performance learners would go back to search for relevant information to deepen their understanding of the case studies or work on additional assignments. This is also corroborated in Fig. 7c where a high probability of transition between the Content and assessment oriented tactic and the Search oriented tactic) was observed. However, the medium-performers were more inclined to separate case-based learning (using the Content oriented tactic for content access and revision) and the tactic used for learning from the cases (using the Assessment oriented tactic focusing on the unit quizzes, unit homework and peer reviews) across different learning sessions.
Discussion
As indicated in Section 5, the choices learners make about the use of learning tactics and strategies are critical for success in online learning, especially in MOOCs (Milligan et al. 2013). Several authors, e.g. (Winne and Hadwin 1998; Rigney 1978), have already highlighted the importance of instructional cues, and how learning design features influence or shape the use of learning tactics and strategies. This study was set out to investigate the links between learning design and learning tactics in a Chinese MOOC context. Therefore, our goal was to answer two major research questions aiming to understand if we can detect meaningful learning tactics from trace data, how the usage of the detected tactics differ between performance groups, and if there are meaningful links between learning designs and the use of learning tactics.
Research Question 1: Links Between Learning Tactics and Learning Design
The analysis of learning actions and time spent online as recorded in trace data, we detected four meaningful learning tactics by using the EM cluster algorithm. The tactics detected were defined as Search oriented, Content and assessment oriented, Content oriented, and Assessment oriented. As consistent with the previous research in the area (Matcha et al. 2019; Ahmad Uzir et al. 2019), the four study tactics were reflective of study opportunities provided in the learning context. For instance, the MOOC used in the study was designed by combining several learning designs, i.e. case-based, project-based and lecture-based learning designs. The use of learning tactics detected in the study was closely related to the pedagogical intents behind the learning designs used in the course (Lockyer et al. 2013). That is, the use of the Content oriented tactic was reflective of accessing the lecture videos according to the lecture-based design. The Content and assessment oriented and Assessment oriented tactics were used when interacting with the units, which were based on the case-based learning design.
These findings are well aligned with previous studies. For instance, in a problem-based learning context, Matcha and her colleagues (Matcha et al. 2019) detected four patterns of tactics reflective of the problem-solving activities such as Diverse Assessment, Short-Practice oriented, and Long-Practice oriented and lecture-based activities, i.e. Lecture oriented. Similarly, Fincham et al. (2019) detected learning tactics that are reflective of the use of a lecture-based design such as Diverse and Highly Content-oriented and a problem-based design such as Assessment-oriented. Previous studies also discussed search tactics but focused more on individual actions, e.g., search forward or backward while watching videos (Sinha et al. 2014). The present study considered search as a cluster of sessions, which indicated learners searching for specific content to deepen their understanding or solve problems (Wang et al. 2018; Weinstein et al. 2000).
As shown in the results of epistemic network analysis, the use of different learning tactics was linked with different units of course, whereby the units were based on different learning designs. The learners employed the Content oriented tactic mainly in lecture-based units (Units 0, 1, and 6), and mainly used the Content and assessment oriented tactic in case-based or problem-based units (Units 2, 4, and 5), and combined the usage of the Content oriented and Assessment oriented tactics in the project-based unit (Unit 3). These findings were corresponded to the intentions of the MOOC teaching team, which embedded instructional cues into the design of learning tasks, assessment tasks, and the overall course design. These cues as task conditions were created to influence how learners adjust their choices of learning tactics (Winne and Hadwin 1998), which is similarly defined by Rigney (1978) using the term “content bridge”. A content bridge is used to signify a unit of subject matter incorporated into learning design, which links and bridges the external world (perception of learning stimuli, materials, and design) and the internal world (excites process in long-term memory). In this study, such content bridge theoretically exists in this MOOC, even the perception of the learning design of MOOC differs among the performance groups as demonstrated by the results show different patterns of the associations between the usage of learning tactics and course units.
As discussed in the literature (Diseth 2007; Postareff et al. 2015), workload might be an important feature in learning design that can potentially shape how learners use learning tactics. Workload in this MOOC had an increasing trend from unit 0 to unit 4, by increasing the need to use content, participate in forum discussions, and engage in practical tasks. This increasing workload was intentionally designed to help learners gradually understand the requirements of the course and calibrate their choices of suitable tactics. The learners, especially those from the two higher performance groups, might be shaped during this process into using more comprehensive and balanced learning tactics. The assessment tasks also influenced learners heavily; for instance, the tasks in unit 2 and 4 emphasised on learning from the provided cases which led to the increase in the Content and assessment oriented tactic through which learners combined their work on assessments and course content in each learning session. Although the links shown in epistemic network analysis may not be fully sufficient to explain the specifics of the formation process in the use of learning tactics, the findings of this study revealed the links between learning design and learning tactics, which were consistent with the pedagogical intentions of the MOOC design and teaching team.
Research Question 2: Differences Across Performance Groups
Process mining and epistemic network analysis unveiled differences in tactic usage across three performance groups. The low-performance group used the Content oriented tactic as their dominant tactic, especially in the first half of the course. This feature of the low-performance learners matches the description of Viewers who primarily watch lectures and hand in few if any assignments (Anderson et al. 2014), or the description of the Auditing and Sampling learners who engage less or even disengage in assessment or search (Ferguson et al. 2015). Compared to the low-performance group, the medium- and high-performance groups used more the Content and assessment oriented tactic in combination or sequence with the balanced usage of other three tactics. This balance was also noted before the literature, such as All-rounders in (Anderson et al. 2014), Keen Completer (Ferguson et al. 2015) or Improvers (Ferguson et al. 2015). Learners from the low-performance group also showed no significant adjustment of their tactic usage over time in the course or across different units; in contrast, the high-performers showed a certain level of self-regulation of the tactic usage by adjusting their choices of tactics to the needs of different course units.
These difference in the usage of tactics can be interpreted by using the well-known theoretical notion of approaches to learn (Biggs 1987). In that sense, the low-performance group use Content oriented tactic as the most dominant tactic for many MOOC units, and learners in this group want to finish the task by only focusing on the learning content, which showed a match of the Surface Strategy. The medium-performance group highly valued the Assessment oriented tactic, which is a characteristic of the Achieving Strategy according to the theory of and empirical research on approaches to learning. The high-performance group learners showed high value for the Search oriented tactic, which is closely linked to the notion of Deep Strategy as theorised in the work on approach to learning.
The level of self-regulation of the tactic usage can be influenced or promoted by feedback and scaffolding from the system or course, such as the concept “inspectable student models” proposed in Zapata-Rivera and Greer (2004) and which are closely related to open learner models(Bull and Kay 2010). Learners interacting with the model may become aware what they really know or how they have behaved, and use this information to regulate their learning process (Zapata-Rivera and Greer 2004), or in the context of our study, to regulate the usage of tactics. Learning analytics dashboard can make use of the properties of inspectable student or open learner models (Bodily et al. 2018). Such dashboards can reveal different usage of tactics from different performance groups, which may help the learners to adjust their choices of tactics to the needs of different course units. The course teachers or the platform designer can also provide feedback to learners. For example, teachers can suggest suitable learning tactics for specific course units before or during learners’ learning, or the platform designer can design personalised and real-time scaffolding to prompt certain learning tactic based on learners’ interaction.
Implications for Research and Practice
Three data analytic methods (cluster analysis, process mining and epistemic network analysis) were applied and combined to detect learning tactics and study the links with learning design. The combined use of data analytic methods increased the comprehensiveness of the temporal analysis of learning tactics in this study. Process mining based on clustering results provides a new approach to exploring a learning session level patterns across learning sessions, instead of focusing only on micro-level events within one session. The frequency analysis, process visualisation and idle time analysis, together, provide a much more detailed and vivid picture of MOOC learners’ learning strategies. For example, differences in tactic usage detected with process mining and epistemic network analysis complemented each other and corroborated the findings by revealing similar and/or additional patterns across the three performance groups.
Combination and consolidation of learning analytic methods are valuable in temporal analysis of learning, not only because they provide comparable findings (Matcha et al. 2019), but they can also potentially address concerns related to validity as posited by Reimann and his colleagues (Reimann et al. 2014; Reimann 2009). In analysis of trace data, the definitions of learning sessions and events require a thorough consideration (Reimann 2009), which in this study was discussed in details. The use of trace data and data analytic methods, as an innovative methodological approach proposed in this study, is a useful attempt comparing to using traditional methods that are based on self-report data and conventional statistics. Particularly, the combined use of process mining and ENA adds value to understanding the dynamic nature of SRL, which is not possible with the use of self-reports (Winne 2010). Researchers can benefit from the proposed approach here, to better articulate patterns of SRL strategy movement through learning designs or other factors that may affect SRL. However, more empirical studies should be conducted to validate the learning analytic method and findings reported in this paper.
This study provides practical implication for educational/learning designers or instructors of online courses including MOOCs. Many online learners experience difficulties to effectively self-regulate their learning (Milligan et al. 2013). Designs of many MOOC have unintentionally or intentionally ignored this fact, which is evidenced by large differences in the use of learning strategies among learners (Davis et al. 2016). The study reported in this paper showed how learning design issues potentially shape the usage of learning tactics by learners. The links between learning design and learning tactics (or cognitive process in the border context of student modelling) have been emphasised in previous cognitive theory of instructions (i.e., the CLAI Model (Arruarte et al. 1996)) or learning design frameworks (i.e., the IRIS Shell (Arruarte et al. 1997)). The application of the proposed learning analytic method can offer useful insights for course designers to evaluate the efficacy of their designs and pedagogical decisions as “the data-assisted approach” (Brooks et al. 2014), in order to inform course improvement and designate strategies to cater to the needs of different sub-populations from learners. The findings of this course and the previous literature also suggest that learners need to establish which learning tactics they should use in a new course, and how to adapt tactic use to different parts of the course. A key task for course designers is incorporate relevant instructional cues or teaching tactics into different learning tasks or systems to assist or motivate learners in their choices of learning tactics (du Boulay and Luckin 2016). Applications of the data analytic approach proposed in this paper can assist the course designers to assess to what extent instructional cues integrated in different parts of a course guided learners to make good choices of learning tactics. For example, when re-designing a case-based or problem-based MOOC, instructors could add cues that can be echoed between learning content and assessment tasks. Instruction information should be given to encourage learners to take a test after studying the case to check their understanding of the case, or prompts should be given in unit homework or feedback on questions with incorrect answers to remind or lead learners to review specific content. Such feedback to learners can facilitate the use of specific learning tactics such as Content and assessment oriented which can be most suitable for specific learning designs.
Conclusion and Future Work
In summary, we proposed a novel learning analytic approach for detection of theoretically-meaningful learning tactics and analysis of process differences between groups of learners with different academic performance. The findings provide insights into the links between learning tactics and learning design which were emphasised theoretically through external conditions of the COPES model, and how these links differed across different performance groups.
A limitation of this study is that we used data from only one MOOC. Therefore, future research focused on generalisation of the findings on links between learning design and learning tactics using different datasets in different subject domains, offered on different platforms, and servicing learner populations from regions of the world is warranted. In this study, we compared the tactic usage between three performance groups, but we did not conclude the tactic usage to be a sufficient explanation for learners’ overall performance differences. It would be valuable in the future to conduct an experimental study which based on the tactic usage and other features (e.g., prior knowledge and engagement level) that can systematically explain the differences in MOOC learners’ performance. Future studies should also investigate the formation process of learning tactics in a detailed manner and be with more specific features of learning design, such as workload expectations and assessment design.
References
van der Aalst, W., Weijters, T., Maruster, L. (2004). Workflow mining: discovering process models from event logs. IEEE Transactions on Knowledge and Data Engineering, 16(9), 1128–1142, Conference Name: IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/TKDE.2004.47.
Ahmad Uzir, N., Gašević, D., Matcha, W., Jovanović, J., Pardo, A., Lim, L.A., Gentili, S. (2019). Discovering time management strategies in learning processes using process mining techniques. In Scheffel, M., Broisin, J., Pammer-Schindler, V., Ioannou, A., & Schneider, J. (Eds.) Transforming Learning with Meaningful Technologies, Lecture Notes in Computer Science (pp. 555–569). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-29736-7_41.
Anderson, A., Huttenlocher, D., Kleinberg, J., Leskovec, J. (2014). Engaging with massive online courses. In Proceedings of the 23rd International conference on World Wide Web, WWW ’14, Event-place: Seoul, Korea (pp. 687–698). New York: ACM. https://doi.org/10.1145/2566486.2568042.
Arruarte, A., Fernández-Castro, I., Ferrero, B., Greer, J.E. (1997). The IRIS Shell: “How to Build ITSs from Pedagogical and Design Requisites” International Journal of Artificial Intelligence in Education (IJAIED), 8, 341–381. https://telearn.archives-ouvertes.fr/hal-00197387.
Arruarte, A., Fernández-Castro, I., Greer, J. (1996). The CLAI Model: a cognitive theory of instruction to guide its development. Journal of Artificial Intelligence in Education; Charlottesville, 7(3), 277–313. https://search.proquest.com/docview/1468384818/citation/6CBD6BF2B8354DA4PQ/1. Num Pages: 37 Place: Charlottesville, United States, Charlottesville Publisher: Association for the Advancement of Computing in Education.
Azevedo, R., Moos, D.C., Johnson, A., Chauncey, A. (2010). Measuring cognitive and metacognitive regulatory processes during hypermedia learning: issues and challenges. Educational Psychologist, 45(4), 210–223. https://doi.org/10.1080/00461520.2010.515934 Publisher: Routledge.
Baeten, M., Dochy, F., Struyven, K. (2008). Students’ approaches to learning and assessment preferences in a portfolio-based learning environment. Instructional Science, 36 (5), 359–374. https://doi.org/10.1007/s11251-008-9060-y.
Bakharia, A., Corrin, L., de Barba, P., Kennedy, G., Gašević, D., Mulder, R., Williams, D., Dawson, S., Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, LAK ’16 (pp. 329–338). Edinburgh: Association for Computing Machinery, https://doi.org/10.1145/2883851.2883944.
Bannert, M., & Reimann, P. (2012). Supporting self-regulated hypermedia learning through prompts. Instructional Science, 40(1), 193–211. https://doi.org/10.1007/s11251-011-9167-4.
Barba, R.H. (1993). The effects of embedding an instructional map in hypermedia courseware. Journal of Research on Computing in Education, 25(4), 405–412.
Biggs, J.B. (1987). Student approaches to learning and studying. research monograph. Australian Council for Educational Research Ltd. https://eric.ed.gov/?id=ED308201.
Biggs, J.B. (1993). From theory to practice: a cognitive systems approach. Higher Education Research & Development, 12(1), 73–85. https://doi.org/10.1080/0729436930120107.
Black, P., McCormick, R., James, M., Pedder, D. (2006). Learning how to learn and assessment for learning: a theoretical inquiry. Research Papers in Education, 21(2), 119–132. https://doi.org/10.1080/02671520600615612.
Bodily, R., Kay, J., Aleven, V., Jivet, I., Davis, D., Xhakaj, F., Verbert, K. (2018). Open learner models and learning analytics dashboards: a systematic review. In Association for Computing Machinery (pp. 41–50). New York, https://doi.org/10.1145/3170358.3170409.
du Boulay, B., & Luckin, R. (2016). Modelling human teaching tactics and strategies for tutoring systems: 14 years on. In International Journal of Artificial Intelligence in Education, (Vol. 26 pp. 393–404). https://doi.org/10.1007/s40593-015-0053-0. http://link.springer.com/10.1007/s40593-015-0053-0.
Brooks, C., Amundson, K., Greer, J. (2009). Detecting significant events in lecture video using supervised machine learning, pp 483–490.
Brooks, C., Bateman, S., Liu, W., McCalla, G., Greer, J., Gašević, D., Eap, T., Richards, G., Hammouda, K., Shehata, S., Kamel, M., Karray, F., Jovanović, J. (2006). Issues and directions with educational metadata. In Proceedings of the 3rd annual LORNET conference on intelligent, interactive, learning object repositories network (CD Edition), Vol. 9. Vancouver: LORNET.
Brooks, C., Erickson, G., Greer, J., Gutwin, C. (2014). Modelling and quantifying the behaviours of students in lecture capture environments. Computers & Education, 75, 282–292. https://doi.org/10.1016/j.compedu.2014.03.002. http://www.sciencedirect.com/science/article/pii/S0360131514000591.
Brooks, C., Greer, J., Gutwin, C. (2014). The data-assisted approach to building intelligent technology-enhanced learning environments. In Larusson, J.A., & White, B. (Eds.) Learning Analytics: From Research to Practice (pp. 123–156). New York: Springer. https://doi.org/10.1007/978-1-4614-3305-7_7.
Brooks, C., Liu, W., Hansen, C., McCalla, G., Greer, J. (2007). Making sense of complex learner data. In Assessment of Group and Individual Learning through Intelligent Visualization Workshop (AGILeViz) (p. 28).
Brown, A.L., & et al. (1982). Learning, remembering, and understanding. technical Report No. 244. https://eric.ed.gov/?id=ED217401.
Buckingham Shum, S., Ferguson, R., Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9. https://learning-analytics.info/journals/index.php/JLA/issue/view/463.
Bull, S., & Kay, J. (2010). Advances in intelligent tutoring systems, studies in computational intelligence. In Nkambou, R., Bourdeau, J., Mizoguchi, R. (Eds.) (pp. 301–322)https://doi.org/10.1007/978-3-642-14363-2_15. Berlin: Springer.
Chen, B., Fan, Y., Zhang, G., Wang, Q. (2017). Examining motivations and self-regulated learning strategies of returning MOOCs learners. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference, LAK ’17. Event-place: Vancouver, British Columbia, Canada (pp. 542–543). New York: ACM. https://doi.org/10.1145/3027385.3029448.
Chen, B., Knight, S., Wise, A. (2018). Critical issues in designing and implementing temporal analytics. Journal of Learning Analytics. https://opus.lib.uts.edu.au/handle/10453/131162.
Dansereau, D.F., Long, G.L., McDonald, B.A., Actkinson, T.R., Ellis, A.M. (1975). Effective learning strategy training program: development and assessment. Tech. rep., Texas Christian Univ Fort Worth Inst For The Study Of Cognitive Systems, https://apps.dtic.mil/docs/citations/ADA014722.
Davis, D., Chen, G., Hauff, C., Houben, G.J. (2016). Gauging MOOC learners’ adherence to the designed learning path. International educational data mining society, https://eric.ed.gov/?id=ED592664.
Derry, S.J., & Murphy, D.A. (1986). Designing systems that train learning ability: From theory to practice. Review of Educational Research, 56 (1), 1–39.
Dicheva, D., Mizoguchi, R., Greer, J.E. (2009). Semantic web technologies for e-learning. Amsterdam: IOS Press. Google-Books-ID: Afwzaz3sz4QC.
Dimitrova, V., Self, J., Brna, P. (2001). Applying interactive open learner models to learning technical terminology. In Bauer, M., Gmytrasiewicz, P.J., & Vassileva, J. (Eds.) User Modeling 2001, Lecture notes in computer science (pp. 148–157). Berlin: Springer. https://doi.org/10.1007/3-540-44566-8_15.
Diseth, G. (2007). Approaches to learning, course experience and examination grade among undergraduate psychology students: testing of mediator effects and construct validity. Studies in Higher Education, 32(3), 373–388. https://doi.org/10.1080/03075070701346949. https://srhe.tandfonline.com/doi/full/10.1080/03075070701346949.
Dunlosky, J. (2013). Strengthening the student toolbox: Study strategies to boost learning. American Educator, 37(3), 12–21. https://eric.ed.gov/?id=EJ1021069.
Entwistle, N., & Ramsden, P. (2015). Understanding student learning (Routledge revivals). Routledge.
Ferguson, R., Clow, D., Beale, R., Cooper, A.J., Morris, N., Bayne, S., Woodgate, A. (2015). Moving Through MOOCS: pedagogy, learning design and patterns of engagement. In Conole, G., Klobučar, T., Rensing, C., Konert, J., & Lavoué, E. (Eds.) Design for teaching and learning in a networked world, lecture notes in computer science, (pp. 70–84). Cham: Springer International Publishing, https://doi.org/10.1007/978-3-319-24258-3_6.
Field, A. (2013). Discovering statistics using IBM SPSS statistics. SAGE. Google-Books-ID: c0Wk9IuBmAoC.
Fincham, E., Gašević, D., Jovanović, J., Pardo, A. (2019). From study tactics to learning strategies: An analytical method for extracting interpretable representations. IEEE Transactions on Learning Technologies, 12 (1), 59–72. https://doi.org/10.1109/TLT.2018.2823317 Conference Name: IEEE Transactions on Learning Technologies.
Gabadinho, A., Ritschard, G., Mueller, N.S., Studer, M. (2011). Analyzing and visualizing state sequences in R with TraMineR. Journal of Statistical Software, 40(4), 1–37. https://doi.org/10.18637/jss.v040.i04, https://archive-ouverte.unige.ch/unige:16809.
Gašević, D., Mirriahi, N., Dawson, S., Joksimović, S. (2017). Effects of instructional conditions and experience on the adoption of a learning tool. Computers in Human Behavior, 67, 207–220 . https://doi.org/10.1016/j.chb.2016.10.026, https://linkinghub.elsevier.com/retrieve/pii/S0747563216307270.
Garcia, T., & Pintrich, P.R. (1996). Assessing students’ motivation and learning strategies in the classroom context: The Motivated Strategies for Learning Questionnaire. In Birenbaum, M., & Dochy, F.J.R.C. (Eds.) Alternatives in assessment of achievements, learning processes and prior knowledge, evaluation in education and human services (pp. 319–339). Dordrecht : Springer. https://doi.org/10.1007/978-94-011-0657-3_12.
Gasevic, D., Jovanovic, J., Pardo, A., Dawson, S. (2017). Detecting learning strategies with analytics: links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113–128 . https://doi.org/10.18608/jla.2017.42.10. https://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/5085.
Gijbels, D., & Dochy, F. (2006). Students’ assessment preferences and approaches to learning: can formative assessment make a difference? Educational Studies, 32(4), 399–409. https://doi.org/10.1080/03055690600850354.
Greene, J.A., & Azevedo, R. (2007). A theoretical review of Winne and Hadwin’s model of self-regulated learning: new perspectives and directions. Review of Educational Research, 77(3), 334–372 . https://doi.org/10.3102/003465430303953. American Educational Research Association.
Greer, J., Thompson, C., Banow, R., Frost, S. (2016). Data-driven programmatic change at universities: What works and how. In Proceedings of the 1st learning analytics for curriculum and program quality improvement workshop (PCLA 2016), (Vol. 25 pp. 30–33 ).
Hadwin, A.F., Nesbit, J.C., Jamieson-Noel, D., Code, J., Winne, P.H. (2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2(2), 107–124. https://doi.org/10.1007/s11409-007-9016-7.
Holstein, K., McLaren, B.M., Aleven, V. (2019). Designing for complementarity: teacher and student needs for orchestration support in AI-enhanced classrooms. In Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., & Luckin, R. (Eds.) Artificial intelligence in education, lecture notes incomputer science(pp. 157–171). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-23204-7_14.
Howell, A.J., & Watson, D.C. (2007). Procrastination: Associations with achievement goal orientation and learning strategies. Personality and Individual Differences, 43 (1), 167–178. https://doi.org/10.1016/j.paid.2006.11.017, http://www.sciencedirect.com/science/article/pii/S0191886906004764.
Huang, X., McCalla, G.I., Greer, J.E., Neufeld, E. (1991). Revising deductive knowledge and stereotypical knowledge in a student model. User Modeling and User-adapted Interaction, 1(1), 87–115 . https://doi.org/10.1007/BF00158953, http://link.springer.com/10.1007/BF00158953.
Jansen, R.S., van Leeuwen, A., Janssen, J., Kester, L., Kalz, M. (2017). Validation of the self-regulated online learning questionnaire. Journal of Computing in Higher Education, 29(1), 6–27. https://doi.org/10.1007/s12528-016-9125-x.
Janssenswillen, G., Depaire, B., Swennen, M., Jans, M., Vanhoof, K. (2019). bupaR: Enabling reproducible business process analysis. Knowledge-Based Systems, 163, 927–930. https://doi.org/10.1016/j.knosys.2018.10.018, http://www.sciencedirect.com/science/article/pii/S0950705118305045.
Jovanović, J., Gašević, D., Dawson, S., Pardo, A., Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33(4), 74–85.
Jovanovic, J., Gasevic, D., Brooks, C., Devedzic, V., Hatala, M., Eap, T., Richards, G. (2008). LOCO-Analyst: semantic web technologies in learning content usage analysis. International Journal of Continuing Engineering Education and Life Long Learning, 18(1), 54–76. https://doi.org/10.1504/IJCEELL.2008.016076, https://www.inderscienceonline.com/doi/abs/10.1504/IJCEELL.2008.016076. Inderscience Publishers.
Kizilcec, R.F., Pérez-Sanagustín, M., Maldonado, J.J. (2016). Recommending self-regulated learning strategies does not improve performance in a MOOC. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale, L@S ’16. Event-place: Edinburgh, Scotland, UK (pp. 101–104). New York: ACM. https://doi.org/10.1145/2876034.2893378.
Kizilcec, R.F., Pérez-Sanagustín, M., Maldonado, J.J. (2017). Self-regulated learning strategies predict learner behavior and goal attainment in massive open online courses. Computers & Education, 104, 18–33 . https://doi.org/10.1016/j.compedu.2016.10.001. http://www.sciencedirect.com/science/article/pii/S0360131516301798.
Knight, S., Wise, A.F., Chen, B. (2017). Time for change: Why learning analytics needs temporal analysis. Journal of Learning Analytics, 4 (3), 7–17–7–17. https://doi.org/10.18608/jla.2017.43.2, https://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/5847.
Kovanovic, V., Gašević, D., Dawson, S., Joksimovic, S., Baker, R. (2015). Does time-on-task estimation matter? Implications on validity of learning analytics findings. Journal of Learning Analytics, 2(3), 81–110. https://doi.org/10.18608/jla.2015.23.6, https://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/4501.
Liu, Z., He, J., Xue, Y., Huang, Z., Li, M., Du, Z. (2015). Modeling the learning behaviors of massive open online courses. In 2015 IEEE international conference on big data (Big Data) (pp. 2883–2885): IEEE .
Lockyer, L., Heathcote, E., Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://doi.org/10.1177/0002764213479367.
Lust, G., Elen, J., Clarebout, G. (2013). Regulation of tool-use within a blended course: Student differences and performance effects. Computers & Education, 60(1), 385–395. https://doi.org/10.1016/j.compedu.2012.09.001, http://www.sciencedirect.com/science/article/pii/S0360131512002059.
Maldonado-Mahauad, J., Pérez-Sanagustín, M., Moreno-Marcos, P.M., Alario-Hoyos, C., Muñoz-Merino, P.J., Delgado-Kloos, C. (2018). Predicting learners’ success in a self-paced MOOC through sequence patterns of self-regulated learning. In Pammer-Schindler, V., Pérez-Sanagustín, M., Drachsler, H., Elferink, R., & Scheffel, M. (Eds.) Lifelong technology-enhanced learning, lecture notes in computer science (pp. 355–369). Cham : Springer International Publishing. https://doi.org/10.1007/978-3-319-98572-5_27.
Malmberg, J., Järvelä, S., Kirschner, P.A. (2014). Elementary school students’ strategic learning: does task-type matter? Metacognition and Learning, 9(2), 113–136. https://doi.org/10.1007/s11409-013-9108-5.
Mandinach, E., & Greer, J. (1992). Models of students’ metacognition, motivation, and learning strategies. In Jones, M., & Winne, P.H. (Eds.) Adaptive learning environments, NATO ASI Series (pp. 251–256). Berlin: Springer. https://doi.org/10.1007/978-3-642-77512-3_14.
Mangaroska, K., & Giannakos, M.N. (2018). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, pp. 1–1. https://doi.org/10.1109/TLT.2018.2868673 .
Matcha, W., Gašević, D., Ahmad Uzir, N., Jovanović, J., Pardo, A., Maldonado-Mahauad, J., Pérez-Sanagustín, M. (2019). Detection of learning strategies: a comparison of process, sequence and network analytic approaches. In Scheffel, M., Broisin, J., Pammer-Schindler, V., Ioannou, A., & Schneider, J. (Eds.) Transforming learning with meaningful technologies, Lecture notes in computer science (pp. 525–540). Cham: Springer International Publishing . https://doi.org/10.1007/978-3-030-29736-7_39.
Matcha, W., Gašević, D., Uzir, N.A., Jovanović, J., Pardo, A. (2019). Analytics of learning strategies: associations with academic performance and feedback. In Proceedings of the 9th international conference on learning analytics & knowledge, LAK19 (pp. 461–470). New York: Association for Computing Machinery. https://doi.org/10.1145/3303772.3303787.
McCalla, G., & Greer, J.E. (1994). Student modelling: the key to individualized knowledge-based instruction. Berlin: Springer.
McKeachie, W.J., Pintrich, P.R., Lin, Y.G. (1985). Teaching learning strategies. Educational Psychologist, 20 (3), 153–160. https://doi.org/10.1207/s15326985ep2003_5.
Meyers, J., Lytle, S., Palladino, D., Devenpeck, G., Green, M. (1990). Think-aloud protocol analysis: an investigation of reading comprehension strategies in fourth-and fifth-Grade students. Journal of Psychoeducational Assessment, 8 (2), 112–127. https://doi.org/10.1177/073428299000800201.
Milligan, C., & Littlejohn, A. (2015). Self-regulated learning behaviour and MOOC participation, https://researchonline.gcu.ac.uk/en/publications/self-regulated-learning-behaviour-and-mooc-participation.
Milligan, C., Littlejohn, A., Margaryan, A. (2013). Patterns of engagement in connectivist MOOCs. Journal of Online Learning and Teaching, 9, 149–159. http://jolt.merlot.org/vol9no2/milligan_0613.htm.
Morrison, G.R., & Guenther, P.F. (2000). Designing instruction for learning in electronic classrooms. New Directions for Teaching and Learning, 2000 (84), 15–22. https://doi.org/10.1002/tl.842, https://onlinelibrary.wiley.com/doi/abs/10.1002/tl.842.
Nguyen, Q., Huptych, M., Rienties, B. (2018). Using temporal analytics to detect inconsistencies between learning design and student behaviours. Journal of Learning Analytics, 5(3), 120–135. http://oro.open.ac.uk/58317/.
Oleksandra, P., & Shane, D. (2016). Untangling MOOC learner networks. In Proceedings of the sixth international conference on learning analytics & knowledge, LAK ’16. Event-place: Edinburgh, United Kingdom (pp. 208–212). New York: ACM. https://doi.org/10.1145/2883851.2883919, http://doi.acm.org/10.1145/2883851.2883919.
Orji, F.A., Greer, J., Vassileva, J. (2019). Exploring the effectiveness of socially-oriented persuasive strategies in education. In Oinas-Kukkonen, H., Win, K.T., Karapanos, E., Karppinen, P., & Kyza, E. (Eds.) Persuasive technology: development of persuasive and behavior change support systems, lecture notes in computer science(pp. 297–309). Cham : Springer International Publishing. https://doi.org/10.1007/978-3-030-17287-9_24.
Oxford, R.L. (1989). Use of language learning strategies: A synthesis of studies with implications for strategy training. System, 17(2), 235–247. https://doi.org/10.1016/0346-251X(89)90036-5, http://www.sciencedirect.com/science/article/pii/0346251X89900365.
Oxford, R.L., & Burry-Stock, J.A. (1995). Assessing the use of language learning strategies worldwide with the ESL/EFL version of the Strategy Inventory for Language Learning (SILL). System, 23 (1), 1–23. https://doi.org/10.1016/0346-251X(94)00047-A, http://www.sciencedirect.com/science/article/pii/0346251X9400047A.
Pask, G., & Scott, B.C.E. (1972). Learning strategies and individual competence. International Journal of Man-Machine Studies, 4(3), 217–253. https://doi.org/10.1016/S0020-7373(72)80004-X, http://www.sciencedirect.com/science/article/pii/S002073737280004X.
Pintrich, P.R., & et al. (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ), https://eric.ed.gov/?id=ED338122.
Pintrich, P.R. (2000). Chapter 14 - the role of goal orientation in self-regulated learning. In Boekaerts, M., Pintrich, P.R., & Zeidner, M. (Eds.) Handbook of self-regulation (pp. 451–502). San Diego : Academic Press. https://doi.org/10.1016/B978-012109890-2/50043-3, http://www.sciencedirect.com/science/article/pii/B9780121098902500433.
Pintrich, P.R., & Maehr, M.L. (Eds.). (2002). New directions in measures and methods, 1 edn. No. 12 in Advances in motivation and achievement. Amsterdam: JAI, An Imprint of Elsevier Science. OCLC: 248047527.
Postareff, L., Parpala, A., & Lindblom-Ylänne, S. (2015). Factors contributing to changes in a deep approach to learning in different learning environments. Learning Environments Research, 18(3), 315–333 . https://doi.org/10.1007/s10984-015-9186-1.
Rachal, K.C., Daigle, S., Rachal, W.S. (2007). Learning problems reported by college students: are they using learning strategies? https://link.galegroup.com/apps/doc/A173375575/HRCA?sid=lms.
Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239–257. https://doi.org/10.1007/s11412-009-9070-z.
Reimann, P., Markauskaite, L., Bannert, M. (2014). e-Research and learning theory: What do sequence and process mining methods contribute? British Journal of Educational Technology, 45 (3), 528–540. https://doi.org/10.1111/bjet.12146, https://onlinelibrary.wiley.com/doi/abs/10.1111/bjet.12146https://onlinelibrary.wiley.com/doi/abs/10.1111/bjet.12146.
Rigney, J.W. (1978). 7 - Learning Strategies: A Theoretical Perspective11This research was supported by ONR contract number N00014-76-C-0838, and by ARPA order number 2284. The support and encouragement of Harold F. O’Neil, Jr., Cybernetics Technology Office, ARPA; and of Marshall Farr and Henry Halff, Personnel and Training Research Programs, ONR, is gratefully acknowledged. Views and conclusions contained in this document are those of the author and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the Defense Advanced Research Projects Agency or of the United States Government. In O’neil, H.F. (Ed.) Learning strategies (pp. 165–205): Academic Press. https://doi.org/10.1016/B978-0-12-526650-5.50012-5, http://www.sciencedirect.com/science/article/pii/B9780125266505500125.
Roll, I., & Winne, P.H. (2015). Understanding, evaluating, and supporting self-regulated learning using learning analytics. Journal of Learning Analytics, 2(1), 7–12. https://doi.org/10.18608/jla.2015.21.2https://learning-analytics.info/index.php/JLA/article/view/4491.
Romero, P., du Boulay, B., Cox, R., Lutz, R., Bryant, S. (2007). Debugging strategies and tactics in a multi-representation software environment. International Journal of Human-Computer Studies, 65 (12), 992–1009. https://doi.org/10.1016/j.ijhcs.2007.07.005http://www.sciencedirect.com/science/article/pii/S1071581907001000.
Saint, J., Gašević, D., Matcha, W., Uzir, N.A., Pardo, A. (2020). Combining analytic methods to unlock sequential and temporal patterns of self-regulated learning. In Proceedings of the tenth international conference on learning analytics & knowledge, LAK ’20 (pp. 402–411). Frankfurt: Association for Computing Machinery. https://doi.org/10.1145/3375462.3375487.
Saint, J., Gašević, D., & Pardo, A. (2018). Detecting learning strategies through process mining. In Pammer-Schindler, V., Pérez-Sanagustín, M., Drachsler, H., Elferink, R., & Scheffel, M. (Eds.) Lifelong technology-enhanced learning, lecture notes in computer science (pp. 385–398). Cham : Springer International Publishing. https://doi.org/10.1007/978-3-319-98572-5_29.
Schunk, D.H. (1989). Self-regulated learning and academic achievement: theory, research, and practice, springer series in cognitive development. In Zimmerman, B.J., & Schunk, D.H. (Eds.) (pp. 83–110). New York: Springer. https://doi.org/10.1007/978-1-4612-3618-4_4.
Scouller, K. (1998). The influence of assessment method on students’ learning approaches: Multiple choice question examination versus assignment essay. Higher Education, 35(4), 453–472. https://doi.org/10.1023/A:1003196224280.
Segal, J.W., Chipman, S.F., Glaser, R. (2014). Thinking and learning skills: Volume 1: relating instruction to research. Routledge.
Shaffer, D.W. (2004). Epistemic frames and islands of expertise: learning from infusion experiences. In Proceedings of the 6th international conference on learning sciences, ICLS ’04. International Society of the Learning Sciences. Event-place: Santa Monica, California (pp. 473–480). http://dl.acm.org/citation.cfm?id=1149126.1149184.
Shaffer, D.W., Collier, W., Ruis, A.R. (2016). A tutorial on epistemic network analysis: analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics, 3(3), 9–45. https://doi.org/10.18608/jla.2016.33.3https://learning-analytics.info/index.php/JLA/article/view/4329.
Sinha, T., Jermann, P., Li, N., Dillenbourg, P. (2014). Your click decides your fate: Inferring information processing and attrition behavior from MOOC video clickstream interactions. arXiv:1407.7131[cs].
Struyven, K., Dochy, F., Janssens, S., Gielen, S. (2006). On the dynamics of students’ approaches to learning: The effects of the teaching/learning environment. Learning and Instruction, 16(4), 279–294. https://doi.org/10.1016/j.learninstruc.2006.07.001http://www.sciencedirect.com/science/article/pii/S0959475206000508.
Wang, Q., Chen, B., Fan, Y., Zhang, G. (2018). MOOCs as an alternative for teacher professional development. Examining Learner Persistence in One Chinese MOOC.
Weinstein, C.E., Husman, J., Dierking, D.R. (2000). Chapter 22 - self-regulation interventions with a focus on learning strategies. In Boekaerts, M., Pintrich, P.R., & Zeidner, M. (Eds.) Handbook of self-regulation (pp. 727–747). San Diego: Academic Press. https://doi.org/10.1016/B978-012109890-2/50051-2, http://www.sciencedirect.com/science/article/pii/B9780121098902500512.
Whitelock-Wainwright, A., Tsai, Y.S., Lyons, K., Kaliff, S., Bryant, M., Ryan, K., Gašević, D. (2020). Disciplinary differences in blended learning design: a network analytic study. In Proceedings of the tenth international conference on learning analytics & knowledge, LAK ’20 (pp. 579–588). Frankfurt: Association for Computing Machinery. https://doi.org/10.1145/3375462.3375488.
Willoughby, T., Wood, E., Khan, M. (1994). Isolating variables that impact on or detract from the effectiveness of elaboration strategies. Journal of Educational Psychology, 86(2), 279–289. https://doi.org/10.1037/0022-0663.86.2.279.
Winne, P.H. (2006). How software technologies can improve research on learning and bolster school reform. Educational Psychologist, 41(1), 5–17. https://doi.org/10.1207/s15326985ep4101_3, http://www.tandfonline.com/doi/abs/10.1207/s15326985ep4101_3.
Winne, P.H. (2010). Improving measurements of self-regulated learning. Educational Psychologist, 45(4), 267–276. https://doi.org/10.1080/00461520.2010.517150.
Winne, P.H. (2011). A cognitive and metacognitive analysis of self-regulated learning : faculty of education, Simon Fraser University, Burnaby, Canada. https://doi.org/10.4324/9780203839010-6, https://www.taylorfrancis.com/.
Winne, P.H., & Hadwin, A.F. (1998). Studying as self-regulated engagement in learning. In Metacognition in educational theory and practice (pp. 277–304).
Winne, P.H., Jamieson-Noel, D., Muis, K. (2002). Methodological issues and advances in researching tactics, strategies, and self-regulated learning. Advances in motivation and achievement: New directions in measures and methods, 12, 121–155.
Winne, P.H., & Nesbit, J.C. (2009). 14 Supporting self-regulated learning with cognitive tools. Handbook of metacognition in education p. 259.
Winne, P.H., & Perry, N.E. (2000). Chapter 16 - measuring self-regulated learning. In Boekaerts, M., Pintrich, P.R., & Zeidner, M. (Eds.) Handbook of self-regulation (pp. 531–566). San Diego: Academic Press. https://doi.org/10.1016/B978-012109890-2/50045-7, http://www.sciencedirect.com/science/article/pii/B9780121098902500457.
Winter, M., Brooks, C., Greer, J. (2005). Towards best practices for semantic web student modelling. In Proceedings of the 12th international conference on artificial intelligence in education (pp. 694–701).
Zapata-Rivera, J.D., & Greer, J.E. (2004). Interacting with inspectable bayesian student models. International Journal of Artificial Intelligence in Education, 4(2), 127–163. https://content.iospress.com/articles/international-journal-of-artificial-intelligence-in-education/jai14-2-01. IOS Press.
Zhou, M., & Winne, P.H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. https://doi.org/10.1016/j.learninstruc.2012.03.004, http://www.sciencedirect.com/science/article/pii/S0959475212000217.
Acknowledgements
We would like to thank icourse163.org for providing the anonymized learning data, and team members of X-Learning Center (GSE, Peking University) for the in-depth discussions about the learning design of this MOOC. We acknowledge Dr. Torsten Juelich for his professional feedback about language and logic of the paper, which greatly improved the writing. This study was funded by the National Natural Science Foundation of China (NSFC-61532001) and Peking University (NO. 2020YBC18).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest in this study. Authors Yizhou Fan and Qiong Wang were part of the MOOC development team, so the analysis of learning design was triangulated by other co-authors who were not invested in the development of the MOOC.
Additional information
Publisher’s Note
SpringerNature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article belongs to the Topical Collection: A festschrift in honour of Jim Greer
Guest Editors: Gord McCalla and Julita Vassileva
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Fan, Y., Matcha, W., Uzir, N.A. et al. Learning Analytics to Reveal Links Between Learning Design and Self-Regulated Learning. Int J Artif Intell Educ 31, 980–1021 (2021). https://doi.org/10.1007/s40593-021-00249-z
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40593-021-00249-z