Most undergraduate science, technology, engineering, and math (STEM) degree programs begin with large, lecture-style courses that enroll hundreds. Such gateway courses are known to challenge learners (e.g., Perez et al., 2014). For example, those enrolled in courses like Introductory Biology must master learning objectives that cover many topics and at declarative, conceptual, and applied levels of understanding (American Association for the Advancement of Science, 2011; Bernacki, 2023). In response to calls to redevelop instruction to improve its quality and the way learners engage in coursework (Olson & Riordan, 2012), many instructors now frequently scaffold learners’ engagement with course content by incorporating instructional designs that prioritize active learning (Lombardi et al., 2021; Theobald et al., 2020). These redeveloped courses involve instructional designs that require students to actively acquire, rehearse, and evaluate their knowledge as they read and study textbooks, complete assignments, watch videos with embedded formative assessments, and engage with others (Lombardi et al., 2021). Navigating such complex course activities in pursuit of learning objectives requires that students be self-regulated learners (SRL; Greene, 2018). The SRL process involves a loosely sequenced cycle in which learners must develop an understanding of the task environment, including the affordances it provides and constraints it imposes, then set a learning goal and plan for achieving it. Learners can thereafter enact that plan, often involving the use of one or more cognitive learning strategies in pursuit of the goal. As they enact a strategy, learners monitor whether they are making progress towards their goal via a set of metacognitive judgements, and decide to sustain their strategy use or whether they should adapt their learning by enacting a different strategy, altering their goal, or adjusting some other aspect of their learning environment or process (Winne & Hadwin, 1998). In practice, however, many early undergraduate science learners reported they felt underprepared to engage in SRL when they enroll in undergraduate coursework (e.g., Perez et al., 2014). Active learning designs can sometimes be overwhelming to students, who reported that they lack familiarity with active learning designs, and possess insufficient confidence, time, or preparation to engage with such instruction (Shekhar et al., 2020). Students who reported these challenges thus require scaffolding to develop the ability to engage the cognitive strategies that active learning designs promote, and to engage in the self-regulatory practices that are needed to manage the more challenging workloads that active STEM learning courses require.

Methods of providing cognitive strategy and SRL scaffolding include workshops, trainings, and embedded classroom activities that develop students’ learning skills. SRL trainings vary widely in the methods of instruction, time and effort cost to complete training, and topics covered (Dignath et al., 2008b). Coursework, face-to-face workshops, and more recently, digital skill training programs have been shown to effectively develop undergraduates’ capacity for SRL (e.g., Theobald, 2021). Meta-analyses confirm that completing learning skill training improves academic skills and performance, but the moderator analyses they include provide little information as to why such programs are effective at developing and scaffolding learners’ ability to self-regulate (see Table 1). As a result, the processes of scaffolding and training SRL skills remains unclear. In recently developed models that recommend promising ways to train SRL skill, scholars propose the importance of training cognitive strategies and metacognitive processes (e.g., McDaniel & Einstein, 2020), and recommend that trainings should introduce surface-level declarative knowledge about a skill and consolidate that knowledge to a conceptual level of understanding (Hattie & Donoghue, 2016). These authors further underscore the importance of including activities that promote transfer and prepare learnings to apply the SRL skills they now understand when they are productive for learning in authentic educational settings in the future (Hattie & Donoghue, 2016; McDaniel & Einstein, 2020; see Table 2).

Table 1 Weighted Effect Sizes on Performance Achieved by Strategy Instruction from Meta-Analyses
Full size table
Table 2 Summary of the Key Features of Theoretical Models for Scaffolding Learning Skill Training
Full size table

In this paper, we report on a pair of studies designed to evaluate two versions of a training program that can be embedded into high-enrollment STEM courses to scaffold students’ development of learning skills. We adopt the SRL framework by Winne and Hadwin (1998) in which they describe (1) the cognitive strategies that learners enact when they engage in learning tasks, as well as (2) the metacognitive processes involved in the preliminary task definition, goal setting, and planning phases of the SRL cycle, and (3) in the metacognitive monitoring and control process that coincides with strategy enactment: a time when students monitor whether their strategy use is advancing them to their goals, or whether adaptation is required. In alignment with these features of the Winne and Hadwin (1998) model of SRL, we examine learners’ degree of mastery of the SRL skills taught in three modules of a digital training program, as can be re-analyzed from an experimental condition in a prior randomized control study (i.e., Science of Learning to Learn training; Bernacki et al., 2020) and those analyses can be replicated in a second study of similar experimental design. In addition to examining the types of SRL skills learners master and how those predict the future learning they are designed to support, we consider the depth to which students have mastered such skills. SRL scholars like Efklides (2011) describe how the skills necessary to engage in SRL are acquired. In the MASRL model, Efklides proposed (2011) that individuals develop their learning skills based on their metacognitive experiences from which they derive metacognitive knowledge of learning processes including enactment of cognitive strategies and methods of engaging in metacognitive, behavioral, and environmental regulation. Training programs should thus provide exposure and opportunities to engage in SRL in order to scaffold learners’ development of metacognitive knowledge of SRL skills.

Subsequent models of skill training and syntheses that document their effects on learning provide guidance and evidence regarding the training components that explain variance in learners’ achievement (Hattie & Donoghue, 2016; McDaniel & Einstein, 2020). A recent meta-analysis of digital methods of training learning skills by Theobald (2021) documents that many of these trainings improve learning outcomes, and moderator analyses highlight the specific topics and activities included in the trainings. For example, a pair of studies by Bernacki et al. (i.e., Bernacki et al., 2020; Bernacki et al., 2021) included in the meta-analysis documented superior performances of science and math learners on subsequent exams after they completed training compared to peers who were randomly assigned into groups that spent equivalent time on additional study of course topics. In each study, main effects were reported but the degree to which individuals who completed training demonstrated skill mastery of cognitive versus metacognitive SRL skills, and the depth of understanding of these skills and how to apply them were not analyzed as moderators on the observed main effects.

In this paper, we re-examine the data from a sample of participants who completed the Science of Learning to Learn training, with the aim to better understand how evidence of their skill mastery during training predicted later STEM course achievement. This modeling approach can reflect the assumptions of Efklides (2011) about metacognitive knowledge and Hattie and Donoghue (2016) about the importance of training students to different levels of understanding of learning skills, and whether skills shown to be understood at declarative versus conceptual levels of understanding can predict learners’ success in coursework after training is complete. In the first study, we organized our research questions around (1) the types of SRL skill mastery found to benefit students’ performance in undergraduate STEM coursework and (2) the depth of understanding of those skills that, when observed, might predict subsequent performance in a course. This approach mirrors the design paradigm proposed by Hattie and Donoghue (2016; Table 2), who focused on scaffolding surface understanding of a learning skill, then consolidating that knowledge to form a deeper understanding of the learning skill, prior to preparing learners to transfer the skill to applied settings. In a second study, we extended our inquiry to include a second sample of students who completed a redesigned version of the original Science of Learning to Learn training to provide additional evidence for such predictive relations. In this re-analysis (i.e., of data from Bernacki et al., 2020) and replication study sequence, we fit path models to explore how metacognitive knowledge of SRL skills during training and the depth of such knowledge might predict exam scores on later exams in their biology courses (i.e., anatomy and physiology in Study 1; introductory biology in Study 2).

These within group analyses can shed additional light on the main effects documented in prior published studies and draw such study designs into closer alignment with models of metacognitive knowledge of learning strategies (Efklides, 2011) and conceptual models that consider skill training design through a theoretical lens (Hattie & Donoghue, 2016; McDaniel & Einstein, 2020). In our prior papers documenting the effects obtained by these trainings (i.e., Bernacki et al., 2020; Bernacki et al., 2021), we initially compared groups who completed the three-module training with those who were assigned an alternate task that involved additional study of course topics (i.e., aligned to course objectives and beneficial to undertake for course performance, but that were not didactically focused on developing learning skills). Overall, we found that on subsequent exams, after controlling for prior achievement, those who completed the learning skill training outperformed those who completed the alternate activity, which served as a comparison group. However, the number of students assigned to the training condition who actually completed it was lower than anticipated. These findings suggested that training learning skills produced promising academic achievement outcomes. Despite this, the three-module training program had low completion rates, which limited the number of students who obtained full benefits from the training. Thus, we resolved to simplify the training, and undertook a redesign effort in alignment with multimedia learning theory (Mayer, 2021). By adopting a multimedia approach, we aimed to sustain the didactic benefits of the training and make it more efficient and engaging by splitting our delivery of content across audio and visual channels of information. In a second study, we replicated our analyses from Study 1 to investigate whether the redesign would change the way that demonstrations of learning skills, now acquired from a video-based multimedia training design, related to performance on subsequent biology exams.

Literature review

Scaffolding development of learning skills

Learning skills can be developed. This long-held assumption (e.g., Hattie et al., 1996; Hattie, 2007; Hattie & Donoghue, 2016) has motivated hundreds of teachers, instructional designers, and researchers to develop methods to help learners acquire skills that are theorized to make them more effective learners. These methods come in many forms and include courses implemented in K-12 settings (Dignath & Büttner, 2008a) and workshops offered at universities’ academic success and learning centers (Jansen et al., 2019; Theobald, 2021). Such initiatives are meant to prepare students for undergraduate coursework, and are offered in high school before matriculation (Bryan et al., 2015), in the summer prior to fall enrollment (Verrell et al., 2015), during the first semester as a stand-alone seminar (Porter & Swing, 2006), as a preliminary topic included prior to the domain-specific learning objectives within a course (Hensley et al., 2021), and as a supplement to be completed outside of schedule course meetings (Bernacki et al., 2020, Bernacki et al., 2021).

Evidence for the effectiveness of interventions that train learning skills

Meta-analyses that synthesize decades of empirical research have shown that these learning skill training efforts are productive methods for improving students’ self-reported skillfulness, based on responses to surveys completed after training (e.g., Broadbent & Poon, 2015; Dignath & Buttner, 2008; Jansen et al., 2019; Theobald, 2021). In addition to self-reported skillfulness, many who develop their skills via trainings have been found to perform well on subsequent tasks immediately after trainings, and in some cases, the lessons learned and skills developed during training have been observed to sustain in course contexts after training and when scaffolding has been removed (e.g., planning and monitoring; Bernacki et al., 2020).

Scaffolding the development of specific learning skills has been found to impact achievement on subsequent academic tasks. Moderator analyses that examine effects on subsequent course performance indicate magnitudes of effect in the medium to large range (Dignath et al., 2008; Donker et al., 2014; Hattie et al., 1996). One of the key moderators of the effects of interventions designed to develop learning skills is the set of learning skills included in a training program. Table 1 documents that when trainings include metacognition and regulation topics in addition to cognitive learning strategies, effect sizes tend to be higher than when they do not (e.g., Dignath et al., 2008). We thus consider the importance of training not just cognitive learning strategies, but SRL skills more broadly.

The inclusion of learning skills in interventions designed to train self-regulated learners

The learning skills that designers have opted to include in training vary considerably, as evidenced by meta-analyses documented in Table 1, which discriminated skill training designs that focus exclusively on cognitive strategies, or that also include metacognitive and regulatory strategies. Additionally, emergent models from cognitive and educational psychologists propose not only coverage of learning skills, but also how methods of training can lead to surface-level and deeper understanding of learning skills, and that such approaches can further benefit learners’ future application of skills and achievement of desired outcomes (Hattie & Donoghue, 2016). We first consider the evidence for the types of learning skills that skill training programs have included, and the theories that explain why they improve learning when they are enacted, and how specific strategies may benefit mastery of course learning objectives. Thereafter, we document published conceptual models that reference learning theories and evidence of the value of training skills to declarative, conceptual, and procedural levels of knowledge to ensure they are introduced, consolidated, and able to be transferred (Hattie & Donoghue, 2016). We consider these dimensions of skill training through a self-regulated learning framework that abides these theoretical assumptions (Winne & Hadwin, 1998), and document the specific cognitive processes that are essential to the enactment of SRL in pursuit of specific learning goals, before elaborating on the metacognitive and regulatory processes included in learning skill training models.

Types of learning skills: a self-regulated learning framework

SRL derives from the social cognitive tradition and involves the skillful regulation of cognition, affect, and behavioral processes in pursuit of a goal (Schunk & Greene, 2018). Theorists who describe the acquisition of SRL skills describe this as a process of accruing metacognitive experiences that develop into metacognitive knowledge of such skills, which, once developed, can be employed during a learning task (Efklides, 2011). In order to help learners acquire the metacognitive knowledge they need to engage these SRL skills, we designed our digital training to focus on cognitive strategies known to benefit learning in STEM coursework involving declarative, conceptual, and procedural learning objectives (Dunlosky et al., 2013; Koedinger et al., 2012). Thereafter, training was focused on developing learners’ metacognitive knowledge related to the regulation of cognition, environment, and behavior in order to increase their skill in selecting appropriate strategies and establish conditions where they could be enacted and monitored until goals were met, as well as appropriate behavioral cues and environments that would enable them to do so.

Cognitive Strategies. Researchers from both cognitive and educational psychology have produced a substantial body of evidence concerning the cognitive learning strategies that have been theorized and shown to benefit performance on academic tasks (Dunlosky et al., 2013). Theoretical models about knowledge, learning, and instruction further document how specific cognitive learning strategies are most efficient for the robust learning of declarative, conceptual, and procedural knowledge (i.e., the Knowledge-Learning-Instruction framework; Koedinger et al., 2012). Syntheses of such research on the use of cognitive strategies including reports by Dunlosky et al. (2013) appraise a host of strategies that have been studied in laboratory and applied contexts. These strategies include re-reading and highlighting, mnemonic development and memorizing, and rehearsal strategies involving flash cards, all of which students commonly report using in college coursework (Karpicke et al., 2009). Reviews also include the infrequently reported but empirically well-supported methods including distributed or spaced practice methods (e.g., Carpenter et al., 2012) that involve self-testing of declarative knowledge through retrieval practice (e.g., Roediger & Karpicke, 2006) and generative strategies including self-explanation, summarizing and elaborative interrogation procedures that target knowledge of procedures and concepts (e.g., Renkl, 2007). In their monograph that evaluates the utility of the cognitive strategies that psychologists study, Dunlosky et al. (2013) rated just a handful of the methods as having substantial evidence of being beneficial to learning. This endorsement is limited to retrieval practice (i.e., distributed self-testing via retrieval practice), which has indeed been consistently beneficial to the learning of declarative, factual knowledge that can be retrieved from long term memory, as well as for retrieval and reconstruction of conceptual knowledge (e.g., Agarwal, 2019; Agarwal et al., 2021).

Additional scholars recommend that, because retrieval practice is highly efficient for learning declarative knowledge but academic tasks typically include learning objectives that span multiple levels of understanding, it is beneficial to develop learners’ ability to engage multiple cognitive strategies, including generative strategies that can be more suited to developing conceptual understanding (Roelle et al., 2022). Generative strategies require the learner to generate products that provide evidence of their knowledge while also strengthening it in the process, and include not only retrieval but also explanation, elaboration, and questioning methods that help learners, select, organize, and integrate information (Fiorella & Mayer, 2016). These processes are closely connected to the Interactive, Constructive, Active, and Passive (ICAP) framework proposed by Chi and Wylie (2014), which guides the design of active learning STEM courses where students are encouraged to use course materials to engage in such generative practices. Additional guidance on how specific generative learning practices can benefit learning is provided in the Knowledge, Learning Instruction (KLI) framework by Koedinger and colleagues (2012). Like Dunlosky et al.’s (2013) qualified and contextual endorsements of strategies like self-explanation as moderately beneficial and of greater value under some task conditions, Koedinger et al. (2012; i.e., in Table 3) draw attention to the value of prompting self-explanation in tasks that involve conceptual knowledge and the incorporation of such practices into study when it involves diagrams and worked examples of phenomena like those found in science coursework and other STEM domains. Because of their demonstrable effects on performance in multiple STEM domains and the key role that these cognitive strategies play in the enactment phase during self-regulated learning, the first module of the Science of Learning to Learn explicitly introduces three cognitive strategies deemed to be highly effective for learning: retrieval practice, self-explanation, and the spacing of these practices, per Dunlosky et al. (2013), Koedinger et al. (2012) and others.

Table 3 Sequence of Topics Addressed in Science of Learning to Learn Digital and Multimedia Training Editions
Full size table

Metacognitive Strategies. The cognitive learning strategies that individuals enact in order to pursue their learning goals are supervised by an ongoing metacognitive monitoring process (Winne, 2001; Winne & Hadwin, 1998) in which learners attend to task conditions, consider the standard they aim to achieve in their learning process, and select and adapt the tactics they employ to construct a product that satisfies this criterion. In order to determine which cognitive strategy is the best choice to enact, a learner must first define a task by appraising the learning goal and resources afforded in a task context (i.e., using course learning objectives and the resources aligned to them for task definition). Learners must then possess sufficient metacognitive knowledge of potential strategies to enact, as well as the ability to monitor whether their current knowledge state is closer to their goal than an earlier state (i.e., metacognitive monitoring and control). Accordingly, the second module of the Science of Learning to Learn intervention builds on the cognitive strategies introduced in the first module and introduces SRL as a process that involves discrete steps, including task definition based on learning objectives, planning and enactment of cognitive strategies that suit these learning goals, and monitoring practices that can be used to inform decisions to sustain or adapt strategy use while pursuing such goals (see Tables 2 and 4).

Table 4 Path Analysis Coefficients for Study 1 Models 1, 2, 3, 4 (N = 49) and Study 2 Model 5 (N = 62)
Full size table

Behavioral and Environmental Regulation Strategies. Because learners engage in a proximal learning task (i.e., studying) in the context of a larger one (i.e., one course, often of many taken simultaneously during a semester, completed on a campus), the self-regulation of learning is best considered as nested within a larger task: self-regulation of oneself in the educational context of university life (Greene, 2018). That is, learners must not only choose appropriate ways of engaging cognitively in learning tasks, but they must also manage their engagement in the larger environment where such study takes place. This includes a need to establish not only an adequate task environment as described in theoretical frameworks of SRL (Greene, 2018; Winne & Hadwin, 1998), but also a productive set of habits that can cue an individual to initiate learning, and to manage distractions while sustaining learning (Fiorella, 2020). Accordingly, the third module of Science of Learning to Learn addresses the need to self-regulate one’s learning environment and behaviors within it. Module topics include the setting and maintaining of goals and establishment of environmental cues that can prompt goal directed behavior (i.e., implementation intentions, Gollwitzer, 1999), as well as the maintenance of motivation necessary to sustain engagement in learning despite adversity (i.e., mental contrasting to preserve goal pursuit in the face of challenge; Kappes & Oettingen, 2014). In light of ample evidence that students struggle to establish productive learning environments at university (e.g., David et al., 2015), additional training is provided to teach students about eliminating distractions that can deplete cognitive resources (i.e., limiting extraneous visual and auditory stimuli, and establishing a schedule to diminish off task behaviors).

Models for scaffolding learning skills development to achieve a depth of knowledge

Based on Hattie and Donahue’s (2016) framework for developing learning skills, in the trainings investigated in the studies reported here, students receive direct instruction that includes (1) definitions of the learning skills, (2) an explanation of the underlying theory and evidence as to how the learning process works, and (3) empirical evidence that deployment of the recommended skills improve undergraduates’ academic performance (see Table 2). Thereafter, students engage in activities that afford them opportunities to rehearse and demonstrate their declarative and conceptual knowledge of the learning skills, and to engage in hypothetical planning activities that prompt them to appraise course objectives, resources, and assessment practices, so that they can plan how to use the recommended learning skills effectively. These learning activities double as assessment opportunities, where the responses students’ commit during activities provide evidence of the robustness of their declarative and conceptual knowledge of learning skills, and their ability to apply skills in an undergraduate course context (i.e., similar to the KLI framework; Koedinger et al., 2012).

Much like the level of understanding of topics identified in learning objectives can vary from simple declarative knowledge to conceptual and procedural levels, so too can learners’ level and depth of knowledge of learning skills. In SRL models that consider the acquisition of metacognitive knowledge of learning skills (e.g., Efklides, 2011), learners are understood to build their metacognitive knowledge about learning through their metacognitive experiences. These experiences accrue over time, and metacognitive knowledge about learning deepens in kind.

Prior research has determined the relative effects of training specific types of learning skills, but little attention has been paid to the depth of knowledge that learners gain about these learning skills. For example, meta-analyses primarily evaluate how cognitive and metacognitive processes associated with academic achievement serve as moderators of SRL training effects, but little consideration is given to the depth at which that knowledge is trained (see Dent and Koenka, 2016 for an example). This lack of investigation misaligns with emergent theories on development of learning skills, which propose knowledge of skills can be introduced at a surface level, deepened through consolidation, and made more apt to transfer through additional training activities (Hattie & Donoghue, 2016). Deep processing occurs when students use prior knowledge and make connections among different topics (Craik & Lockhart, 1982; Marton & Saljo, 1984; Ramsden, 1992). Deep processing strategies include elaboration (i.e., making meaningful connections from prior knowledge and among knowledge topics) and organization strategies (e.g., use of outlines, and concept maps; Schroeder et al., 2018). Learning of conceptual and application knowledge, in which students would have to understand the relationships between concepts and apply that knowledge to new information, would be considered deep processing. In contrast, shallow processing, such as learning of declarative or definitional-type information, in which students learn information without making connections to prior knowledge has been found to produce smaller effects on academic achievement (e.g., Garcia & Pintrich, 1994). Craig and Lockhart (1972) concluded that deep processing helps information reach long-term memory, in comparison to shallow processing. A student must be metacognitively aware of appropriate strategies (e.g., self-explanation) to employ during study in order to engage in deep processing and long-term retention of information. In order to develop students’ cognitive strategies, the metacognitive skills required to select, enact, monitor, and adapt strategy use, and the ability to transfer such knowledge into practice settings, learning skill training designers have begun to develop digital skill training approaches that can be offered to learners.

Affordances of digital skill training to efficiently scaffold learning and SRL

A considerable body of literature has accrued in more recent years on educators and researchers’ efforts to scaffolding learning and the acquisition and deployment of learning skills in digital spaces (Broadbent & Poon, 2015; de Bruijn-Smolders et al., 2016; Devolder et al., 2012; Theobald, 2021; Zheng, 2016). These reviews repeatedly demonstrate digital platforms’ potential to provide learners with opportunities to engage with resources that promote active learning relevant to course topics, as well as to provide learners with digital resources explicitly designed to scaffold enactment of strategies and learning processes. Unique among these reviews, Zheng (2016) includes not only a systematic review of the kinds of engagements afforded to individuals who learn in digital spaces, but also the amount of time students typically engaged with scaffolds and trainings that were observed to relate to academic performance. In prior meta-analyses focused on skill training programs and interventions (e.g., Hattie, 2008), the time cost of trainings has been of great interest to researchers, and also to those who might consider the adoption of resources designed to train students’ learning skills based on the effectiveness of the training, as well as the costs required to obtain such benefits.

Those who design digital scaffolds and trainings often do so in order to make these learning supports broadly available and scalable, but also to prioritize learning efficiency, where learning activities should be sufficient to achieve “robust learning” where products of learning activities endure, but the time cost required to achieve such robust outcomes is optimized (Koedinger et al., 2012, p. 761). Accordingly, designers must determine not only what skills to train, but how much time should be spent on training activities so that a level of understanding and degree of exposure provided by a training produces metacognitive knowledge that remains available for future retrieval and use.

In alignment with the design proposed by Hattie and Donoghue (2016), we designed [Science of Learnng to Learn] trainings to introduce then elaborate on topics, and assess learners’ knowledge of learning skills at each of these levels, with the intention to analyze associations between demonstrations of declarative, conceptual, and applied knowledge of learning skills with future exam performances so we could better understand how these findings could provide support for Hattie and Donoghue’s recommendations overall, and how they can inform the selection of learning skill training activities to incorporate into future training designs. Next, we present a study that examined how students learned within an original design of a digital learning skill training program and how such learning related to achievement. Thereafter, we describe a redesign of the training conducted in an effort to decrease the cognitive demand of the original training by redesigning it into a multimedia format, which was in turn intended to improve completion rates and efficiency when completing the training.

Two current studies to investigate how knowledge of learning skills predict achievement

In a recent study (Bernacki et al., 2020), we examined whether a brief, digital skill training titled “the Science of Learning to Learn” could positively affect the achievement of undergraduate science learners in the early undergraduate coursework where students often feel unprepared to learn. Results from that study demonstrated that completing a digital training designed to promote knowledge of and ability to apply cognitive, metacognitive, and behavioral and environmental regulation strategies improved student performance on subsequent exams. Completing this training required roughly two hours of students’ time (i.e., students typical completed in 90–120 min) and conferred an effect of about 1/3 of a letter grade on subsequent quizzes and exams (d ~ 0.3; e.g., from a B to a B + and corresponding grade point average difference of 3.4 to 3.7 on the 0 to 4 scale at U.S. universities). In this paper, we report on two subsequent analyses that investigate how the skills that participants previously demonstrated during training in these prior studies on skill training explain the performances of learners who complete digital training modules.

In Study 1, we focus on members of the treatment condition in the original experimental study (i.e., Bernacki et al., 2020) in order to evaluate how their performances during skill training – which indicate their understanding of cognitive, metacognitive, and behavioral and environmental regulation skills, at declarative, conceptual and application levels of metacognitive knowledge about learning skills – explain their performances on course exams that assess their learning of science content. By identifying the learning skills that appear to have the greatest effect on immediate and delayed exam performance, we can better understand how developing students’ particular learning skills may be critical to mastering curricular learning objectives. These insights can also inform the development of future interventions and courses redesigned to provide efficient, scalable digital learning support for learners whose current metacognitive knowledge about learning may be insufficient for challenging undergraduate coursework.

The second study we report summarizes one of these redesign efforts. In this version of the Science of Learning to Learn training, digital videos replaced the textual content in the original modules so that learners could save time by viewing video rather than reading, thus improving the learning efficiency obtained (Koedinger et al., 2012). As in the first study, we analyze the degree to which levels of knowledge of cognitive strategies, self-regulated learning, and behavioral and environmental regulation predicted performance of trained students in a second biology course. In Study 2, we also examined this sample of students’ (N = 62) data to determine whether the multimedia redesign achieved its intended effects of producing higher completion rates and lower completion times. Our final set of inferential testing spans the data from both samples to determine whether the associations between demonstrations of learning skill during multimedia-based training and biology exam performances were consistent across studies of training designs.

Study 1

Method

The original experimental study was conducted at a large public institution in the southwest region of the United States that held distinctions as a Minority-Serving Institution (MSI), Hispanic-Serving Institution (HSI), and Asian American, Native American, and Pacific Islander Serving Institution (AANAPISI).The university also typically enrolls a student body where the majority are first generation college students. The sample of learners observed in the experimental condition included all those who completed multiple modules and 80% or more of the learning activities (see Bernacki et al., 2020). When assessing training effects, we reasoned that exposure to the majority of topics was sufficient to consider individuals to have completed training with sufficient fidelity to compare to others assigned to a control condition. In this investigation into the specific components of the training that may have contributed to the benefits we observed in that study, we needed to further constrain our consideration to those who completed the entirety of the training. Per the model of learning skill development via training that we aimed to test (Hattie & Donoghue, 2016; see Table 2), we anticipated that the initial declarative knowledge of a learning skill is acquired through initial exposure to the topic. Thereafter, additional benefits to metacognitive knowledge of learning skills are conferred when topical coverage is deepened through elaboration of the concepts that explain it; such conceptual knowledge is both consolidated and made observable when students are prompted to demonstrate their conceptual understanding in generative activities. Students’ engagement in activities at the end of modules follow and encourage them to apply their knowledge to the specific course in which they are enrolled. Such practice opportunities should promote transfer, per Hattie and Donoghue (2016), and the quality of their ability to transfer the learning skill is made evident in these activities. Implicit in this design are contingencies that are known to impact learning processes and are formally acknowledged in our guiding theoretical framework for SRL (Ben-Eliyahu & Bernacki, 2015; Winne & Hadwin, 1998). Contingencies are inherent to the design of the topics covered in modules (i.e., metacognitive monitoring topics rely upon knowledge of strategies covered in a different module; behavioral and environmental regulation rely on knowledge related to planning in the metacognitive self-regulation module), making the nature of one’s answer to one topic conditioned on their exposure to other topics. Further, exposure within topics builds from surface to deep to appliable knowledge, as theorized to develop by Hattie and Donoghue (2016). We thus examine those with complete data so that all individuals’ performances can be observed under these assumptions.

Participants

From the overall sample of students enrolled in sections of an anatomy and physiology course taught at a large public university in the Southwestern United States (N = 349), 72% consentedFootnote 1 to participate (n = 251) and 125 were assigned to the experimental condition that was offered the training. Of these, 104 (83%) completed one or more modules and 49 completed all three modules and activities within them. To address the aims of the current study that focuses on variance in the metacognitive knowledge of learning skills demonstrated on activities within training and relationship to performance on the course exams, we constrained our focal sample to these students who were exposed to all three modules and who completed all items (n = 49).

Measures

Digital Science of Learning to Learn Training. Students completed three modules that trained students on enacting cognitive strategies (e.g., self-testing, spacing, self-explanation), improving self-regulated learning (e.g., defining the task, identifying resources, setting goals, monitoring progress), and engaging in behavioral and environmental regulation (e.g., maintaining perspective, understanding implementation intentions, managing one’s learning environment). See Table 4 for a summary of topics and Supplemental Table S1 for an elaborated table of all items from Studies 1 and 2. Figure 1 shows an example of the static digital training design in Study 1 which was developed to introduce SRL skills, deepen understanding of them, and provide opportunities for learners to practice applying them. The embedded assessments that appear in Supplemental Table S1 provide specific observations of degree of mastery of each level of understanding of the skills. In Table 5, we report means and standard deviations reflecting the percent of points earned on the items included in each module, as well as their intercorrelations and relationship to exam scores.

Fig. 1
figure 1

Design of the Science of Learning to Learn Skill Training Content in Digital (Study 1) and Digital Multimedia (Study 2) Editions

Note. Both training versions were delivered entirely in digital format. Study 1 involved text and static images. Study 2 involved text and video content that simultaneously combined visual and auditory modalities

Full size image
Table 5 Correlations between Training Performance per Unit and Exam Scores across studies
Full size table

Exams. The biology course included three unit exams and one cumulative final exam, which were developed by the team of course instructors and deployed across sections. We focused our analyses on the second unit exam, which was the next available assessment administered after the skills training was fully administered, and the cumulative final exam, which served as a delayed measure of academic achievement. The unit two exam contained 50 multiple choice items and 4 open response items; the cumulative final exam was a 100-item multiple choice exam. Each achieved levels of internal consistency that were found to be adequate (αs ≥ 0.75; see Bernacki et al., 2020, 2021).

Procedure

Students consented to participate in the study and completed a biology pretest and demographics questionnaires during the first week of the semester. Skill training modules were administered starting in the second week of the semester and could be completed over the following three weeks (i.e., during weeks 2–4). Students who elected to complete the modules received a completion indicator in the learning management system at the end of each module, and credit was awarded for full completion of each module, summing to a total that amounted to less than 1% of the course grade. For those who chose not to consent and those randomly assigned to the control condition, an alternate set of learning activities were provided that enabled them to earn the same participation credit amount.

Students completed the learning modules at a time of their choosing and at their own pace but were asked to complete an entire module within one session during each week when a module was available. As students progressed through each learning module, they received directions, read about learning skills, answered questions that assessed their learning from the readings, and responded to prompts that encouraged them to identify resources in their courses they could use when applying the skills in future coursework. Students were told that completing the skills training would help them prepare for the second course exam and the three-module set took most students between 90 and 120Footnote 2 minutes to complete. Finally, students took the unit two exam in the eighth week of the semester and the final exam during the last week of the semester (i.e., week 16).

Data analyses

In our data analyses, we focused on the learning module responses that were scored via a rubric created to evaluate students’ performance on declarative and comprehension level items and application activities (see Supplemental Table S1 for items and alignment to topic and level, as well as scoring criteria per item). For instance, students were asked to “List the 3 ways that self-testing helps you learn.” Each student response was coded based on the accuracy of their answer. To receive full credit, the student would have to state that self-testing (1) helps learning by increasing retention (i.e., helps you remember the information for longer periods of time), (2) fills in gaps in knowledge/helps determine what you do not know, and (3) improves future studying/learning. In addition, each learning module item was also coded by skill trained (i.e., cognitive, metacognitive, and behavioral/environmental) and depth of processing (i.e., declarative, conceptual, and application level).

Raters had previously completed multiple years of training in education and/or psychology programs and were supervised by a team of educational psychologists who trained each rater in the use of the instruments used to assess responses to items. Each response was coded by three raters independently. A high degree of average measure intra-class correlation (ICC) reliability was found between the subscales. The average measure ICC was 0.938 with a 95% confidence interval from 0.910 to 0.960, F (48,1392) = 16.104, p < .001. Taken individually, the ICC (2,3) for the skills trained was 0.854, 0.923, and 0.858 (cognitive, metacognitive, and behavioral subscales, respectively). The ICC (2, 3) for the depth of processing variables was 0.956, 0.819, 0.809 for the total declarative, conceptual, and application items, respectively. Next, we assessed the skills trained by the depth of processing (i.e., cognitive training at the declarative level). We found a high degree of ICC (2, 3) reliability of all scales ranging from 1.00 to 0.684, with the exception of the metacognitive skills trained at the conceptual level (0.510; moderately reliable per Shrout & Fleiss, 1979). These measures were submitted to path analyses to test predictive relations by topic, depth of knowledge of learning skills, and their interaction with performance on a subsequent (Unit 2) and delayed, cumulative exam (Final Exam).

Results

Preliminary analyses

To examine the association of performance on each of the activities with performance on exams, we first conducted a series of partial correlational analyses comparing training score to the second and final exam, controlling for pretest score, using students who completed training (n = 49). Table 6 displays partial correlations of learners’ skill mastery scores during training, broken down by the topic and depth of knowledge. Partial correlations revealed that training of cognitive strategies, metacognitive strategies, and behavioral and environmental regulation strategies significantly related to the second exam and final exam performance. Controlling for prior knowledge and ability related to biology learning, the higher students scored in training of these skills, the higher their second exam and final exam scores.

Table 6 Partial Correlations Controlling for Prior Knowledge between Training Scores, Exams (N = 49)
Full size table

Based on the strength of associations observed between performance in training activities and scores on the second and final exams and with the understanding that individual items are likely to predict overlapping variance in exam scores, a series of path analyses were next fit to examine how training specific metacognitive knowledge about learning skills and specific depths of such knowledge predicted exam scores.

Hypothesized model

We were interested in the immediateFootnote 3 predictive effect that SRL skill mastery as of the completion of the intervention could have on exam performance (exam 2) and the lasting predictive effect it might have at the semester (final exam). We expected that the combination of skills trained at various depths of processing would predict academic performance, especially for the application items (see Figs. 2, 3, and 4), which included hypothesized models based on theoretical assumptions based on module alignment to conceptualizations of skill training effects proposed by Hattie and Donoghue (2016) and McDaniel and Einstein (2020; Table 2).

Fig. 2
figure 2

Hypothesized and Final Models Examining Prediction of Performance Based on Students’ Skill Mastery Per Module (Model 1)

Full size image
Fig. 3
figure 3

Hypothesized and Final Models Examining Prediction of Performance Based on Students? Mastery of Module Topics at Depths of Knowledge Across All Modules (Model 2)

Full size image
Fig. 4
figure 4

Hypothesized and Final Model Performance as Predicted by Level of Knowledge of Cognitive, Metacognitive, and Behavioral Regulation Topics (Declarative, Conceptual and Application; Model 3)

Note. Bottom panel shows only statistically significant paths, and standardized parameters estimates. All paths are reported in Table 6

Full size image

Decisions about the inclusion of variables and paths within the models were guided by theoretical considerations, then model fitting procedures accordingly. SRL is known to be associated with academic achievement, and cognitive strategies instruction has been found to relate with achievement, though less strongly than skills related to metacognitive and behavioral/environmental (see Hattie et al., 1996; Dignath et al., 2008; Dignath & Buttner, 2008; Donker et al., 2014). Thus, training of regulation including goal setting, monitoring, control, and self-evaluation should have lasting predictive effects on immediate and delayed exam performance. In addition to the training of these strategies, the depth in which they are processed should influence the lasting impact (i.e., per Hattie & Donoghue, 2016), and metacognitive knowledge demonstrated at conceptual or application level should have stronger predictive relations on examination performances.

Metacognitive knowledge of learning skills: cognitive strategies, self-regulated learning, and behavioral and environmental regulation and predictive effects on biology performance

First, we fit a model where the learning skills trained in each module (i.e., cognitive, metacognitive, behavioral/environmental regulation; Model 1) were hypothesized to predict subsequent academic performance on the next exam (exam 2) and a delayed, summative measure of performance in the course (final exam), after controlling for the prior knowledge and skill demonstrated on an initial biology knowledge pretest. The model with the standardized path coefficients on the statistically significant paths is provided in Fig. 2b. Table 3 summarizes direct effects of variables presented in the final model. The path analysis model showed an adequate model fit, chi-square = 5.341, df = 3, p = .145 and CFI = 0.954 and RMSEA = 0.127 and SRMR = 0.095. Variance explained (R2) for the second exam was 47% and 29% for the final exam. Training of cognitive strategies had a direct predictive effect on exam 2 performance (β = 0.250, p = .043), indicating that students who acquired more of an understanding of cognitive strategies during training had higher performance on exam 2. No statistically significant effect on final exam was observed. Performance on tasks in the second module focused on SRL was not a statistically significant predictor of exam performances in the presence of other predictors (β = 1.08 to 0.221, Table 3). Training of behavioral/environmental effects also had direct predictive effects on exam 2 performance (β = 0.363, p = .003) and final exam performance (β = 0.368, p = .008), indicating that students who acquired more of an understanding of behavioral/environmental strategies during training had higher exam 2 and final exam performance.

Depth of metacognitive knowledge: predictive effects of declarative, conceptual, and applied levels of metacognitive knowledge of learning skills on biology performance

Next, we fit a model that described how one’s demonstrations of metacognitive knowledge of skills at levels of depth of processing (i.e., performance on all items reflecting declarative, conceptual, application level knowledge across all three topics) predicted performance on exam 2 and final exam performance while controlling for prior biology knowledge. The final model with the standardized path coefficients on the significant paths is provided in Fig. 3. Table 3 summarizes direct effects of variables presented in the final model. The path analysis model showed a good model fit, chi-square = 2.520, df = 3, p = .47, CFI = 1.000, RMSEA = 0.000, and SRMR = 0.071. Variance explained (R2) was 48% for the second exam and 35% for the final.

Prior knowledge had a direct predictive effect on exam 2 (β = 0.219, p = .042) and final exam (β = 0.260, p = .027). Training of declarative knowledge effects had a direct predictive effect on exam 2 performance (β = 0.483, p < .001) and final exam performance (β = 0.313, p = .012), indicating that students who acquired more declarative knowledge about strategy skills during training had higher exam 2 and final exam performance. Training of conceptual knowledge had a direct predictive effect on exam 2 performance (β = 0.277, p = .021) and final exam performance (β = 0.420, p = .001), indicating that students who acquired more conceptual knowledge about strategy skills during training had higher exam 2 and final exam performance. Training related to application of knowledge to learning in the biology course had no direct predictive effect on exam 2 or final exam scores.

Type of metacognitive knowledge by depth of processing and prediction of biology performance

To investigate whether the combination of training strategy skills at the various depths of processing affect immediate and lasting exam performance would require a model that included nine predictor variables: Cognitive/Declarative, Cognitive/Conceptual, Cognitive/Application, Metacognitive/Declarative, Metacognitive/Conceptual, Metacognitive/Application, Environmental-Behavioral/Declarative, Environmental-Behavioral/Conceptual, and Environmental-Behavioral/Application. When we initially fit all nine paths from the three modules and three depths of knowledge to exam scores, this produced a model that was just identified, and thus provided no useful data-model fit indices. We used theory and prior analyses to eliminate noncontributing paths. The first paths that were deleted were the application depth of processing since prior analyses indicated that application did not significantly predict exam performance. Model 3 included six predictor variables (Cognitive/Declarative, Cognitive/Conceptual, Metacognitive/Declarative, Metacognitive/Conceptual, Environmental-Behavioral/Declarative, and Environmental-Behavioral/Conceptual) that we hypothesized should explain variance in the two criterion variables (exam 2 and final exam; Fig. 3, right). In order to appraise data-model fit and avoid a saturated model, we freed one path between declarative knowledge of cognitive strategies, which was uncorrelated with final exam performance. This model with six and five predictive paths to the second and final exam fit the data well, X2(1) = 0.140, p = .70, CFI = 1.000, RMSEA = 0.000, and SRMR = 0.007, and warranted interpretation according to our hypotheses. Variance explained (R2) 51% for the second exam and 38% for the final.

Exam 2. For the second course exam, performances on activities training metacognitive skills at the declarative (β = 0.260, p = .029) and conceptual (β = 0.213, p = .044) level were statistically significant predictors of performance, indicating that students who acquired more declarative and conceptual skills about metacognitive strategies during training had higher scores on the second exam. Similarly, for the second exam, training of behavioral and environmental skills at the declarative (β = 0.244, p = .040) and conceptual (β = 0.229, p = .044) depth of processing significantly predicted performance, indicating that the quality of declarative and conceptual knowledge of behavioral and environmental skills learners could demonstrate during training related positively to their performance on the second exam.

Final Exam. For the final exam, two combinations of strategy skills and levels of knowledge predicted performance. As seen on the second exam, training of behavioral/environmental skills at the declarative depth of processing (β = 0.280, p = .020) significantly predicted final exam performance, indicating that as students acquired more behavioral/environmental knowledge at the declarative knowledge, the higher their final exam score. In addition, training of cognitive strategies at the conceptual level (β = 0.234, p = .045) significantly predicted final exam performance, indicating that the more successful training was of cognitive skills at the conceptual level, the higher their final exam performance.

Discussion of study 1

In this first study, we investigated whether performance on components of an SRL training (i.e., cognitive strategies and metacognitive, and behavioral/environmental regulation skills) predicted exam performance in a college STEM course. Previous results (Bernacki et al., 2020, 2021) included the statistically significant effects on exam performance where students assigned to Science of Learning to Learn outperformed those who completed additional study on biology topics. We here augment those findings with additional evidence that students’ knowledge of behavioral and environmental regulation strategies, as well as declarative knowledge of cognitive strategies contribute to these effects on learning.

Whereas Hattie et al. (1996) concluded after a meta-analysis that programs that last for a short amount of time are likely to provide only an immediate impact on performance that would diminish over time, Bernacki et al. (2020) found that 90 to 120 min of the Science of Learning to Learn digital learning skill training was sufficient to achieve enduring effects on course exams administered soon and well after the training. In this study, the path analyses we conducted provide us with an opportunity to observe how the types and depth of metacognitive knowledge of learning skills those in this training conditions explained variance in the biology exam performances these learners achieve. We consider these findings as they reflect assumptions of theorists who propose that such training should promote knowledge of cognitive strategies and skills for the regulation of cognition and behavior (i.e., McDaniel & Einstein, 2020), as well as those who propose that knowledge should be developed to promote an initial level of understanding, which could then be consolidated into a deeper conceptual level of understanding, so that it might be transferred into applied settings (i.e., Hattie & Donoghue, 2016). We observed partial support for each of these assumptions. Training focused on knowledge of cognitive strategies and skills related to regulation of behavior and environment were predictors of success; metacognitive knowledge failed to attain a statistically significant relationship with exam scores. We did not find direct predictive effects of knowledge of metacognitive processes, which runs counter to assumptions in frameworks of SRL (Winne & Hadwin, 1998) and laboratory studies where engagement in metacognitive monitoring and control strategies predict success in biology tasks (Deekens et al., 2018; Binbasaran & Greene, 2015). In addition to the independent utility of individual cognitive strategies or methods of managing one’s environment or establishing cues to prompt behavior, self-regulated learning involves a collection of processes that rely upon one another to impact learning. That is, individuals need to develop their abilities to appraise tasks, the goals others set for them, and the strategies to enact, and then also develop skills for monitoring and for making responsive judgements based on tasks, goals, and known methods that could provide opportunities to adapt to more productive methods. It may be that such skills are harder to train, and also that individual demonstrations of knowledge are insufficient indicators that one can self-regulate. Additional modeling of predictors may need to include interactions and contingencies, much like those described in theoretical frameworks that describe complex, dynamic relations within the SRL process (e.g., Ben-Eliyahu & Bernacki, 2015; Winne & Hadwin, 1998).

When individuals demonstrated knowledge of learning skills at declarative and conceptual levels, they performed better on biology exams. These findings provide support for the importance of developing learners’ knowledge as proposed by Hattie and Donoghue (2016). However, the comparable lack of predictive validity observed for application-level knowledge of learning skills suggests a different implication: the digital skill training might benefit from a revised approach to developing students’ ability to transfer knowledge into future practice (Hattie & Donoghue, 2016). Similar to challenges incurred in the measurement of SRL skills above, improving individuals’ ability to transfer skills learned in one setting into applied settings is a challenge (Ceci & Barnett, 2002), as is the development of measures to probe one’s ability to transfer or apply skills. Some evidence that training improved students’ subsequent engagement with resources designed to promote use of retrieval practice strategies and planning and monitoring was found in the original study (e.g., use of planning resources and tools to monitor knowledge, goal pursuit, and performance; Bernacki et al., 2020), and additional behavioral evidence might be useful to augment demonstrations of metacognitive knowledge at the application level that were gleaned from training in this study.

Findings from the original experimental study confirmed that training was found to benefit the academic achievement of the group who received it (Bernacki et al., 2020). In this study, a closer examination of individuals’ responses during training provides further insight that training of cognitive strategies and regulatory skills to declarative and conceptual levels of metacognitive knowledge of these skills may contribute to the benefits obtained. In addition to the variance in degrees to which trained individuals demonstrated metacognitive knowledge of the skills addressed in the original training, the percent of individuals who completed the training offered to them and the time it took them to do so could be improved; the low participation rates and lengthy completion times observed in the study undermined the broad, practical benefits training could provide to the population it was designed to scaffold. We thus undertook a new round of design effort, again leveraging theories related to instructional design in an effort to produce an efficient and effective skill training program that could improve the learning efficiency obtained by learners. We next document our efforts to redesign the training according to Multimedia Learning Theory (Mayer, 2021) and test whether a video-based multimedia version intended to make the training briefer and more accessible could sustain the predictive relationships with biology exam performance and produce a training design that achieved greater learning efficiency.

Study 2

In the Cognitive Theory of Multimedia Learning (CTML), Mayer (2021) argues that instructional designers can better manage the cognitive load of academic tasks when they make use of multimedia modes of delivery that leverage both audio and visual channels of working memory. Learning from multimedia is common in active learning designs (Lombardi et al., 2021) and has previously been deployed in studies aimed at scaffolding students’ learning processes (Kuhlmann et al., 2023). More specifically, multimedia materials help students by (1) limiting extraneous cognitive processing (i.e., cognitive resources exhausted toward understanding material irrelevant to the learning goal), (2) managing intrinsic cognitive processes (i.e., cognitive resources exhausted toward understanding the complexity of the material), and (3) fostering generative cognitive processes (i.e., cognitive resources exhausted toward actively understanding the relevant material).

Feedback from both students and instructors about the original [Science of Learning to Learn] training included concerns that the training required a substantial amount of reading. While the vignettes describing learning skills and their deployment were rich and descriptive in ways that helped promote understanding, these descriptions took a substantial amount of text to narrate and time to read. When considering this feedback through the lens of CTML and specifically to theory and research on the modality principle (e.g., Moreno, 2006), we reasoned that we could potentially lessen the extraneous cognitive load that text-based narratives could place on learners by producing videos that capture visualizable features of these vignettes in animated form and retain audio narrative of those elements best delivered in written or spoken language. Substantial evidence from multimedia learning researchers indicated that designing instructional videos might be a productive approach to managing cognitive load and improving the efficiency of the learning skill training. Findings from a meta-analysis of 105 studies exploring the effects of learning from instructional videos supported that adding instructional videos as scaffolds in college courses led to strong learning benefits (g = 0.80; Noetel et al., 2021). Taken together, instructional videos are powerful multimedia resources that help students to more effectively and efficiently manage the cognitive processes necessary for learning, and to foster active cognitive processes that lead to long-term and meaningful learning.

The multimedia Science of Learning to Learn skill training program

We completed this video-based redesign and replaced text content with video content, then deployed a video-based multimedia training that retained the activitiesFootnote 4 in which students demonstrated their metacognitive knowledge of learning skills. This included 12 total videos distributed across three modules that trained students on cognitive study strategies (e.g., self-testing, spacing, self-explanation; topics and runtimes appear in Table 4), improving self-regulated learning (e.g., defining the task, identifying resources, setting goals, monitoring progress), and goal achievement (e.g., maintaining perspective, understanding implementation intentions, managing one’s learning environment) in biology. The instructional videos were developed by the research team based on prior work from Bernacki et al., 2020 and designed to align with multimedia design principles (Mayer, 2021) to help students effectively manage verbal and visual information, which in turn increases student’s comprehension and depth of processing from the videos compared to the static materials used in the Study 1 modules (see Fig. 1 for a comparison of materials between Study 1 and Study 2).

Similar to Study 1, the training modules also required students to complete activities intended to promote rehearsal of knowledge and encourage deeper processing of what was learned from the videos. The same questions were used to assess declarative, conceptual, and application-level knowledge of the module topics, and most items remained unedited from Study 1 to Study 2; however, several items were slightly re-written to enhance the clarity of the question being asked (see Table S1, which presents original and edited items from Study 1 and Study 2). For Study 2, two application items were removed from Module 1, these included, “Now list other resources in the course (in your textbook, online resources provided, etc.). Can any features of these resources enable you to use the learning strategies you learned? How?” and “Using your description of the available resources, make a plan for how you might use self-testing, spacing, and self-explanation in order to improve your learning and performance. Describe your plan in the space below.” We removed these items from Module 1 because they are mostly redundant with application concepts assessed in Module 2, which involves application of strategies within the SRL cycle. We also included six new items in Module 3, which included one new declarative question, “What of the following is true about parallel multitasking?”, and five application questions, for example, “Think about the place where you study most often. List the potential distractions that it poses.” The inclusion of these items was undertaken to further strengthen students’ ability to transfer their knowledge of these skills. The new items were included following additional material in the Module 3 instructional videos that discussed parallel multitasking and its potential interference with learning. The declarative item assessed students’ retention of parallel multitasking and the new application items were added to enhance students’ commitment to goal setting and pursuit using mental contrasting with implementation intentions (MCII). Specifically, the additional MCII questions required students to think through how they might regulate their environment to limit distractors and potential multitasking to help them successfully pursue and achieve their learning goals.

Method

Participants

In partnership with an instructor group who taught an introductory biology course for students pursuing biology majors at the same Southwestern US university, we recruited students through announcements in the learning management system to obtain their consentFootnote 5 to participate in the study. We worked with instructors to provide training and alternate activities via Qualtrics, a platform the university licensed for administering surveys and offered completion codes for each module students completed. This design enabled learners who opted to participate in the study to be assigned to experimental and control conditions and those who chose to not to participate could be branched to complete control activities, avoiding random assignment, and receive completion codes to obtain the same credit. From a course with four sections that enrolled 312 total students, 170 consented to participate in the study (55%); 85 were assigned to the experimental condition and 62 of these learners (73%) completed all components of the multimedia [Science of Learning to Learn] training during Unit 1 of the course; these cases comprise the focal sample. A preliminary analysis of the demographics confirmed that the composition of this focal sample did not differ from the composition of the body of students enrolled in the course, and demographic breakdowns mirrored the Study 1 sample.

Learning outcome measures

Unit Exams. The three-unit exams administered after the multimedia training was completed contained 50 multiple choice items and 2 to 5 short answer items each; all exams were developed by the biology course instructor group. A sample exam item was, “Suppose you discover a new species of animal that does not have body segments, but is triploblastic, coelomate, and cephalized. Under the molecular phylogenetic scheme, you would assign it to which phylum?” The exams were scored by the biology course instructors and had high internal reliability (unit exam 1, α = 0.91; unit exam 2, α = 0.89; unit exam 3, α = 0.93).

Final Exam. The cumulative final exam contained 85 multiple choice items and 10 short answer items and was developed by the biology course instructors. A sample exam question was, “Which of the following are the only vertebrates in which blood flows directly from respiratory organs to body tissues without first returning to the heart?” The exams were scored by the biology course instructors and had high internal reliability (α = 0.94).

Procedure

Students consented to participate in the study during the first week of the semester. The skills training was offered in the fourth week of the semester, prior to the first exam in the course. Students completed the learning modules at their own pace but were asked to complete an entire module within one session. For example, students could choose whether they completed each of the three modules in separate, shorter sessions, or all three modules in one long session. As students progressed through each learning module, they received directions, watched instructional videos on the respective learning concepts, answered questions that assessed their learning from the videos, and responded to prompts that encouraged them to identify resources in their courses they could use when adopting the learning concepts in future coursework. After submitting their response, students were shown correct responses and asked to provide a comparative evaluation of their answer to the correct answer. Students were told that completing the skills training would help them prepare for the first course exam and each module took about 30 minutes to complete. Students completed the unit 1 exam during week 6, unit 2 exam during week 11, unit 3 exam during week 15, and the final exam during the last week of the semester (i.e., week 17).

Preliminary data analyses

All Study 2 items were coded and scored using the same process and rubric as in Study 1. Two raters independently scored 20% of the data for each item. The intra-class correlation (ICC) reliability was calculated across all the items, across each module, and across each depth of processing. The average measure ICC for all items was 0.841 with a 95% confidence interval from 0.695 to 0.939, F (13, 715) = 6.290, p < .001. Median completion times for the 3-module sequence amongst completers ranged from 60 to 90 min (i.e., derived from survey submission data), a considerable drop from completion times of 90 to 120 min inferred from completion events reported for the Study 1 version by Bernacki 2020.

Results

In addition to the design goal of diminishing completion times and increasing completion rates, the aims of the multimedia redesign and analyses in this study were to (1) improve scaffolding of knowledge of metacognitive processes involved in self-regulated learning (i.e., module 2) so that they contributed positively to predictions of exam performance, and (2) confirm that the relationships between knowledge of cognitive strategy and behavioral and environmental regulation skills and exam performances were sustained from the original design. To test these relationships, we first examined bivariate correlations between training module performances and exam performances, then fit path models that examined predictive relations between performance in each module and performance on course exams across Study 1 and Study 2 samples (i.e., parallel models with a latent factor indicated by performance on all exams administered after treatment; 3 exams in Study 1 and 4 exams in Study 2). Bivariate correlations appear in Table 5 and Path models appear in Figure 5. Comparitive relations from Study 2 to those observed in Study 1 are in Table 5, in which correlations between metacognitive knowledge of skills demonstrated in Modules 1 on cognitive strategies and 3 on behavioral and environmental regulation correlated positively with performance on exams in both studies. Correlation coefficients include some of low strength (Rs 0.188 to 0.29; Davis, 1971), many in the moderate range (i.e., R = .30 to 0.49) and a few into the high range (e.g., Exam 1 with Module 1 [R = .570] and Module 3 scores [R = .590] in Study 1). The range of relationships between Module 2 performance and exam performance was improved from the “negligible” range of R = − .137 to 0.101 when the format was reading intensive in Study 1 to R = .206 to 0.330 in Study 2 after the multimedia redesign (i.e., low to moderate positive relations, per Davis, 1971). The redesign thus resolved the counterintuitive negative bivariate correlation between a key learning process in the self-regulated learning framework and the academic performance it has been theorized and shown to produce. We next fit parallel models for Study 1 and 2 data where exam performances were predicted by metacognitive knowledge demonstrated in module activities. When included in a path model that examines the contribution of demonstration of metacognitive knowledge of skills, the pattern of effects for cognitive strategies and behavioral and environmental regulation was sustained, though parameters estimates were slightly lower than in Study 1 and only metacognitive knowledge of behavioral and environmental regulation remained a statistically significant predictor (Figure 5, middle panel). Module 2 was a positive predictor of exam performance (β = 0.195) but was not statistically significant.

Fig. 5
figure 5

Note. Grey lines indicate paths that were estimated but that failed to reach statistically significant parameter estimates

Performance during [Science of Learning to Learn] Digital and Multimedia Skill Training and Relationship to Exam Performance.

Full size image

Study 2 discussion

As in Study 1, the bivariate relationships between demonstrations of skill mastery in modules on cognitive strategies and behavior and environmental regulation were positively associated with exam scores, though coefficients were smaller in Study 2. The results of the path model confirm that the behavioral and environmental skills demonstrated during training again predicted exam performance positively and at a statistically significant level (β = 0.252). Performance on items testing cognitive strategy knowledge was positively associated with exam performance (0.174) but when submitted among other predictors failed to predict unique variance in exam scores in the path model.

This second study thus confirmed the value of iterative redesign of digital training meant to develop students’ learning skills, improving completion rates and times, and smoothing the relations between performance during training and performance on course exam. New in this second study was the observation of a positive relationship between students’ demonstration of metacognitive knowledge of SRL skills. This may indicate that learners were more able to comprehend the complex relations among the phases of the SRL model, though additional measurement approaches may be needed to further model how metacognitive knowledge of combinations of skills (e.g., ability to define a task based on learning objectives and knowledge of cognitive learning strategies well-suited to develop that level of understanding) are needed in order to explain significant amounts of variance in exam scores over and above that explained by knowledge of cognitive strategies and behavioral and environmental regulation. This second study does largely corroborate the first and provide additional support for assumptions about developing knowledge of learning skills to sufficient depth (i.e., Hattie & Donoghue, 2016) and on specific topics (i.e., McDaniel & Einstein, 2020).

General discussion

In the pair of studies we conducted to examine how metacognitive knowledge of learning skills explains the benefits of digital learning skills training observed in prior experimental studies, we first examined how the metacognitive knowledge that 49 biology undergraduates demonstrated during digital skill training predicted their performance on subsequent exams in Study 1. Then we redesigned that training to align to principles of the Cognitive Theory of Multimedia Learning (Mayer, 2021) such as the modality principle, and in Study 2 examined the same relationships between training and exam performances of 62 additional learners who completed the multimedia version of the [Science of Learning to Learn] training. We found that predictive relationships between skill mastery and exam performance were sustained for behavioral and environmental regulation strategies, slightly weaker for cognitive strategies, and markedly improved for metacognitive SRL processes.

The original design of the modules included ample expository text and activities to foster and assess declarative and conceptual knowledge of learning skills and promote their application in undergraduate courses, as proposed by Hattie and Donoghue (2016) in their model of learning skill training design. Declarative and metacognitive knowledge of learning skills, particularly related to cognitive strategies and the regulation of one’s behaviors in a learning environment, were observed to predict achievement. However, these data were derived from a small sample of those compelled to complete a reading-intensive training that required most students 90–120 min to complete. Findings from Study 2 mostly replicated these predictive relationships and provide a second instance of corroborating evidence that such levels of metacognitive knowledge of learning skills are sufficient to benefit academic performance. Study 2 analyses also indicated that while initial attempts to develop students’ SRL skills in the original training had a slight negative relationship to exam performances, upon redesign, the associate became positive though still not statistically significant.

Whereas the completion rate in the original training was approximately 40% of those who were provided the opportunity to complete it, completion rates in study 2 increased to above 70%, and the median completion times for those in Study 2 were shorter than in study 1 (i.e., from 90 to 120 min, down to 60–90 min for the majority of learners). Based on these completion rates and times and a largely replicated pattern of effects, with the exception of the additional relationships involving the SRL module. These analyses involved a larger and ostensibly more representative sample of students drawn from a highly diverse student enrollment in an early undergraduate course at a university with minority-serving Institution (MSI), Hispanic-serving Institution (HSI), and Asian American, Native American, and Pacific Islander Serving Institution (AANAPISI) designations that also enrolls a student body where the majority are first generation college students. Such findings, in an understudied context, warrant attention and further research.

Theoretical implications

The study provides insights about the importance of developing the metacognitive knowledge of learning skills that students must possess in order to engage in self-regulated learning in real world settings (Efklides, 2011). Results provide support for recommendations by McDaniel and Einstein (2020) and Hattie and Donoghue (2016) about which types of skills to develop, as well as the recommendation to introduce surface knowledge and consolidate it in preparation for transfer (i.e., Hattie & Donoghue, 2016); declarative and conceptual levels of knowledge of learning skills both explained statistically significant amounts of variance in performance on later academic tasks.

Metacognitive knowledge of learning skills

We scaffolded students’ development of metacognitive knowledge of cognitive strategies including self-testing, spacing, and self-explanation because (1) cognitive strategies have been described as critical to enact in models of self-regulated learning (e.g. Efklides, 2011; Winne & Hadwin, 1998) and (2) these particular strategies are documented as well-suited to improve retrieval of declarative and conceptual knowledge (Dunlosky et al., 2013). When modeled as predictors of biology exam performance, they did predict achievement on the upcoming exam. Teaching students about behavioral regulation strategies including mental contrasting and implementation intentions as well as how to avoid distractions had initial and enduring effects on exam performance. Surprisingly, skill mastery demonstrated during scaffolding of metacognitive strategies did not predict the second or final exam performance. This was the case despite prior research that attributed large effect sizes to trainings’ inclusion of metacognitive processes on subsequent performance (see Dignath et al., 2008; Dignath & Buttner, 2008; Sitzmann & Ely, 2011; Theobald, 2021 for reviews). Because metacognitive processes operate on knowledge and enactment of cognitive strategy use and regulation of not only these strategy selections but one’s responses to environmental conditions, the lessons taught in module 2 on SRL overlap substantially with both the module that focus explicitly on cognitive and behavioral and environmental topics (i.e., a discrete set of cognitive strategies, as well as processes used to regulate environment and behavior in light of one’s goals). It is thus difficult to determine whether the skill mastery demonstrated in this second module alone validly represents SRL skills, or whether more summative measures (i.e., total direct and indirect effects and perhaps effects of interactions between cognitive and metacognitive skills) might need to incorporate metacognitive knowledge about strategies that could be used and regulatory processes that could be enacted. Indeed, theory would suggest these relations are contingent on one another, and SRL might require modeling of interactions between topics to more adequately model how scaffolding complex processes like SRL affects classroom performance (Bernacki, 2018; Greene, 2018; Winne & Hadwin, 1998).

Depth of processing

We next explored whether depth of processing within training would influence examination performance as a way of testing the assumption in the Hattie and Donoghue (2016) model that developing superficial and deeper conceptual knowledge matter when training people to learn. We further expected that learning how to apply knowledge would predict examination performance above and beyond learning declarative or conceptual knowledge about the skills (i.e., transfer in Hattie & Donoghue, 2016 and also Craik & Lockhart, 1982; Marton & Saljo, 1984; Ramsden, 1992). Indeed, learning declarative and conceptual knowledge about the strategies during training increased both immediate (i.e., second exam) and lasting (i.e., final exam) performance. Declarative knowledge was more highly predictive of the second examination in comparison to conceptual knowledge, whereas conceptual knowledge was more highly predictive of the final examination performance. However, application knowledge did not predict the second or final exam. These findings generally confirm Hattie and Donoghue’s (2016) assumption that there is value in introducing learning skills at a surface level, then deepening and consolidating knowledge of them. However, additional effort will be required to improve measurement of these skill areas including dimensions of transfer (Ceci & Barnett, 2002), and additional research will be required to investigate how combinations and degrees of knowledge can be trained to promote transfer from within training to future learning tasks like those in undergraduate STEM coursework.

Metacognitive knowledge by depth of processing

In Study 1 and per Hattie and Donoghue (2016), we aimed to understand how scaffolding learners’ more surface-level declarative knowledge of learning skills, their deeper conceptual understanding, and their facility at applying those skills to hypothetical examples during training might relate to classroom achievement. To do this, we evaluated the combination of strategy skills trained at each specific depth of processing. We found that students who acquired more declarative and conceptual skills about metacognitive and behavioral/environmental regulation skills during training acquired higher scores on the second exam. Assessing the levels of understanding separately from metacognitive and behavioral/environmental regulation skills provided a better insight into the benefits of specific features of training necessary for predicting second exam performance. That is, learning of metacognitive strategies did increase the second exam performance but only through mastery demonstrated at declarative and conceptual levels. For the final exam, training of behavioral/environmental skills at the declarative level of knowledge and cognitive strategies at the conceptual level increased final exam performance. These findings provide further support for Hattie and Donoghue’s (2016) surface level and consolidation design paradigm, yet also indicate that lesser levels of understanding may be sufficient to enact and benefit from certain learning skills.

Limitations and future research

A conceptual limitation of our investigation is that both Hattie and Donoghue (2016) and McDaniel and Einstein (2020) propose in their models that development of metacognitive knowledge of learning skills be accompanied by efforts to support motivations and beliefs about learning. We focused only on the skill dimensions, and future experiments will be needed to consider how skills interact with the “will and thrill” components of these models.

Methodological limitations also remain after two studies and will need to be addressed in future research. First, each of the samples drawn were from intervention groups at the same university and in similar biology courses. Additional studies that examine deployments of the [Science of Learning to Learn] and other learning skill trainings in different contexts are warranted to examine questions about the types and depth of knowledge about learning skills that should be targeted in skill training programs to scaffold STEM learning and achievement.

Second, and as noted in the Study 1 discussion, whereas declarative and conceptual knowledge of learning skills could be assessed reliably, we encountered difficulty in the measurement of students’ ability to apply the learning skills that were trained. It would be beneficial for future researchers to consider alternative ways to measure students’ ability to transfer lessons learned during training and enact them in practice. Measures of application-level knowledge did not predict achievement and may be misaligned to the ways students actually apply such knowledge in courses like those in our studies and in other academic tasks. Whereas redesign improved the relationship between demonstrations of skill mastery on SRL topics, additional analyses to better understand metacognitive processes that include the planning and monitoring of strategies like those taught in the other modules would improve understanding of ways to productively scaffold skill development.

Third, multimedia redesign improved completion rates and times, but many students were excluded from analyses because they did not attempt all modules and questions within them, which limited our statistical power to detect statistically significant relationships between metacognitive knowledge of learning skills and academic performance. Additional analyses with larger samples and samples with higher percentages of training condition students who completed the training can power more complex analyses and reduce the threats of selection bias when evaluations of inter-dependent lessons in such trainings necessarily rely on full completion of training programs. Finally, the topical coverage of the [Science of Learning to Learn] training aligns to the cognitive strategies and regulatory skills likely to benefit learners in STEM courses, but additional topics may need to be included to fully represent the assumptions of digital skill training models. For example, the beliefs and aspects of goal commitment hypothesized to contribute to skillful learning by McDaniel and Einstein (2020) are not scaffolded in the [Science of Learning to Learn] training, other than what is implicit within goal setting and commitment through SRL and implementation intentions. Understanding student beliefs and motivations can help further clarify the ways that motivations might moderate the efficacy of the intervention on scaffolding skill development and improving course performance.

Conclusions and educational implications

This pair of studies provided evidence that learners who possessed declarative and conceptual knowledge of cognitive strategies and methods of regulating their behaviors and environment outperformed those who lack knowledge of these learning skills on exams in their early undergraduate science courses. Knowing the types of learning skills and depth of metacognitive knowledge of them that predict achievement can help instructors and support staff develop the kinds of scaffolding that incoming students may need in order to overcome their beliefs that they will struggle in their STEM coursework. Such scaffolding could further ameliorate the concerns students have about engaging in active learning (Shekhar et al., 2020), and increase the consistency of the benefits that active learning has been observed to provide to learners (Theobald et al., 2020). Study findings provide additional, predictive evidence about the reasons why brief digital training might benefit students, and these findings can be used to further refine digital training designs. Reliance on skill training and multimedia learning theory were sufficient to inform the development of an effective training that was made more efficient though a video-based redesign, yet additional design challenges remain. Educators who wish to scaffold learners in STEM contexts can continue to refine the learning skill trainings that can develop learners’ metacognitive knowledge of learning skills. Additional efforts to incorporate tools like learning analytics reports and dashboards might be warranted to inform instructors about the students who may benefit from such targeted intervention via training, or how additional forms of scaffolding learners might be deployed or faded in timely fashion to develop students’ ability to self-regulate their learning.