Retrieval Practice and Study Planning in MOOCs: Exploring Classroom-Based Self-regulated Learning Strategies at Scale

  • Dan Davis
  • Guanliang Chen
  • Tim van der Zee
  • Claudia Hauff
  • Geert-Jan Houben
Conference paper

DOI: 10.1007/978-3-319-45153-4_5

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9891)
Cite this paper as:
Davis D., Chen G., van der Zee T., Hauff C., Houben GJ. (2016) Retrieval Practice and Study Planning in MOOCs: Exploring Classroom-Based Self-regulated Learning Strategies at Scale. In: Verbert K., Sharples M., Klobučar T. (eds) Adaptive and Adaptable Learning. EC-TEL 2016. Lecture Notes in Computer Science, vol 9891. Springer, Cham


Massive Open Online Courses (MOOCs) are successful in delivering educational resources to the masses, however, the current retention rates—well below 10 %—indicate that they fall short in helping their audience become effective MOOC learners. In this paper, we report two MOOC studies we conducted in order to test the effectiveness of pedagogical strategies found to be beneficial in the traditional classroom setting: retrieval practice (i.e. strengthening course knowledge through actively recalling information) and study planning (elaborating on weekly study plans). In contrast to the classroom-based results, we do not confirm our hypothesis, that small changes to the standard MOOC design can teach MOOC learners valuable self-regulated learning strategies.


MOOC Self-regulated learning 

1 Introduction

Open, informal learning environments, such as MOOCs, provide learners with an unprecedented level of autonomy in the learning process. While certainly empowering in one sense, this new paradigm also places the onus on the individual learner to both conceive and follow a learning process on their own.

Given that one target audience of MOOCs is disadvantaged people without experience in or access to formal educational settings [6], one cannot assume that all learners have the skill set to independently direct their own learning process. Moreover, current MOOCs are frequently designed without any of these considerations; they simply deliver the content to the learner without concern for fostering effective learning strategies.

The analysis of learning strategies (their benefits and effectiveness) has been a long-standing focus of classroom-based learning research. Some of the learning strategies most popular with learners, such as note-taking and rereading, have been found to be outperformed by what is known as retrieval practice (or the testing effect) [2, 8]: a study strategy which focuses on the active recalling of information from memory (as opposed to rereading a passage or rewatching a video), which has a substantial, positive effect on future recall attempts [16].

A second study strategy found to be particularly effective in classroom-based learning is that of study planning. Research in this area has found students who spend time thinking about, explicitly stating, and reflecting on their goals on a daily, weekly, or even yearly level show increases in both engagement and academic performance [13, 18, 21].

Both retrieval practice and study planning are aspects of Self-Regulated Learning (SRL). SRL is defined as a learner’s proactive engagement with the learning process through various personal management strategies in order to control & monitor cognitive and behavioral processes towards a learning outcome [20, 22]. By making learners more adept at regulating their own learning process, MOOCs can also act as resources for not only domain-specific knowledge, but also as a tool for teaching people how to learn.

In this paper we investigate to what extent SRL strategies beneficial in the classroom can be successfully transferred to the MOOC setting. We implemented retrieval practice and study planning prompts aimed at promoting SRL in two edX MOOCs. Our work is guided by the following Research Questions:  

Do learners engage with SRL interventions as much as they do with course content (videos, quizzes, etc.)?


Does inserting retrieval cues after MOOC lecture videos increase test performance?


Does providing a scaffolded means of study planning promote learner engagement and self-regulation?

Based on our experimental results, we conclude that such interventions are not beneficial enough to increase MOOC learners’ success (in terms of grades earned) or engagement (in terms of activity levels observed in the course environment).

2 Related Work

In this section, we separately explore previous work in classroom-based and MOOC-based SRL interventions.

2.1 Classroom-Based SRL

Retrieval Practice. A study in [1] focused on metacognition, or the ability to regulate one’s own cognition, such as learning processes. The study found that providing metacognitive prompts to the sample of undergraduate students resulted in improved learning outcomes. Similar to our work, [1] also observed high levels of non-compliance with the metacognitive prompts/interventions (instructional events intended to improve student learning performance [1]), thus raising the challenge of motivating students to engage with such activities.

On the topic of the “testing effect”, in the context of video watching, Johnson and Mayer [8] found that, compared to only re-watching, students remember more about the content when they are asked to respond to questions about the video’s content after viewing it. This lab study with 282 participants found high support for the testing effect, with subjects exposed to this condition showing higher rates of both learning transfer and retention of knowledge a week after the lesson [8].

Roediger and Butler [16] offer a review of the existing literature on retrieval practice and outline five key findings: (i) retrieval practice increases long-term learning compared to passive re-visiting, (ii) repeated retrieval practice is more effective than a single instance, (iii) retrieval practice with feedback increases the testing effect, (iv) some lag between study and testing is required for retrieval practice to work, and (v) retrieval practice not only benefits a specific response/finite body of knowledge; it allows for transfer of knowledge to different contexts [16].

Research has also been done to determine how to best implement retrieval practice; with a study including fifty middle-school students, Davey and McBride [5] found that, compared to rereading, actively retrieving and elaborating on knowledge from memory leads to better long-term learning.

The most notable difference between these works and our research is the learner population. MOOCs have an unprecedented level of heterogeneity, with learners coming from all corners of the globe with profoundly diverse backgrounds.

Study Planning. The goal setting intervention by Schippers et al. [18] was introduced to an entire class of students in an undergraduate business school. This intervention, which required four hours of student engagement at the beginning of their program, had a positive impact across a prolonged period of time. The reported results include a 98 % reduction in the gender and a 38 % reduction in the ethnicity gap after one year (compared to the previous year’s cohort of students).

Palmer and Wehmeyer [14] implemented the “Self-Determined Learning Model of Instruction” [21] to a sample of students ages six through ten and found that even students of this age range were able to both successfully learn and effectively practice self-determined goal setting strategies.

In the context of a high school social studies class Zimmerman et al. [23] found that students perform better (in this case measured by final grade) when they are able to set their own goals and benchmarks, rather than having to adapt to those imposed upon them by parents or teachers. The findings of [23] suggest that setting one’s own goals works in tandem with increases in academic efficacy, thus improving performance.

Through a “self-monitoring” course intervention, Sagotsky et al. [17] found that (elementary-middle school) students who actively monitored and reflected on their learning progress and behavior on a regular basis exhibited higher academic achievement (grades) and “more appropriate study behavior” (such as being on-task and engaged) [17]. This self-monitoring group also performed significantly better in both measures (achievement and behavior) than the control group which was only prompted to set study goals and not to reflect on them.

In a study including 27 undergraduate students preparing for an exam, Mahoney et al. [11] exposed students to one of three interventions: (i) continuous self-monitoring, (ii) intermittent self-monitoring, and (iii) receiving instructor feedback. The results showed that students who engaged in self-monitoring (especially continuous) exhibited higher levels of engagement and better scores on quantitative problems in the exam.

As mentioned with the literature on retrieval practice, the MOOC learner population is infinitely variant. So while the above findings on study planning may hold true in the classroom or laboratory (with required attendance and homogeneous samples), there is a chance that the findings do not translate directly to MOOCs.

2.2 Learning Strategy Research in MOOCs

MOOC-based research is beginning to recognize & address the current instructional design shortcomings of MOOCs. Nawrot and Doucet [12] suggest that MOOCs are in need of increased learner support, based on survey results which queried MOOC learners’ experiences. They found time management in particular to be a major hindrance to learners. They propose to augment existing MOOC designs with time management support/guidance in order to curtail the dismal retention rates that MOOCs so frequently see.

By gathering information about MOOC learners’ study habits through a post-course survey, Hood et al. [7] observed that learners coming from different professional backgrounds demonstrate different levels of SRL strategies—with higher-educated learners reporting higher levels of SRL. Our research aims to address this discrepancy in SRL skills and provide a scaffolded method to develop SRL strategies in our MOOC learners from all backgrounds and contexts. Instead of self-reported engagement data, however, our study views SRL and engagement in terms of log traces generated by the learning platform.

In one computer science MOOC, Kizilcec et al. [10] tested the effectiveness of sending out different types of encouraging emails to students and found them to be ineffective in increasing learner engagement with the course discussion forum. In a pre-course intervention Kizilcec et al. [9], introduced a subset of MOOC learners to a SRL support module in which seven SRL strategies were explained. Included as part of the pre-course survey, the study found this intervention to elicit no significant differences in course engagement or persistence (in terms of the number of lectures watched). As a consequence the authors proposed that such recommendations/interventions should be more robustly implemented into the structure of the MOOC. In our research we expand upon this and provide a venue (study planning advice & text input) for students to actively plan their learning strategies for the week.

In both [3, 19] increasing learners’ engagement with MOOC discussion forum was targeted in order to increase the overall retention rate. Coetzee et al. [3] introduced a voting/reputation system which allowed learners to vote on which posts are more or less valuable. Their main findings were that (i) the reputation system increases the response time and number of responses in the forum and (ii) forum activity is positively correlated with higher retention and final grades as compared to the learners who were exposed to the standard discussion forum design. The experiment by Tomkin and Charlevoix [19] aimed to discover the effect of having the course team (instructor & teaching assistants) engage with the forum. For one condition, the course team did not engage at all, and for the other, they were highly engaged, providing feedback to questions, comments, and compiling weekly summaries of the key discussion points. In contrast to the formulated hypotheses, the course team intervention resulted in no significant impact on completion rates, learner engagement, or course satisfaction.

To conclude, existing MOOC research has, so far, focused largely on observing the learning strategies employed by learners (without actively intervening), and a small-but-growing number of studies have tried to actively intervene and encourage SRL. Our research aims to expand on this existing work by designing and testing SRL interventions in MOOCs based on a theoretical foundation of teaching strategies found to be effective in traditional classroom settings.

3 Approach

In this section, we first describe the research hypotheses we developed based on RQ1, RQ2, and RQ3. Since our interventions were designed for two specific MOOCs, we first introduce them before outlining our implementation of the two interventions (retrieval practice and study planning).

3.1 Research Hypotheses

Regarding RQ1, and taking into consideration prior attempts at learning strategy research in MOOCs, we draw the following hypothesis:


Learners do not engage with the SRL interventions as much as they engage with the main course content, such as videos and quizzes [1, 9].


Based on prior work in the area of retrieval practice we developed the following hypotheses related to RQ2:


Actively retrieving/producing knowledge leads to better exam scores than passive rereading [4, 5, 16].


With respect to RQ3, we draw the following two hypotheses from the existing literature on study planning:


Encouraging learners to actively plan and reflect on their study habits will increase their engagement with the course [11, 17].


Learners who actually plan and reflect on their course of study will exhibit higher engagement and achieve higher grades [17, 23].


3.2 MOOCs Studied

We implemented our interventions in two edX1 MOOCs (Table 1), which were developed at the Delft University of Technology and ran in 2015. Although the choice of MOOCs was opportunistic, we consider them to be representative of a wide variety of MOOCs offered on platforms such as edX.

We deployed the retrieval practice intervention in Functional Programming, a 13-week MOOC which introduces basic functional programming language constructs. Nearly 28,000 learners enrolled, and 5 % eventually passed the course. The effectiveness of study planning was evaluated in Industrial Biotechnology, a 7-week MOOC that introduced learners to basic biotechnology concepts. Enrollment into this MOOC was lower, while the pass rate was similar to Functional Programming.
Table 1.

Overview of the two courses included in the present study.



Pass rate

#Learners in cohorts


Functional Programming


5.05 %



Industrial Biotechnology


4.08 %



On the edX platform, A/B testing (i.e. providing a different view of a MOOC to a randomly chosen subset of learners) is readily available. Upon enrolling, learners are randomly assigned into one of the provided Cohorts, which is either the control group (no intervention) or one of the experimental groups (an intervention of some form). One practical limitation of edX’s Cohorts feature is that learners cannot be assigned retroactively to a group—any learner who registered to a MOOC before the Cohorts feature was activated will not be assigned to any group. This aspect is reflected in Table 1: 9, 836 (or 35 %) of the Functional Programming learners and 1, 963 (or 18 %) of the Industrial Biotechnology learners are assigned to either the control or one of the experimental groups. Although in our analysis we could have considered all non-assigned learners as part of the control group (as those learners were not exposed to any intervention), we opted not to do so to keep the groups balanced.

3.3 Retrieval Practice

In the original course design (i.e. no intervention) of Functional Programming, each week’s video lecture is broken up into two or three segments. And although the students must navigate themselves from one segment to the next, there are no other learning materials or activities between. In order to activate the learning process, we inserted retrieval practice cues designed to make learners stop and process the information presented in the video lecture.

In each course week, we inserted a retrieval cue directly after the final lecture video, thus prompting the learners to stop and think before moving on to the weekly quiz. The only exception to this design was one particular course week2 where we inserted retrieval practice cues after each of the three segments of the weekly lecture, as in the previous edition of the course learners had perceived that week’s material as the most challenging.

This experiment had three groups (or conditions): (1) the control group without an intervention, (2) the “cued” group, and (3) the “given” group. The “cued” group was shown the following prompt along with a blank text input box:

Please respond in 3–5 sentences to the following question: “In your opinion, what are the most important points from the previous video?”

Note that these responses were not seen, graded, or given any feedback from the instructor—serving strictly as an activity for learners to exercise and improve memory recall. The “given” group, instead of being asked to create a summary themselves, was provided with a 3–5 sentence summary of the video as generated by one of the authors highly familiar with the functional programming paradigm. We included the “given” group in our study to determine the effect of actively retrieving information from memory versus being provided a summarizing text for passive reading.

3.4 Study Planning

The study conducted within Industrial Biotechnology introduced learners to SRL techniques by inserting a study planning module in each week’s Introduction section. In order to stimulate learners to actively think about and plan their course of study and learning objectives for the week, we inserted the following question&examples followed by a blank text input box:

In the space below, please describe, in detail, your study plan and desired learning objectives for the week regarding your progress:

  • I plan to watch all of the lecture videos.

  • I will write down questions I have about the videos or assignments and discuss them in the forum.

The initial prompts were bookended by a reflection prompt at the end of each week in which learners were instructed to reflect on their planning and execution:

How closely did you follow your study plan from the beginning of the week? Did you successfully meet all of your learning objectives? In the space below, explain how you can improve upon your study habits in the following weeks in order to meet your goals.

4 Findings

In this section, we describe our findings in line with our research hypotheses described in Sect. 3.1. Across both courses we find support for H1 (learners engage less with interventions than course content items). Of the 3,262 learners in the “cued” condition in Functional Programming, 2,166 (66.4 %) logged at least one video-watching event in the course. Among these same learners only 719 (22 %) clicked on any of the retrieval practice interventions. Of the 998 learners exposed to the study planning modules in Industrial Biotechnology, 759 (76.1 %) logged at least one video-watching event. Among these same learners, only 147 (14.7 %) clicked on any of the study planning modules.

4.1 Retrieval Practice

We first tested whether the learners of the cued, given, and control groups score differently in the weekly quizzes. To this end we performed a MANOVA test with the highly engaged learners (characterized by having spent more than the group’s mean time watching videos in Week 1 which is \({\approx }22\) min) in each of the three conditions as a fixed factor and the grades on the weekly quizzes as a dependent variable. The MANOVA test followed by the post hoc Games-Howell (equal variances not assumed) test yielded no significant differences between each group’s weekly quiz grade.
Fig. 1.

KDE plot showing the distribution of weekly quiz grades across the groups of highly engaged learners. All lines were fit using a Gaussian kernel function. None of the differences between groups are statistically significant at the \(\alpha =0.01\) level.

In the previous analysis all highly engaged students from each condition were included. However, as many students did not engage with the intervention, this can give a distorted view of its effects. Therefore, we next isolated those learners who actively engaged (characterized by viewing the intervention for at least 10 s) with an intervention prompt at least once.

Using these new group definitions, we still observe no statistically significant differences between the groups as a result of a MANOVA (to test the difference between weekly quiz scores), and a one-way ANOVA (to test the difference between course final scores). In turn, we fail to reject the null hypothesis of H2 in terms of both weekly quizzes and final course grades. Figures 1 and 2 illustrate these null findings via Kernel Density Estimation (KDE) plots.
Fig. 2.

KDE plot showing the distribution of final course grades across the groups of highly engaged learners. All lines were fit using a cosine kernel function. None of the differences between groups are statistically significant at the \(\alpha =0.01\) level.

4.2 Study Planning

We analyzed the differences between the two experimental groups in Industrial Biotechnology—those who were exposed to a study planning module intervention (condition) and those who were not (control)—and found no significant differences in their final grades, course persistence, and many engagement metrics, thus leading us to fail to reject the null hypothesis of H3. However, in support of H4 at the 99 % confidence level, we do find the following statistically significant results when narrowing the sample to compare highly engaged learners (characterized by having spent more time watching Week 1 videos than the average learner, \({\approx }\)33 min) in the control group and the learners in the condition group who engaged with a study planning module at least once (referred to as “Study Planners”).

Comparing Engagement Between Groups. To determine whether there is a significant difference in the engagement levels between the highly engaged learners in the control group (N = 329) and the conditioned group (those who clicked on at least one study planning module, N = 146). In Table 2 we employ two Mann-Whitney U tests, as the data is not normally distributed, showing that the study planners have a higher session count than the highly engaged learns (U = 20,070, p = 0.003), as well as a higher total amount of time spent in the course in hours (U = 19,983, p = 0.002).

The results suggest that students who engaged with the study planning intervention are significantly more engaged with other aspects of the course as well. An alternative interpretation, however, could be that students who are highly engaged with the course also tend to engage more with the planning intervention.
Table 2.

Results of a Mann-Whitney U test comparing the two conditions (study planners vs. highly engaged learners in the control group) in terms of two learner engagement metrics: total amount of logged sessions in the course and total amount of time spent in the course in hours.


Study planning




Session count



Time in course (hours)



Comparing Course Persistence Between Groups. With regard to H4 (engagement-related) we operationalize learners’ persistence as the corresponding week of a learner’s latest quiz submitted or video watched (slightly different from that used in [9], where persistence measured the overall amount of course materials accessed). Whereas the analyses in Table 2 included activity throughout the entire course, irrespective of the course week, one symptom of SRL is a learners’ persistence through the course, or how many weeks the learner makes it through. We define a learner’s “Final Week Reached” as the latest week in the course in which the learner either watched a video or submitted a quiz question. We ran an ANOVA to compare how far into the course learners in each group reached.

The ANOVA yielded significant results, F(2,734) = 21.66, p < 0.001. Post hoc Games-Howell tests show that the group who engaged with the study planning module (N = 146, M = 4.60) persisted deeper into the course than highly engaged learners in the control group (N = 329, M = 3.84, p < 0.001) and highly engaged learners who were exposed to, but did not engage with, the study planning module (N = 262, M = 3.28, p < 0.001). Figure 3 presents a kernel density estimation plot in order to visualize the differences between groups.
Fig. 3.

KDE plot showing the course persistence of the three groups of learners. All lines were fit using a Gaussian kernel function.

Comparing Final Grades Between Groups. To answer the second aspect of H4 (grade-related), we conducted an ANOVA to determine whether there was a significant difference in final grade between the three groups of highly engaged learners listed above. The univariate test was significant, F(2,735) = 17.147, p < 0.001. The results are presented in Table 3.
Table 3.

Results of the ANOVA comparing final course grades among learners who engaged with the study planning module (Mean = 46.42) against those of the two other groups. A final score of 100 would indicate a perfect score.



Mean difference

Study planners










The follow-up Games-Howell test revealed that learners who engaged with the study planning module (M = 46.42) earned higher grades than the highly engaged learners in the control group (M = 36.44, p = 0.003) and highly engaged learners who did not engage with the intervention (Non-Planners, M = 29.10, p < 0.001). These results are visualized in Fig. 4 and illustrate how Study Planners’ final grades are higher than the others’.
Fig. 4.

KDE plot showing the distribution of final grades earned between the three groups of highly engaged learners. All lines were fit using a Gaussian kernel function.

Study Planners Engagement Correlations. Focusing specifically on the learners who interacted with the study planning module intervention, we analyze the relationship between the extent to which they engaged with the intervention and their behavior elsewhere in the course. To do so, we computed a Pearson correlation coefficient to assess the relationship between a learner’s average planning module response length (in text characters) and engagement-related variables such as: (i) total amount of time spent in the course, (ii) number of unique sessions logged, (iii) average length (in seconds) of learners’ sessions, (iv) total amount of time spent watching videos, and (v) number of discussion forum sessions. The results are shown in Table 4. Two example correlations (unique sessions logged and time watching videos) are illustrated in the scatter plots in Fig. 5 to show the slope and overall fit of the regression line. Consistent with the Pearson correlation coefficients of 0.268 and 0.346, the plots indicate positive, small-to-moderate correlations.
Table 4.

Pearson correlation coefficient test results reporting the relationship between learners’ average planning module response length and five course engagement metrics. All correlations shown are significant at the \(\alpha =0.01\) level.


Pearson correlation


Total time in course



Session count



Avg. session length



Time spent watching videos



Forum sessions



Fig. 5.

Scatterplots illustrating two example results of the five Pearson correlation coefficient tests run in order to characterize the relationship between the amount of text characters entered in the study planning module and two key course engagement metrics: session count (left) and time spent watching video lectures (right).

The results suggest that increases in the amount of text learners write in the study planning module are correlated with small-to-moderate increases in a number of key course engagement metrics.

Overall, we find that mere exposure to study planning and retrieval practice interventions is not sufficient to significantly increase learner engagement or final grades. Only when narrowing the samples to learners who actually engaged with the study planning intervention do we see significant results. However, the same does not apply for learners who engaged with the retrieval practice cues, where even learners who engaged with the retrieval cues show no significant difference in any measure of performance.

5 Conclusions

In this work, we empirically investigated two types of instructional interventions found to be effective in traditional educational environments (study planning and retrieval practice) in the MOOC setting. In contrast to our hypotheses, we found both to be largely ineffective in boosting learner success and engagement. Only when accounting specifically for learners who engaged with the intervention directly do we observe significant increases in final grades and engagement in one of the two MOOCs studied. However, given that between 14 %–22 % of the learners meet this criteria in our studies, we too note the “non-compliance” described in [1] as a problem.

Another point of improvement for future studies is that of the frequency and chronology of the interventions. For example, future testing of retrieval practice should be made more continuous and incorporate more lag time [4, 16].

Taking both the existing literature and the present study into account, we will design future theory-based (this research focuses on Zimmerman’s [22] model, but future work should also investigate the effectiveness of other models, such as that of Pintrich [15]) interventions to be much more appealing and prominent in the context of the course—be it visually or perhaps making them compulsory.


“Week 7: Functional Parsers and Monads”.


Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Dan Davis
    • 1
  • Guanliang Chen
    • 1
  • Tim van der Zee
    • 2
  • Claudia Hauff
    • 1
  • Geert-Jan Houben
    • 1
  1. 1.Web Information SystemsDelft University of TechnologyDelftThe Netherlands
  2. 2.Graduate School of Teaching (ICLON)Leiden UniversityLeidenThe Netherlands

Personalised recommendations