Does Taking a MOOC as a Complement for Remedial Courses Have an Effect on My Learning Outcomes? A Pilot Study on Calculus

Massively Open Online Courses (MOOCs) present new opportunities for facilitating teaching and learning [14]. MOOCs allow flexible learning anytime and anywhere, diversifying the variety of tasks that can be included in any course structure [15]. Lately, several case studies have documented different ways in which elite universities have integrated these courses into their curricula, broadening their teaching and learning strategies by implementing blended or hybrid learning approaches [5, 8, 17].

Two trends were observed in these case studies. The first trend (1) is using MOOCs as a complement of traditional teaching. For example, a study shows how Stanford University integrated MOOCs in a traditional course by asking students to watch video lectures, participate in discussion forums, complete quizzes and program assignments in an online platform [13]. 26 students had to complement their learning with information about topics not addressed in the MOOC. The results show that students’ attendance increased by 20 % and their engagement with the course content increased by 40 % [3]. Another example along these lines was developed by the University of Washington, which introduced blended learning in a traditional biology class. They were able to reduce its fail rate from 17 % to 4 %. Furthermore, the approval rates of the course increased from 14 % to 24 % since the initiative [2].

On the other hand, (2) MOOCs are used as remedial courses. Examples of these are the zero level courses developed by some universities. Universidad Carlos III de Madrid [11] analyzed the effect of a zero level course. In this experience, students took a diagnostic and a final exam, and the results indicated that students increased by 21 % the score in the final exam after the course. Regardless of other case studies in North America and Europe [1, 7], the effect of the MOOCs deserves further exploration in other countries to enrich current literature.

In order to contribute to the understanding of the MOOC-based models that use MOOCs to complement or substitute traditional remedial courses, this paper reports on the findings of a pilot study at the School of Engineering in Pontificia Universidad Católica de Chile (UC-Engineering). Specifically, we investigated the effects of 4 MOOCs on calculus for freshmen. From now on we call these MOOCs “service MOOCs” according the framework proposed in [17]. That is, MOOCs that students take voluntarily (partially or completely), and as a complement to the curriculum or a traditional course but no institutional recognition is given for completing this MOOC. In Sect. 2, we describe the context in which this study was carried out, as well as the research questions addressed. Also in this section, we describe the participants of the study, the data gathering techniques and the procedures we used for the analysis. In Sects. 3 and 4, respectively, we report the main results obtained and the lessons learned from the study as well as its limitations. Finally, in Sect. 5, we present the main conclusions, and future avenues. Altogether, this work provides a better understanding of the effects of this type of MOOC-based initiatives in terms of students’ adoption and learning outcomes.

This section reports on the results obtained from the analysis to address the two research questions. Subsect. 3.1 presents the results about the students’ adoption of the MOOC initiative, and Subsect. 3.2 about the effects on students’ learning outcomes.

3.1 Students’ Adoption of the MOOC Initiative

R1.1. Up to 16 % of the students were active in the MOOCs. Active students used the MOOCs more before the diagnosis exam than during the required courses. Between 5 % and 16 % were active in the MOOCs. As shown in Table 3, M2 is the course that concentrated most of the activity, followed by M1, M4, and M3. M2 is a MOOC about trigonometry, a content that is no longer evaluated in the college admission test since 2014².

Table 3.

Active MOOC students vs. Non-active

Course	Before diagnostic Phase, BDP		During remedial phase, DRP
	Active	Non-active	Active	Non-active
M1	14 % (N = 84)	86 % (N = 505)	7 % (N = 42)	93 % (N = 547)
M2	16 % (N = 97)	84 % (N = 492)	13 % (N = 79)	87 % (N = 510)
M3	8 % (N = 48)	92 % (N = 541)	5 % (N = 29)	95 % (N = 560)
M4	12 % (N = 73)	88 % (N = 516)	10 % (N = 56)	90 % (N = 533)

Figure 1 shows the activity of the students during the pilot study. The average number of interactions per day per MOOC during the three days before the diagnostic exam (from January 11^th, which is when the students found out they had been accepted, to January 13^th) is 591 (with a total of 7.095 learner actions traced), whereas there are only 61 daily interactions per MOOC during the face-to-face required courses (with a total of 3.701movements registered from January 14^th through January 29^th). Specifically, students interacted more with each MOOC during their participation in the required course. M1 and M2 were the MOOCs most used.

Open image in new window

R1.2. Students used the courses for exercising. Table 4 shows that the exercise sections registered more interactions than the video sections. This result is observed in all courses and in both phases. By both phases, we mean before the diagnostic exam and during the remedial courses.

Table 4.

Interactions captured in each MOOC section Before the Diagnostic Exam Phase (BDE) and During Remedial Phase (DRP) and proportions of interactions per MOOC per phase

	BDE		DRP
	Videos-lectures	Exercises	Video-lectures	Exercises
M1	503 (39 %)	793 (61 %)	194 (38 %)	316 (62 %)
M2	439 (22 %)	1.516 (78 %)	240 (28 %)	626 (72 %)
M3	37 (10 %)	341 (90 %)	44 (20 %)	181 (80 %)
M4	248 (23 %)	853 (77 %)	40 (16 %)	205 (84 %)
Total	1.227 (26 %)	3.503 (74 %)	580 (28 %)	1.328 (72 %)

The lessons reported in this section were obtained from reflecting on the pilot study results from both the student’s adoption and the students’ learning outcomes. In an effort to highlight those aspects of the study that could be applied to other contexts, we report on the limitations and analyze the issues that emerge from this work and would deserve further work.

First, students are not yet enough prepared to adopt MOOCs if proposed as a complement to traditional courses and if they are not mandatory. The results of our study show that between 8 % (the minimum) (N = 48) and 16 % (the maximum) (N = 97) of the students were active in the MOOCs under study for the diagnostic exam. The activity in the MOOCs decreased during the traditional remedial courses period to 5 % (the minimum) (N = 29) and 13 % (the maximum) (N = 79) of the students, depending on the MOOC. Considering how the online initiative was promoted within the students, these percentages are less than what we expected. Prior studies show that the adoption is higher when MOOCs are proposed as a mandatory course.

Second, MOOCs are a good mechanism to help students refresh their previous knowledge on a particular topic regardless of not having any support, but they need to be carefully integrated with a traditional course in order to impact on students’ learning outcomes. The data of this study shows that those students that used the MOOC before the diagnostic exam had significantly more chances to pass this exam and skip the traditional required courses. Also, we observe through a regression analysis that passing the exam is not only dependent on the use of the MOOC, but also influenced by students’ NEM, MAT, CIE and/or RKG scores. This last result is not surprising, since previous studies show the importance of the students’ prior knowledge to succeed in a MOOC [12]. However, what it is interesting is that, when students participate in the MOOC as a complement to the traditional remedial course, no effects on the learning outcomes are observed and prior knowledge is the only variable able to predict the learning outcomes. Other case studies about blended learning approaches are especially useful when the MOOC is fully integrated as part of the traditional course [2, 7, 12]. These results suggest that service MOOCs that are not fully integrated with traditional courses might be not as beneficial for the students in terms of learning outcomes.

Third, the study of students’ adoption of MOOCs might signal what students are expecting to reinforce regarding the lack of opportunities to learn required skills and contents. A curriculum narrowing effect has emerged from the fact that the national admission test is not evaluating trigonometry, a branch of mathematics that is required for succeeding in engineering calculus courses. Therefore, the availability of M2 might have raised student awareness of the importance of this topic for succeeding not only in the diagnostic test, but also in their first year of college. Further research on MOOCs used as a complement for improving academic preparation for college should be addressed.

Fourth, the interactivity patterns show that students tend to be active in the MOOCs more intensively before the exams, but this activity is very different between the MOOCs’ topics and the phase of the study. The results of this study show that most of the movements on the course were registered before the diagnostic exam and before the exams of each remedial course. However, students show a better self-regulation pattern in the activity when the MOOC is aligned with the remedial face-to-face course. Several studies indicate that thanks to the work in virtual platforms, students can follow their own learning pace [4]. This is obvious, for example, when observing the different hours of the day that the students access the online course in our pilot study. But previous work has reported that although most of the participants in a MOOC tend to follow a linear path through the course content, these paths can vary depending on characteristics such as the age or the country of origin [10]. In addition, differences were observed on the activity patterns in each of the courses. Course M2 registered more movements than the other 3, followed by M1, then M4 and finally M3. Since all the courses where prepared by the same teachers and used the same resources, we suggest that this difference can be due to the needs of the students on the different course topics. For example, M2 and M3, which were the MOOCs registering a higher activity, work on topics that students do not practice in their previous studies before entering the university. But it could also be due to the quality of the MOOCs. Moreover, we need to take into account the students’ diversity, since some students might be interested only in certain parts of the course. Also there are students that lose interest as they advance in the courses, because they feel unable to achieve the MOOCs’ goals [7].

And fifth, service MOOCs should be designed for diversifying learning activities and exercises. We showed that most of the students’ activity was registered in the exercises. Recent work shows the importance of including exercises for practicing, especially in topics related with sciences and technology [16]. The results of this study corroborate the importance of designing MOOCs that include activities for exercising.

There is little empirical research that analyzes the effects of MOOC-based models in remedial courses in terms of students’ adoption and learning outcomes. This pilot study serves to prove that promoting the use of MOOCs as a complement for remedial traditional courses gives those students better chances of succeeding in the corresponding exams. Also, their interactivity in the MOOCs varies greatly given that students can follow their own learning pace.

Future work includes further investigation of the results obtained. First, more information needs to be extracted to better understand the reasons that moved active students to participate in the MOOCs and the reasons of those who did not. For example, the course content could not have been interesting enough, so evaluations on the MOOCs’ content would be needed to be able to judge this aspect. Second, we need to better understand how students’ self-regulate in these type of courses and what type of support they need to encourage future freshmen students to use the MOOCs and obtain better results in the diagnostic exam and remedial courses. Also, we should consider analyzing the students’ social learning aspects. Finally, and taking into account that the MOOCs are available also during the calculus courses of the first year, future work includes analyzing how is the adoption of these MOOCs during the first semester and what are the learning outcomes of those who used them more intensively.