Does Taking a MOOC as a Complement for Remedial Courses Have an Effect on My Learning Outcomes? A Pilot Study on Calculus
This paper presents the results of a pilot study about students’ adoption and learning outcomes of 4 MOOCs proposed as a complementary resource for traditional remedial courses on calculus. While the MOOCs were not mandatory, traditional remedial courses were required for those freshmen failing a diagnostic exam. The effects on 589 freshmen students were investigated. The data analysis shows that up to 16 % of the students were active in the MOOCs under study, mostly during the days before taking the diagnostic exam that preceded the traditional face-to-face remedial courses. Trace data about learner actions within the platform were collected as well as the students’ scores. According to a statistical comparison of the students’ exam scores and their interaction behavior with the MOOCs, we observe that active students had more chances of passing the diagnostic exam and skipping the required remedial courses. However, we found no significant differences on the remedial course exam scores between the students that were active in the MOOCs and those that were not. These findings suggest that MOOCs are a good solution to strengthening skills and reviewing concepts, but that more guidance is needed when used as a complement to traditional f2f courses.
KeywordsMoocs Remedial courses Higher education Pilot study Adoption Learning outcomes
Massively Open Online Courses (MOOCs) present new opportunities for facilitating teaching and learning . MOOCs allow flexible learning anytime and anywhere, diversifying the variety of tasks that can be included in any course structure . Lately, several case studies have documented different ways in which elite universities have integrated these courses into their curricula, broadening their teaching and learning strategies by implementing blended or hybrid learning approaches [5, 8, 17].
Two trends were observed in these case studies. The first trend (1) is using MOOCs as a complement of traditional teaching. For example, a study shows how Stanford University integrated MOOCs in a traditional course by asking students to watch video lectures, participate in discussion forums, complete quizzes and program assignments in an online platform . 26 students had to complement their learning with information about topics not addressed in the MOOC. The results show that students’ attendance increased by 20 % and their engagement with the course content increased by 40 % . Another example along these lines was developed by the University of Washington, which introduced blended learning in a traditional biology class. They were able to reduce its fail rate from 17 % to 4 %. Furthermore, the approval rates of the course increased from 14 % to 24 % since the initiative .
On the other hand, (2) MOOCs are used as remedial courses. Examples of these are the zero level courses developed by some universities. Universidad Carlos III de Madrid  analyzed the effect of a zero level course. In this experience, students took a diagnostic and a final exam, and the results indicated that students increased by 21 % the score in the final exam after the course. Regardless of other case studies in North America and Europe [1, 7], the effect of the MOOCs deserves further exploration in other countries to enrich current literature.
In order to contribute to the understanding of the MOOC-based models that use MOOCs to complement or substitute traditional remedial courses, this paper reports on the findings of a pilot study at the School of Engineering in Pontificia Universidad Católica de Chile (UC-Engineering). Specifically, we investigated the effects of 4 MOOCs on calculus for freshmen. From now on we call these MOOCs “service MOOCs” according the framework proposed in . That is, MOOCs that students take voluntarily (partially or completely), and as a complement to the curriculum or a traditional course but no institutional recognition is given for completing this MOOC. In Sect. 2, we describe the context in which this study was carried out, as well as the research questions addressed. Also in this section, we describe the participants of the study, the data gathering techniques and the procedures we used for the analysis. In Sects. 3 and 4, respectively, we report the main results obtained and the lessons learned from the study as well as its limitations. Finally, in Sect. 5, we present the main conclusions, and future avenues. Altogether, this work provides a better understanding of the effects of this type of MOOC-based initiatives in terms of students’ adoption and learning outcomes.
2 The Pilot Study
2.1 Context and Research Questions
About 600 Freshmen College students are admitted to the UC-Engineering every year. In order to get accepted in this program, students must be in the top positions of their high school ranking, besides demonstrating outstanding achievement in high school and in an admission exam that evaluates their knowledge in math, science, and language. Even so, students come with different understanding of basic calculus concepts, and their knowledge on these topics is often insufficient to successfully address the calculus courses that are imparted in the first year.
In the recent years, UC-Engineering freshmen have been required to take a calculus diagnostic exam right after they are informed that they have been admitted. The exam is divided into 4 modules: Algebra and Functions (M1), Trigonometry (M2), Polynomials and Complex Numbers (M3), and Sequences and Series (M4). Students that fail in a specific content are required to take a 2-day traditional course on each of the failed modules. In these courses, professors reinforce main theoretical topics, besides facilitating students’ learning with guided exercises. After each course, students have to take a final exam to evaluate their progress in the respective module content.
RQ1. What is the students’ adoption of this MOOC initiative? This question aims at studying the students’ use of the MOOCs in terms of their interactions with the course content in order to better understand who, how and when they use the provided courses.
RQ2. What are the effects of participating in the MOOCs in terms of students’ learning outcomes? This question aims at better understanding two aspects: (1) whether or not using the online platform before the diagnostic exam gives the students a better probability of passing it; and (2) whether or not students that use the MOOCs have better scores in the traditional remedial courses’ final exams.
2.2 Description of the Pilot Study
The pilot study took place at UC-Engineering between December 27th 2015 and 29th January, 2016. The MOOCs were produced by 3 teaching assistants and were deployed in the Open edX platform as part of the UC-Engineering online initiative1. The MOOCs did not follow the same structure than the traditional remedial courses. Nonetheless all the contents of the MOOC were designed to align with the learning objectives and topics addressed in the traditional remedial courses. The MOOCs were all open to anyone interested, both from and outside the UC-Engineering.
Pilot study timeline
27th Dec. 2015–10th Jan. 2016
Dissemination effort via e-mail, web-page and flyers to potential engineering students
Publication of the Admission Results (00:00 h)
Presentation session of the accepted students and registration to the platform.
Calculus Diagnostic Exam
Publication of exam results
18th Jan.–20th Jan.
M1 (Algebra and Functions)
Final exam of the traditional course M1
Link to the complementary service MOOC M1:
20th Jan.–25th Jan.
Final exam of the traditional course M2
Link to the complementary service MOOC M2:
25th Jan.–27th Jan.
M3 (Polynomials and Complex Numbers)
Final exam of the traditional course M3
Link to the complementary service MOOC M3:
27th Jan.–29th Jan.
M4 (Sequences and Series)
Final exam of the traditional course M4
Link to the complementary service MOOC M4:
2.3 Participants and Sample
Number of Students in each phase according to mathematical content.
Traditional remedial courses
504 (86 %)
85 (14 %)
53 (83 %)
11 (17 %)
170 (29 %)
419 (71 %)
219 (78 %)
62 (22 %)
261 (44 %)
328 (56 %)
208 (93 %)
15 (7 %)
325 (55 %)
264 (45 %)
104 (61 %)
67 (39 %)
2.4 Data Collection and Analysis
The data gathered from the sample of study came from many different sources. First, we worked with the students’ scores in the diagnostic exam (ScoresDE-M1, ScoresDE-M2, ScoresDE-M3 and ScoresDE-M4) and the scores obtained at each final exam of the required course (ScoresRE-M1, ScoresRE-M2, ScoresRE-M3 and ScoresRE-M4). These exams contemplate a 0–100 % scale, where a 100 % score would mean that they got every question right, and students passed the exams if they got a score of 50 % or higher.
The students’ activity and interaction patterns with the MOOCs are represented by the number of movements each student made in each MOOC before the diagnostic test and during the required courses. The movements were extracted from the MOOCs’ computational logs, where every action or movement each student does in the platform is registered (Logfiles). The numbers of active and non-active students are the measures of “adoption” in this study.
The students’ prior knowledge was defined as the students’ admission scores composed by: Math (MAT), Science (CIE), and Language (LEN) Chilean University Admission Exams scores, along with a score according to their high school grades (NEM) and class ranking (RKG). All these individual scores have a scale from 0 to 850. Finally, PING is the weight average admission score, computed as: 20 % NEM, 20 % RKG, 10 % LEN, 35 % MAT and 15 % CIE. These data is what we take as a reference of students’ prior knowledge and skills. Lastly, in order to understand academically where the students that adopted the MOOCs platform before the diagnostic exam came from, we divided the cohort in quartiles according to their PING. The groups are Q1, Q2, Q3 and Q4; where Q1 is the group with the lowest PING and Q4 is the one with the highest scores.
In order to address RQ1about the students’ adoption of the MOOC initiative and their behavior in the platform, we first organize the students into “active” and “non-active” depending on their usage of the platform in two periods: (1) before the diagnostic exam (Before Diagnostic Phase, BDP), and (2) during remedial courses (During Remedial Phase, DRP). We classified the students into these two groups by analyzing the number of movements that each student registered on the different MOOCs in each phase.
After classifying the students into active and non-active, we plotted the number of movements in a bar graph from the beginning of the study until the end to analyze the activity patterns in the different periods. Also, we analyzed the students’ interactions with both the video-lectures and the exercises (quizzes and other activities). We used this data to get an idea about whether the students used the MOOC for reviewing theoretical concepts through video-lectures or exercising.
In order to address RQ2 about the students’ learning outcomes we conducted several statistical analyses and looked for correlations between the students’ activity in the MOOCs with the scores they each obtained in the diagnostic exam and in the remedial course exams. These calculations allowed us to understand whether the interactivity levels have an influence on their results.
Then, in order to understand if the active students had more chances of passing the exams, we performed a t-test for the scores between the non-active and active students in both diagnostic exams and the remedial courses. Given that the results observed in this first analysis were significant for the diagnostic test, we applied a proportion test to the percentage of approval rates between active and non-active students. Thirdly, in order to understand the effect of the platform along with other variables that characterize the students’ prior knowledge, we performed a stepwise multivariable regression analysis that related the scores of the diagnostic or the remedial exams using as initial predictors the national admission exam scores NEM (high school GPA score), MAT (mathematics score), CIE (science score), and RKG (ranking score), and the categorical variable “active” or “non-active” student, which represents the platform adoption strategy of the student. All statistical analyses were carried out using MINITAB 17 (www.minitab.com).
This section reports on the results obtained from the analysis to address the two research questions. Subsect. 3.1 presents the results about the students’ adoption of the MOOC initiative, and Subsect. 3.2 about the effects on students’ learning outcomes.
3.1 Students’ Adoption of the MOOC Initiative
Active MOOC students vs. Non-active
Before diagnostic Phase, BDP
During remedial phase, DRP
14 % (N = 84)
86 % (N = 505)
7 % (N = 42)
93 % (N = 547)
16 % (N = 97)
84 % (N = 492)
13 % (N = 79)
87 % (N = 510)
8 % (N = 48)
92 % (N = 541)
5 % (N = 29)
95 % (N = 560)
12 % (N = 73)
88 % (N = 516)
10 % (N = 56)
90 % (N = 533)
Interactions captured in each MOOC section Before the Diagnostic Exam Phase (BDE) and During Remedial Phase (DRP) and proportions of interactions per MOOC per phase
503 (39 %)
793 (61 %)
194 (38 %)
316 (62 %)
439 (22 %)
1.516 (78 %)
240 (28 %)
626 (72 %)
37 (10 %)
341 (90 %)
44 (20 %)
181 (80 %)
248 (23 %)
853 (77 %)
40 (16 %)
205 (84 %)
1.227 (26 %)
3.503 (74 %)
580 (28 %)
1.328 (72 %)
3.2 Effects of the MOOC Initiative on Students’ Learning Outcomes
Diagnostic exam scores and final exam results from required courses to the students’ use of each MOOC.
Percentage of students that passed the diagnostic test, classified as Active and Non-active users.
Active users (n)
Non-active users (n)
Fisher’s exact test P-value
94 % (79)
84.1 % (425)
43.3 % (42)
26 % (128)
58.3 % (28)
43.1 % (233)
69.9 % (51)
53.1 % (274)
Regression analysis of the different course scores.
Traditional remedial courses
Adoption rates according to PING quartiles before the diagnostic exam.
4 Lessons Learned
The lessons reported in this section were obtained from reflecting on the pilot study results from both the student’s adoption and the students’ learning outcomes. In an effort to highlight those aspects of the study that could be applied to other contexts, we report on the limitations and analyze the issues that emerge from this work and would deserve further work.
First, students are not yet enough prepared to adopt MOOCs if proposed as a complement to traditional courses and if they are not mandatory. The results of our study show that between 8 % (the minimum) (N = 48) and 16 % (the maximum) (N = 97) of the students were active in the MOOCs under study for the diagnostic exam. The activity in the MOOCs decreased during the traditional remedial courses period to 5 % (the minimum) (N = 29) and 13 % (the maximum) (N = 79) of the students, depending on the MOOC. Considering how the online initiative was promoted within the students, these percentages are less than what we expected. Prior studies show that the adoption is higher when MOOCs are proposed as a mandatory course.
Second, MOOCs are a good mechanism to help students refresh their previous knowledge on a particular topic regardless of not having any support, but they need to be carefully integrated with a traditional course in order to impact on students’ learning outcomes. The data of this study shows that those students that used the MOOC before the diagnostic exam had significantly more chances to pass this exam and skip the traditional required courses. Also, we observe through a regression analysis that passing the exam is not only dependent on the use of the MOOC, but also influenced by students’ NEM, MAT, CIE and/or RKG scores. This last result is not surprising, since previous studies show the importance of the students’ prior knowledge to succeed in a MOOC . However, what it is interesting is that, when students participate in the MOOC as a complement to the traditional remedial course, no effects on the learning outcomes are observed and prior knowledge is the only variable able to predict the learning outcomes. Other case studies about blended learning approaches are especially useful when the MOOC is fully integrated as part of the traditional course [2, 7, 12]. These results suggest that service MOOCs that are not fully integrated with traditional courses might be not as beneficial for the students in terms of learning outcomes.
Third, the study of students’ adoption of MOOCs might signal what students are expecting to reinforce regarding the lack of opportunities to learn required skills and contents. A curriculum narrowing effect has emerged from the fact that the national admission test is not evaluating trigonometry, a branch of mathematics that is required for succeeding in engineering calculus courses. Therefore, the availability of M2 might have raised student awareness of the importance of this topic for succeeding not only in the diagnostic test, but also in their first year of college. Further research on MOOCs used as a complement for improving academic preparation for college should be addressed.
Fourth, the interactivity patterns show that students tend to be active in the MOOCs more intensively before the exams, but this activity is very different between the MOOCs’ topics and the phase of the study. The results of this study show that most of the movements on the course were registered before the diagnostic exam and before the exams of each remedial course. However, students show a better self-regulation pattern in the activity when the MOOC is aligned with the remedial face-to-face course. Several studies indicate that thanks to the work in virtual platforms, students can follow their own learning pace . This is obvious, for example, when observing the different hours of the day that the students access the online course in our pilot study. But previous work has reported that although most of the participants in a MOOC tend to follow a linear path through the course content, these paths can vary depending on characteristics such as the age or the country of origin . In addition, differences were observed on the activity patterns in each of the courses. Course M2 registered more movements than the other 3, followed by M1, then M4 and finally M3. Since all the courses where prepared by the same teachers and used the same resources, we suggest that this difference can be due to the needs of the students on the different course topics. For example, M2 and M3, which were the MOOCs registering a higher activity, work on topics that students do not practice in their previous studies before entering the university. But it could also be due to the quality of the MOOCs. Moreover, we need to take into account the students’ diversity, since some students might be interested only in certain parts of the course. Also there are students that lose interest as they advance in the courses, because they feel unable to achieve the MOOCs’ goals .
And fifth, service MOOCs should be designed for diversifying learning activities and exercises. We showed that most of the students’ activity was registered in the exercises. Recent work shows the importance of including exercises for practicing, especially in topics related with sciences and technology . The results of this study corroborate the importance of designing MOOCs that include activities for exercising.
5 Conclusions and Future Work
There is little empirical research that analyzes the effects of MOOC-based models in remedial courses in terms of students’ adoption and learning outcomes. This pilot study serves to prove that promoting the use of MOOCs as a complement for remedial traditional courses gives those students better chances of succeeding in the corresponding exams. Also, their interactivity in the MOOCs varies greatly given that students can follow their own learning pace.
Future work includes further investigation of the results obtained. First, more information needs to be extracted to better understand the reasons that moved active students to participate in the MOOCs and the reasons of those who did not. For example, the course content could not have been interesting enough, so evaluations on the MOOCs’ content would be needed to be able to judge this aspect. Second, we need to better understand how students’ self-regulate in these type of courses and what type of support they need to encourage future freshmen students to use the MOOCs and obtain better results in the diagnostic exam and remedial courses. Also, we should consider analyzing the students’ social learning aspects. Finally, and taking into account that the MOOCs are available also during the calculus courses of the first year, future work includes analyzing how is the adoption of these MOOCs during the first semester and what are the learning outcomes of those who used them more intensively.
This work was supported by FONDECYT (Chile) N 11150231, the MOOC-Maker Project (561533-EPP-1-2015-1-ES-EPPKA2-CBHE-JP), and the Comisión Nacional de Investigación Científica - CONICYT Ministry of Education, Chile, Ph.D.
- 1.Kop R., Fournier H., Sitlia H. The value of learning analytics to networked learning on a personal learning environment. In: Long P., Siemens G., Conole G., Gasevic D. (eds.) Proceedings of the First International Conference on Learning Analytics and Knowledge, Banff, Alberta, Canada, February 27–March 01 2011. National Research Council Canada, Banff (2011). http://nparc.cisti-icist.nrccnrc.gc.ca/npsi/ctrl?action=rtdoc&an=18150452
- 2.Aronson, N., Arfstrom, K., Tam, K.: Flipped Learning in Higher Education Always Learning. Pearson, London (2013)Google Scholar
- 3.Collier, A., Caulfield, M.: What happens in distributed flips? Investigating students’ interactions with MOOC videos and forums [Web log post], 2 May 2013. http://redpincushion.me/2013/05/02/what-happens-in-distributed-flips/
- 4.Fernandez, L.: B-learning as an alternative to lecturing for first semesters of engineering studies. II Congreso Internacional sobre Aprendizaje, Innovación y Competitividad (CINAIC) (2013)Google Scholar
- 5.Luján, S.: From the traditional lecture to the MOOC: twelve years of evolution of a subject about web application programming. Revista Docencia Universitaria 11, 279–300 (2013)Google Scholar
- 6.SCOPEO—SCOPEO INFORME Nº2: MOOC: Estado de la situación actual, posibilidades, retos y futuro. Junio 2013. Scopeo Informe No. 2 (2013). http://scopeo.usal.es/wp-content/uploads/2013/06/scopeoi002.pdf
- 7.Yin, R.K.: Preparing to Collect Cases Study Evidence, Case Study Research: Design and Methods, pp. 96–98. Sage publications, London (2013)Google Scholar
- 8.Zhang, Y.: Benefiting from MOOC. In: World Conference on Educational Multimedia, Hypermedia and Telecommunications, vol. 2013, no.1, pp. 1372–1377 (2013)Google Scholar
- 9.Del Campo, B., Macià, M., Manjabacas, G.: ¿Qué podemos hacer para solventar las carencias en matemáticas de los alumnos de nuevo ingreso?. Actas de las XX Jornadas de Enseñanza Universitaria de Informática, Jenui 2014, pp. 295–302, Oviedo, Julio 2014 (2014)Google Scholar
- 10.Guo, P.J., Reinecke, K.: Demographic differences in how students navigate through MOOCs. In: Proceedings of the First ACM Conference on Learning @ Scale Conference - L@S 2014, pp. 21–30 (2014). doi:10.1145/2556325.2566247
- 11.Muñoz, P, Méndez, E., Delgado, C.: SPOCs for Remedial Education: Experiences at the Universidad Carlos III de Madrid (2014)Google Scholar
- 12.Amaya, A., Valles, M.: Beneficios de los MOOC en la Educación Superior. Memorias del encuentro internacional de educación a distancia, vol. 4. Universidad de Guadalajara (2015)Google Scholar
- 13.Israel, M.: Effectiveness of integrating MOOCs in traditional classrooms for undergraduate students. Int. Rev. Res. Open Distrib. Learn. 6(5), 102–118 (2015)Google Scholar
- 15.Soffer, T., Cohen, A.: Implementation of Tel Aviv University MOOCs in academic curriculum: a pilot study. Int. Rev. Res. Open Distrib. Learn. 16, 80–97 (2015)Google Scholar
- 16.Alario-Hoyos, C., Kloos, C.D., Estévez-Ayres, I., Fernández-Panadero, C., Blasco, J., Pastrana, S., Villena-Román, J.: Interactive activities: the key to learning programming with MOOCs. In: Proceedings of the European Stakeholder Summit on Experiences and Best Practices in and Around MOOCs, EMOOCS 2016, p. 319 (2016)Google Scholar
- 17.Perez-Sanagustín, M., Hilliger, I., Alario-Hoyos, C., Delgado Kloos, C., Rayyan, S.: Describing MOOC-based Hybrid initiatives: The H-MOOC Framework. European MOOCs Stakeholders Summit EMOOCs 2016Google Scholar
Open Access. This chapter is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, a link is provided to the Creative Commons license and any changes made are indicated.
The images or other third party material in this chapter are included in the work's Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work's Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.