Keywords

1 Introduction

Massively Open Online Courses (MOOCs) present new opportunities for facilitating teaching and learning [14]. MOOCs allow flexible learning anytime and anywhere, diversifying the variety of tasks that can be included in any course structure [15]. Lately, several case studies have documented different ways in which elite universities have integrated these courses into their curricula, broadening their teaching and learning strategies by implementing blended or hybrid learning approaches [5, 8, 17].

Two trends were observed in these case studies. The first trend (1) is using MOOCs as a complement of traditional teaching. For example, a study shows how Stanford University integrated MOOCs in a traditional course by asking students to watch video lectures, participate in discussion forums, complete quizzes and program assignments in an online platform [13]. 26 students had to complement their learning with information about topics not addressed in the MOOC. The results show that students’ attendance increased by 20 % and their engagement with the course content increased by 40 % [3]. Another example along these lines was developed by the University of Washington, which introduced blended learning in a traditional biology class. They were able to reduce its fail rate from 17 % to 4 %. Furthermore, the approval rates of the course increased from 14 % to 24 % since the initiative [2].

On the other hand, (2) MOOCs are used as remedial courses. Examples of these are the zero level courses developed by some universities. Universidad Carlos III de Madrid [11] analyzed the effect of a zero level course. In this experience, students took a diagnostic and a final exam, and the results indicated that students increased by 21 % the score in the final exam after the course. Regardless of other case studies in North America and Europe [1, 7], the effect of the MOOCs deserves further exploration in other countries to enrich current literature.

In order to contribute to the understanding of the MOOC-based models that use MOOCs to complement or substitute traditional remedial courses, this paper reports on the findings of a pilot study at the School of Engineering in Pontificia Universidad Católica de Chile (UC-Engineering). Specifically, we investigated the effects of 4 MOOCs on calculus for freshmen. From now on we call these MOOCs “service MOOCs” according the framework proposed in [17]. That is, MOOCs that students take voluntarily (partially or completely), and as a complement to the curriculum or a traditional course but no institutional recognition is given for completing this MOOC. In Sect. 2, we describe the context in which this study was carried out, as well as the research questions addressed. Also in this section, we describe the participants of the study, the data gathering techniques and the procedures we used for the analysis. In Sects. 3 and 4, respectively, we report the main results obtained and the lessons learned from the study as well as its limitations. Finally, in Sect. 5, we present the main conclusions, and future avenues. Altogether, this work provides a better understanding of the effects of this type of MOOC-based initiatives in terms of students’ adoption and learning outcomes.

2 The Pilot Study

2.1 Context and Research Questions

About 600 Freshmen College students are admitted to the UC-Engineering every year. In order to get accepted in this program, students must be in the top positions of their high school ranking, besides demonstrating outstanding achievement in high school and in an admission exam that evaluates their knowledge in math, science, and language. Even so, students come with different understanding of basic calculus concepts, and their knowledge on these topics is often insufficient to successfully address the calculus courses that are imparted in the first year.

In the recent years, UC-Engineering freshmen have been required to take a calculus diagnostic exam right after they are informed that they have been admitted. The exam is divided into 4 modules: Algebra and Functions (M1), Trigonometry (M2), Polynomials and Complex Numbers (M3), and Sequences and Series (M4). Students that fail in a specific content are required to take a 2-day traditional course on each of the failed modules. In these courses, professors reinforce main theoretical topics, besides facilitating students’ learning with guided exercises. After each course, students have to take a final exam to evaluate their progress in the respective module content.

Although this strategy has been a way of promoting students’ calculus readiness, the experience from the last two years has shown some limitations: (1) low participation rates in the required remedial courses due to the fact that students that do not live in Santiago had difficulties to attend face-to-face courses; and (2) lack of individualized instruction considering that not all the students need to review the same topics. In order to address these limitations, last year the school decided to produce a service MOOC for each module and offer them as a complementary support for students’ learning in the specific theoretical concepts. Since participating in the MOOCs was voluntarily, the main objective of this study was to analyze the impact of this initiative both in terms of students’ adoption and learning outcomes. Specifically, two research questions were addressed:

  • RQ1. What is the students’ adoption of this MOOC initiative? This question aims at studying the students’ use of the MOOCs in terms of their interactions with the course content in order to better understand who, how and when they use the provided courses.

  • RQ2. What are the effects of participating in the MOOCs in terms of students’ learning outcomes? This question aims at better understanding two aspects: (1) whether or not using the online platform before the diagnostic exam gives the students a better probability of passing it; and (2) whether or not students that use the MOOCs have better scores in the traditional remedial courses’ final exams.

2.2 Description of the Pilot Study

The pilot study took place at UC-Engineering between December 27th 2015 and 29th January, 2016. The MOOCs were produced by 3 teaching assistants and were deployed in the Open edX platform as part of the UC-Engineering online initiativeFootnote 1. The MOOCs did not follow the same structure than the traditional remedial courses. Nonetheless all the contents of the MOOC were designed to align with the learning objectives and topics addressed in the traditional remedial courses. The MOOCs were all open to anyone interested, both from and outside the UC-Engineering.

The MOOCs were available before the students knew that they had been admitted in UC-Engineering. MOOCs were announced by e-mail and flyers a week before releasing the admission results to all those that had manifested their interest in studying at UC-Engineering. Additional outreach to students involved posting in the official Engineers’ web page, so all prospective students were informed that they could register on the platform and take MOOC. Once accepted, all freshmen were registered in the MOOC provider platform during the admission day, so all of them could access the 4 MOOCs. All the MOOCs are self-paced, so no restrictions or deadlines were proposed. Students were also informed that the participation in the MOOC courses was voluntary. Students were required to take a diagnostic exam to assess their prior knowledge and skills in calculus. Depending on their results on the diagnostic exam, students had to attend the mandatory specific remedial courses that were imparted traditionally before the first semester begins. Table 1 shows a time line of the different milestones in this case study, showing also the duration of each traditional remedial course and the dates of the final exams that the students took after participating in a required course to evaluate their progress in the respective content.

Table 1. Pilot study timeline
Full size table

2.3 Participants and Sample

Although the MOOCs were open to anyone, in this study we only took as a sample for the analysis those students that were admitted in UC-Engineering and took the diagnostic exam on calculus. 589 students (N = 589) took the diagnostic exam on calculus. Those who passed the diagnostic exam (Students Passing Diagnostic, SPD) and those who did not (Students Failing Diagnostic, SFD) were the sample of analysis of our study. Since not all attended the remedial courses if they failed the exam, we separated the sample into two groups: students that attended the traditional remedial courses (Students Attending Remedial, SAR), and distinguished among those who passed the corresponding final exam (Students Passing Remedial, SPR) and those who did not (Students Failing Remedial, SFR) (Table 2).

Table 2. Number of Students in each phase according to mathematical content.
Full size table

2.4 Data Collection and Analysis

The data gathered from the sample of study came from many different sources. First, we worked with the students’ scores in the diagnostic exam (ScoresDE-M1, ScoresDE-M2, ScoresDE-M3 and ScoresDE-M4) and the scores obtained at each final exam of the required course (ScoresRE-M1, ScoresRE-M2, ScoresRE-M3 and ScoresRE-M4). These exams contemplate a 0–100 % scale, where a 100 % score would mean that they got every question right, and students passed the exams if they got a score of 50 % or higher.

The students’ activity and interaction patterns with the MOOCs are represented by the number of movements each student made in each MOOC before the diagnostic test and during the required courses. The movements were extracted from the MOOCs’ computational logs, where every action or movement each student does in the platform is registered (Logfiles). The numbers of active and non-active students are the measures of “adoption” in this study.

The students’ prior knowledge was defined as the students’ admission scores composed by: Math (MAT), Science (CIE), and Language (LEN) Chilean University Admission Exams scores, along with a score according to their high school grades (NEM) and class ranking (RKG). All these individual scores have a scale from 0 to 850. Finally, PING is the weight average admission score, computed as: 20 % NEM, 20 % RKG, 10 % LEN, 35 % MAT and 15 % CIE. These data is what we take as a reference of students’ prior knowledge and skills. Lastly, in order to understand academically where the students that adopted the MOOCs platform before the diagnostic exam came from, we divided the cohort in quartiles according to their PING. The groups are Q1, Q2, Q3 and Q4; where Q1 is the group with the lowest PING and Q4 is the one with the highest scores.

In order to address RQ1 about the students’ adoption of the MOOC initiative and their behavior in the platform, we first organize the students into “active” and “non-active” depending on their usage of the platform in two periods: (1) before the diagnostic exam (Before Diagnostic Phase, BDP), and (2) during remedial courses (During Remedial Phase, DRP). We classified the students into these two groups by analyzing the number of movements that each student registered on the different MOOCs in each phase.

After classifying the students into active and non-active, we plotted the number of movements in a bar graph from the beginning of the study until the end to analyze the activity patterns in the different periods. Also, we analyzed the students’ interactions with both the video-lectures and the exercises (quizzes and other activities). We used this data to get an idea about whether the students used the MOOC for reviewing theoretical concepts through video-lectures or exercising.

In order to address RQ2 about the students’ learning outcomes we conducted several statistical analyses and looked for correlations between the students’ activity in the MOOCs with the scores they each obtained in the diagnostic exam and in the remedial course exams. These calculations allowed us to understand whether the interactivity levels have an influence on their results.

Then, in order to understand if the active students had more chances of passing the exams, we performed a t-test for the scores between the non-active and active students in both diagnostic exams and the remedial courses. Given that the results observed in this first analysis were significant for the diagnostic test, we applied a proportion test to the percentage of approval rates between active and non-active students. Thirdly, in order to understand the effect of the platform along with other variables that characterize the students’ prior knowledge, we performed a stepwise multivariable regression analysis that related the scores of the diagnostic or the remedial exams using as initial predictors the national admission exam scores NEM (high school GPA score), MAT (mathematics score), CIE (science score), and RKG (ranking score), and the categorical variable “active” or “non-active” student, which represents the platform adoption strategy of the student. All statistical analyses were carried out using MINITAB 17 (www.minitab.com).

3 Results

This section reports on the results obtained from the analysis to address the two research questions. Subsect. 3.1 presents the results about the students’ adoption of the MOOC initiative, and Subsect. 3.2 about the effects on students’ learning outcomes.

3.1 Students’ Adoption of the MOOC Initiative

R1.1. Up to 16 % of the students were active in the MOOCs. Active students used the MOOCs more before the diagnosis exam than during the required courses. Between 5 % and 16 % were active in the MOOCs. As shown in Table 3, M2 is the course that concentrated most of the activity, followed by M1, M4, and M3. M2 is a MOOC about trigonometry, a content that is no longer evaluated in the college admission test since 2014Footnote 2.

Table 3. Active MOOC students vs. Non-active
Full size table

Figure 1 shows the activity of the students during the pilot study. The average number of interactions per day per MOOC during the three days before the diagnostic exam (from January 11th, which is when the students found out they had been accepted, to January 13th) is 591 (with a total of 7.095 learner actions traced), whereas there are only 61 daily interactions per MOOC during the face-to-face required courses (with a total of 3.701movements registered from January 14th through January 29th). Specifically, students interacted more with each MOOC during their participation in the required course. M1 and M2 were the MOOCs most used.

Fig. 1.
figure 1

Total amount of movements in the 4 MOOCs before the calculus exam and during the courses

Full size image

R1.2. Students used the courses for exercising. Table 4 shows that the exercise sections registered more interactions than the video sections. This result is observed in all courses and in both phases. By both phases, we mean before the diagnostic exam and during the remedial courses.

Table 4. Interactions captured in each MOOC section Before the Diagnostic Exam Phase (BDE) and During Remedial Phase (DRP) and proportions of interactions per MOOC per phase
Full size table

3.2 Effects of the MOOC Initiative on Students’ Learning Outcomes

R2.1. Students who were active in the MOOCs before the diagnostic exam showed better scores on this exam, but no significant effect was observed in the scores of students that were required to take final exams after traditional face-to-face courses. Results in Table 5 indicate that there is no statistically significant difference in the final scores of the remedial exams (ScoreRE-M1…M4) between those students that were active in the MOOCs and those who were not active. The only exception corresponds to ScoreRE-M4, where active students obtained a lower mean score compared to the non-actives ones. In contrast, we found that the mean scores of the active users were significantly higher than the non-active students in all cases of the Diagnostic test (ScoreDE-M1…M4).

Table 5. Diagnostic exam scores and final exam results from required courses to the students’ use of each MOOC.
Full size table

R2.2. Students that were active users in the MOOCs before the diagnostic exam reported statistically higher approval rates in this test. Results in Table 6 show that the percentage of active users passing the Diagnostic Exam is higher than those who were non-active. This result is especially different (with more than 17.3 points of difference) for the one that took the M2 MOOC, which corresponds to the MOOC that registered the higher amount of learner actions (see Fig. 1).

Table 6. Percentage of students that passed the diagnostic test, classified as Active and Non-active users.
Full size table

R2.3. Being active in the MOOC platform appears to be a predictor variable for the score of the Diagnostic Exam, but not for the scores on the final exams of required courses, in which the only predictor variable is the math scores the students got on their University Admission Exams (MAT). Table 7 shows the results of the stepwise multivariable regression analysis. This analysis allowed us to have a better understanding of what variables explain better the approval rates in each of the phases. The results in Table 7 show that several of the predictors were statistically significant for the diagnostic exam phase, including the categorical variable “Active user” (taken as a measure of adoption). For the traditional remedial courses, only the MAT score was a statistically significant predictor of the final exam score in each course.

Table 7. Regression analysis of the different course scores.
Full size table

R2.4. The activity rates on the MOOCs do not depend on the PING (student’s final admission score). Table 8 shows the percentage of active students that fall in each of the quartiles by PING. The results show that the percentages of active students are similar independent to the quartile they belong to.

Table 8. Adoption rates according to PING quartiles before the diagnostic exam.
Full size table

4 Lessons Learned

The lessons reported in this section were obtained from reflecting on the pilot study results from both the student’s adoption and the students’ learning outcomes. In an effort to highlight those aspects of the study that could be applied to other contexts, we report on the limitations and analyze the issues that emerge from this work and would deserve further work.

First, students are not yet enough prepared to adopt MOOCs if proposed as a complement to traditional courses and if they are not mandatory. The results of our study show that between 8 % (the minimum) (N = 48) and 16 % (the maximum) (N = 97) of the students were active in the MOOCs under study for the diagnostic exam. The activity in the MOOCs decreased during the traditional remedial courses period to 5 % (the minimum) (N = 29) and 13 % (the maximum) (N = 79) of the students, depending on the MOOC. Considering how the online initiative was promoted within the students, these percentages are less than what we expected. Prior studies show that the adoption is higher when MOOCs are proposed as a mandatory course.

Second, MOOCs are a good mechanism to help students refresh their previous knowledge on a particular topic regardless of not having any support, but they need to be carefully integrated with a traditional course in order to impact on students’ learning outcomes. The data of this study shows that those students that used the MOOC before the diagnostic exam had significantly more chances to pass this exam and skip the traditional required courses. Also, we observe through a regression analysis that passing the exam is not only dependent on the use of the MOOC, but also influenced by students’ NEM, MAT, CIE and/or RKG scores. This last result is not surprising, since previous studies show the importance of the students’ prior knowledge to succeed in a MOOC [12]. However, what it is interesting is that, when students participate in the MOOC as a complement to the traditional remedial course, no effects on the learning outcomes are observed and prior knowledge is the only variable able to predict the learning outcomes. Other case studies about blended learning approaches are especially useful when the MOOC is fully integrated as part of the traditional course [2, 7, 12]. These results suggest that service MOOCs that are not fully integrated with traditional courses might be not as beneficial for the students in terms of learning outcomes.

Third, the study of students’ adoption of MOOCs might signal what students are expecting to reinforce regarding the lack of opportunities to learn required skills and contents. A curriculum narrowing effect has emerged from the fact that the national admission test is not evaluating trigonometry, a branch of mathematics that is required for succeeding in engineering calculus courses. Therefore, the availability of M2 might have raised student awareness of the importance of this topic for succeeding not only in the diagnostic test, but also in their first year of college. Further research on MOOCs used as a complement for improving academic preparation for college should be addressed.

Fourth, the interactivity patterns show that students tend to be active in the MOOCs more intensively before the exams, but this activity is very different between the MOOCs’ topics and the phase of the study. The results of this study show that most of the movements on the course were registered before the diagnostic exam and before the exams of each remedial course. However, students show a better self-regulation pattern in the activity when the MOOC is aligned with the remedial face-to-face course. Several studies indicate that thanks to the work in virtual platforms, students can follow their own learning pace [4]. This is obvious, for example, when observing the different hours of the day that the students access the online course in our pilot study. But previous work has reported that although most of the participants in a MOOC tend to follow a linear path through the course content, these paths can vary depending on characteristics such as the age or the country of origin [10]. In addition, differences were observed on the activity patterns in each of the courses. Course M2 registered more movements than the other 3, followed by M1, then M4 and finally M3. Since all the courses where prepared by the same teachers and used the same resources, we suggest that this difference can be due to the needs of the students on the different course topics. For example, M2 and M3, which were the MOOCs registering a higher activity, work on topics that students do not practice in their previous studies before entering the university. But it could also be due to the quality of the MOOCs. Moreover, we need to take into account the students’ diversity, since some students might be interested only in certain parts of the course. Also there are students that lose interest as they advance in the courses, because they feel unable to achieve the MOOCs’ goals [7].

And fifth, service MOOCs should be designed for diversifying learning activities and exercises. We showed that most of the students’ activity was registered in the exercises. Recent work shows the importance of including exercises for practicing, especially in topics related with sciences and technology [16]. The results of this study corroborate the importance of designing MOOCs that include activities for exercising.

5 Conclusions and Future Work

There is little empirical research that analyzes the effects of MOOC-based models in remedial courses in terms of students’ adoption and learning outcomes. This pilot study serves to prove that promoting the use of MOOCs as a complement for remedial traditional courses gives those students better chances of succeeding in the corresponding exams. Also, their interactivity in the MOOCs varies greatly given that students can follow their own learning pace.

Future work includes further investigation of the results obtained. First, more information needs to be extracted to better understand the reasons that moved active students to participate in the MOOCs and the reasons of those who did not. For example, the course content could not have been interesting enough, so evaluations on the MOOCs’ content would be needed to be able to judge this aspect. Second, we need to better understand how students’ self-regulate in these type of courses and what type of support they need to encourage future freshmen students to use the MOOCs and obtain better results in the diagnostic exam and remedial courses. Also, we should consider analyzing the students’ social learning aspects. Finally, and taking into account that the MOOCs are available also during the calculus courses of the first year, future work includes analyzing how is the adoption of these MOOCs during the first semester and what are the learning outcomes of those who used them more intensively.