A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students

Textbooks have traditionally been an essential part of the post-secondary experience for the majority of students in the United States. In a typical scenario, a professor assigns a textbook as the core instructional material for her class; students are obligated to purchase this book and use it to study the material in preparation for each class period. While the costs of these textbooks vary, Hilton et al. (2014) found that, across a series of general education courses (including science, math, humanities, and business) at seven different colleges, the average textbook price was approximately $90.00.

While all students face high textbook costs, individuals from lower socioeconomic backgrounds may face particular difficulties. Paulsen and St. John (2002) found that low and lower-middle income students reported that the financial implications of attending college were important factors in their choices regarding college. Provasnik and Plenty (2008) reported that individuals with lower incomes are more likely to delay college enrollment than wealthier peers. For some college students, the total cost of textbooks can exceed total tuition costs (Goodwin 2011). Some students, then, may be forced either not to purchase textbooks (presumably resulting in less learning) or take fewer classes (resulting in slower time to graduation) in order to manage or reduce college costs (Buczynski 2007).

Electronic textbooks promise a more affordable option for students. Electronic textbooks typically cost less than traditional textbooks due to the lack of printing costs. Rockinson-Szapkiw et al. (2013) found that utilizing electronic textbooks did not negatively impact student cognitive outcomes.

Another, even less expensive solution to rising textbook costs can be found in the utilization of open educational resources (OER). We next present a review of literature relating to OER and studies pertaining to the perceptions and efficacy of OER.

Open textbooks, which are a collection of OER aggregated in a manner that resembles a traditional textbook, take many shapes and forms. Typically, free digital versions of the textbook are made available to students. In addition, students who wish to purchase print versions of the textbooks can do so, at prices as low as $5 per textbook. While the quality of open textbooks varies, many go through rigorous editorial and design processes. Perhaps not surprisingly, students are favorably disposed towards replacing costly commercial textbooks with free open textbooks. Bliss et al. (2013) studied open textbook adoption at eight different institutions of higher education. Fifty-eight teachers and 490 students across the eight colleges completed surveys regarding their experiences in utilizing the open texts. Bliss and colleagues found that approximately 50 % of students said that the OER textbooks were of the same quality as traditional textbooks and nearly 40 % said that they were better. In their free-response comments, students focused on several benefits of the open textbooks, including cost-savings. For example, one student said, “I have no expendable income. Without this free text I would not be able to take this course.” In the same study, researchers found that 55 % of teachers adopting OER reported that the open materials were of the same quality as the materials they had previously used, and 35 % felt that they were better. One teacher in the study pointed out that “The materials were free to my students, which reduced a barrier to their chances for academic success.”

While Stratton et al. (2007) noted that results have been mixed in studies examining the relationship between student finances and their success in continuing through completion, several studies have indicated that greater financial resources correlate positively with student persistence. For example, Paulsen and St. John (2002) demonstrated that “the responsiveness of poor and working-class students to tuition increases is alarmingly high–reducing their probability of persisting by 16 and 19 %, respectively, per $1000 increment in tuition” (p. 229). While Paulsen and St. John did not discuss the cost of textbooks, it is interesting to note that the figure they used for an increase in tuition ($1000), is approximately the same amount of money full-time college students typically spend on textbooks per year. Thus one could argue that reducing textbook costs to zero could potentially increase persistence rates.

While not usually measured directly, it is possible that the use of no-cost or low-cost OER might free students’ resources to support increased credit loads which then enhance progress toward graduation. Wiley et al. (2015) analyzed the cost savings in courses with sections that used OER and sections that did not. The average cost of commercial textbooks across the courses was $140.85 which represented a potential total cost of $1,324,017.68 for that sample. In that instance, OER could have saved over one million dollars in textbook costs, which could have been applied directly to tuition for additional courses.

While financial reasons might be particularly persuasive to students and other educational stakeholders, the core purpose of education is to support learning. If the adoption of open textbooks decreases costs but also negatively influences student learning, educators should well view them with skepticism. While encouraging this skepticism, the authors fully acknowledge that institutions and educators everywhere trade improved affordability for lower outcomes on a regular basis. For example, colleges universally forego providing a full-time tutor for each student. Even though Bloom’s two-sigma work suggests this would greatly increase student learning, colleges instead choose to place students in educationally sub-optimal but significantly more affordable classes with many other students and a single instructor. Because this particular trade of sub-optimization for affordability is well established and broadly accepted, it is essentially invisible to many faculty. By contrast, a decrease in student learning associated with the adoption of open textbooks would be novel and likely to draw the negative attention of faculty, students, and other stakeholders. However, if learning outcomes actually improved in settings where open textbooks are utilized, there may be significant policy implications.

Perhaps because OER is relatively new, little research has been performed on how its utilization influences student learning. To date, six studies have compared student performance with and without implementing OER. These studies vary in rigor and all state that there are limitations to their findings. Nevertheless, they constitute the research done to the present time.

Lovett et al. (2008) measured the efficacy of an OER statistics module in comparison with the traditional educational model at Carnegie Mellon University. In two separate semesters, they invited students who had registered for an introductory statistics class at Carnegie Mellon to participate in an experimental online version of the course. Of those who volunteered, approximately one-third were randomly selected to take the online course, while the remaining two-thirds who had volunteered became the control group. The control group took the traditional, face-to-face statistics class at Carnegie Mellon. Researchers compared the results of these two groups in fall 2005 by examining their test scores (three midterms and one final exam), and found that there was no significant difference between the two groups. This experiment was replicated in spring 2006 with the same, non-significant, result. Thus, utilizing OER resulted in cost-savings without improving—or sacrificing—learning outcomes.

In another study focused on Carnegie Mellon’s open statistics modules, Bowen et al. (2014) compared the use of a traditional textbook in a face-to-face lecture class with that of a blended approach utilizing OER. Six hundred and five students took the OER version of the course, while 2439 took the traditional version. Bowen and colleagues found that, while students who utilized OER scored slightly higher than their peers on standardized exams, the difference was not statistically significant. A potential confound was that those utilizing OER received blended learning instead of traditional face-to-face instruction. Thus it is possible that the pedagogy masked the influence of OER. Nevertheless, it is relevant to note that in this study that the use of OER did not lead to lower student outcomes.

In a non-experimental case study, Hilton and Laman (2012) compared the performance of 690 students using an open textbook in an introductory psychology class to the performance of 370 students who used a traditional textbook in a previous semester. They concluded that students who used the open textbook achieved better grades in the course, had a lower withdrawal rate, and scored better on the final examination.

Feldstein et al. (2012) found that students in courses using open textbooks typically had higher grades and lower failure and withdrawal rates than those in courses with traditional textbooks. However, they did note significant limitations to their study suggesting that they provided only interesting data to be more rigorously pursued in the future.

Similarly, a case study presented by Hilton et al. (2013) focused on four math classes at Scottsdale Community College. These classes used the same departmental exam for each course for several years, which allowed faculty members to compare how students did on department exams when OER were used as compared with previous semesters. OER replaced traditional learning materials in fall 2012, and student results at the end of this semester were approximately the same as those obtained by students in fall 2011 and fall 2010.

Pawlyshyn et al. (2013) found that when OER material was integrated into the math courses at Mercy College, student learning significantly increased. The pass rates of math courses increased from 63.6 % in fall 2011 (when traditional learning materials were employed) to 68.9 % in fall 2012 when all courses were taught with OER. Similarly, students who were enrolled in OER versions of a reading course performed better than their peers who enrolled in the same course using non-OER materials.

Recent research indicates that a majority of faculty members perceive OER to be of approximately the same quality as traditional textbooks. Allen and Seaman (2014) surveyed 2144 college professors regarding OER. Of the 34 % (729) who were aware of OER, 61.5 % indicated OER had about the same “trusted quality” as traditional resources, 26.3 % said that traditional resources were superior, and 12.1 % said that OER were superior. Similarly, 68.2 % said that the “proven efficacy” were about the same, 16.5 % said that OER had superior efficacy, and 15.3 % said that traditional resources had superior efficacy.

Allen et al. (2015) studied an experimental class of 478 students that used OER known as ChemWiki for its primary textbook, while a control class of 448 utilized a commercial textbook. These two sections were taught the same semester at consecutive hours using the same faculty member and teaching assistants in order to control for potential confounds. Students in these classes received the same exams. No significant differences were found between the two groups. Beginning of the semester pre-tests combined with final exams showed no significant differences in individual learning gains between the two groups, thus indicating that OER could be substituted without any negative impact on learning.

While the aforementioned research provided interesting contextual case studies and varying degrees of statistical rigor, much more work needs to be done to ascertain the relationships between the use of OER and student academic performance. The purpose of this study was, therefore, to explore whether the use of open textbooks at 10 colleges significantly predicted learning outcomes in a group of 16,727 post-secondary students.

This is by far the largest study of its kind conducted to date—nearly 5000 post-secondary students using OER and over 11,000 control students using commercial textbooks, distributed among ten institutions across the United States, enrolled in 15 different undergraduate courses. In three key measures of student success—course completion, final grade of C- or higher, course grade– students whose faculty chose OER generally performed as well or better than students whose faculty assigned commercial textbooks.

In two key measures of enrollment intensity, which is an indicator of student progress toward graduation, students in courses that used OER were significantly different than students in courses with commercial textbooks. Even when controlling for differences in previous enrollment, students in courses using OER enrolled in a significantly higher number of credits in the next semester. This may be due to the cost savings associated with OER. In community college settings where tuition costs are based directly on the number of credits taken with no cap on costs for “full-time” enrollment, funds saved on textbooks can be applied directly to enrollment in additional courses.

The mechanisms underlying these improvements differ from those typically hypothesized to underlie improvement in student outcomes. Historically, comparison studies of instructional products have often been conducted to test hypotheses about differences in student outcomes attributable to alternate modes of delivery or instructional design approach (see Russell 2015). The authors do not believe differences of mode of delivery or instructional design between OER and commercial textbooks to be the primary mechanisms responsible for the differences in outcomes observed in this study. On the contrary, our informal review reveals strikingly similar, essentially equivalent instructional designs in the OER and commercial textbooks. We believe the effects demonstrated in this study result from differing degrees of access and affordability facilitated by open licenses used by OER.

The moderate differences in completion rates and final grades between the control and treatment groups are likely a function of access. Some percentage of students in the control group probably failed to purchase the commercial materials assigned by their faculty due to cost or other factors. For example, one survey suggested that 23 % of students regularly forego purchasing required textbooks due to their high cost (Florida Virtual Campus 2012). Students’ lack of access to the core instructional materials for the course put them at an academic disadvantage. All students in the treatment group had access to all the course materials from the very first day of class because they were openly licensed. Consequently, we would expect some enhanced probability of success for members of the treatment group.

The differences in enrollment intensity between the control and treatment groups are likely a function of affordability. Students whose faculty assign OER save a significant amount of money compared to students whose faculty assign commercial textbooks. Some treatment students will chose to reinvest these savings by taking an additional course in order to accelerate their graduation. Consequently, we would expect members of the treatment group to take more credits than the control group, on average.

Detecting differences in student outcomes based on access and affordability, rather than instructional design, points to several new horizons for educational research. Hundreds of millions of dollars and person-hours have been invested in improving in-class instructional designs, intelligent tutoring systems, adaptive instructional systems, and other design-related innovations intended to improve student outcomes. The current study demonstrates that at least one non-instructional design option exists that can effectively improve student outcomes.