Systematic Reviews on Flipped Learning in Various Education Contexts
This chapter shares the author's experiences of and reflections on conducting systematic reviews of flipped classroom research. The author first discusses the rationale for conducting systematic reviews and how the reviews contribute to the flipped learning field. After that, he lighlights some possible strategies, regarding literature search, article selection, and research synthesis, to improve the quality of systematic reviews.
In recent years, numerous studies about the flipped (or inverted) classroom approach have been published (Chen et al. 2017; Karabulut-Ilgu et al. 2018). In a typical flipped classroom, students learn course materials before class by watching instructional videos (Bishop and Verleger 2013; Lo and Hew 2017). Class time is then freed up for more interactive learning activities, such as group discussions (Lo et al. 2017; O’Flaherty and Phillips 2015). In contrast to a traditional lecture-based learning environment, students in flipped classrooms can pause or replay the instructor’s presentation in video lectures without feeling embarrassed. These functions enable them to gain a better understanding of course materials before moving on to new topics (Abeysekera and Dawson 2015). Moreover, instructors are no longer occupied by direct lecturing and can thus better reach every student inside the classroom. For example, Bergmann and Sams (2008) provide one-to-one assistance and small group tutoring during their class meetings.
The growth in research on flipped classrooms is reflected in the increasing number of literature review studies. Many of these are systematic reviews (e.g., Betihavas et al. 2016; Chen et al. 2017; Karabulut-Ilgu et al. 2018; Lundin et al. 2018; O’Flaherty and Phillips 2015; Ramnanan and Pound 2017). One would expect that if the scope of review has remained unchanged, contemporary reviews would include and analyze more research articles than the earlier reviews. Moreover, because flipped classroom practice is becoming more innovative (e.g., gamified flipped classroom), recent reviews should provide new insights into future research and practice. However, this is not always the case.
Summary of the systematic reviews of flipped classroom research written by the author (in chronological order)
2 Rationale for Conducting Systematic Reviews
To avoid repeating previous research efforts, researchers should first understand the current state of the literature by either examining existing reviews or conducting their own systematic review. Phrases such as “little research has been done” and “there is a lack of research” are extensively used to justify a newly written article. However, I sometimes doubt the grounds for these claims. There is no longer a lack of research in the field of flipped learning. In mathematics education alone, for example, 61 peer-reviewed empirical studies were published between 2012 to 2016 (Lo et al. 2017). Karabulut-Ilgu et al. (2018) found 62 empirical research articles on flipped engineering education as of May 2015. Through a systematic review of the literature, a more comprehensive picture of current research can be revealed.
In fact, before conducting my studies of flipped learning in secondary schools, I carried out a systematic review in the context of K-12 education (Lo and Hew 2017). At the time of writing (October 2016), only 15 empirical studies existed. We therefore knew little (at that time) about the effect of flipped learning on K-12 students’ achievement under this instructional approach. With such a small number of research published, the systematic review thus provided a justification for our planned studies (see Lo et al. 2018 for a review) and those of other researchers (e.g., Tseng et al. 2018) to examine the use of the flipped classroom approach in K-12 contexts.
In addition to understanding the current state of the literature, systematic reviews help identify research gaps. In flipped mathematics education, for example, Naccarato and Karakok (2015) hypothesized that instructors “used videos for the delivery of procedural knowledge and left conceptual ideas for face-to-face interactions” (p. 973). However, researchers have not reached a consensus on course planning using the lens of procedural and conceptual knowledge. While Talbert (2014) found that students were able to acquire both procedural and conceptual knowledge by watching instructional videos, Kennedy et al. (2015) discovered that flipping conceptual content might impair student achievement. More importantly, we found in our systematic review that very few studies evaluated the effect of flipping specific types of materials, such as procedural and conceptual problems (Lo et al. 2017). To flip or not to flip the conceptual knowledge? That is a key question for future studies of flipped mathematics learning.
3 Contribution of Systematic Reviews
Some possible goals and contributions of systematic reviews of flipped classroom research
To inform future flipped classroom practice
• Develop a 5E flipped classroom model for history education (Lo 2017).
• Make 10 recommendations for flipping K-12 education (Lo and Hew 2017).
• Establish a set of design principles for flipped mathematics classrooms (Lo et al. 2017).
To examine the overall effect of flipped learning
• Provide quantitative evidence that flipped learning improves student achievement more than traditional learning in the following subject areas.
– Mathematics education (Lo et al. 2017)
– Health professions (Hew and Lo 2018)
– Engineering education (Lo and Hew 2019)
• Reveal that the use of the following instructional activities can further promote the effect of flipped learning.
– A brief review of pre-class materials (Lo and Hew 2019)
First, the overarching goal of some of our systematic reviews was to inform future flipped classroom practice. Using the findings of the reviewed studies, we have developed a 5E flipped classroom model for history education (Lo 2017), made 10 recommendations for flipping K-12 education (Lo and Hew 2017), and established a set of design principles for flipped mathematics classrooms (Lo et al. 2017). Taking the design principles for flipped mathematics classrooms as an example, our Principle 4 suggested that short videos could be used to enable effective multimedia learning. This principle was based on the problem (reported in the literature) that students tend to disengage when watching long videos. To avoid making similar mistakes, we recommended that each video be limited to six minutes and all combined video segments be no more than 20–25 min. With this principle applied, Chen and Chen (2018) confirmed that the assigned workload was bearable for the students in their flipped research methodology course.
Second, the goal of our systematic reviews was to examine the effect of flipped learning versus traditional learning on student achievement. These reviews focus on flipped mathematics education (Lo et al. 2017), health professions (Hew and Lo 2018), and engineering education (Lo and Hew 2019). Researchers have conducted several systematic reviews of flipped learning in the health professions (Chen et al. 2017; Ramnanan and Pound 2017) and engineering education (Karabulut-Ilgu et al. 2018). Ramnanan and Pound (2017) reported that medical students were generally satisfied with flipped learning and preferred this instruction approach to traditional lecture-based learning. However, strong satisfaction with learning does not necessarily mean improved achievement. Examining student learning outcomes, Karabulut-Ilgu et al. (2018) classified their flipped-traditional comparison studies into five categories: (1) More effective, (2) more effective and/or no difference, (3) no difference, (4) less effective, and (5) less effective and/or no difference. As in Chen et al. (2017), they presented the effect size of each flipped-traditional comparison study. However, as Karabulut-Ilgu et al. (2018) acknowledged, no definitive conclusion can be made without a meta-analysis of student achievement in flipped classrooms.
We therefore attempted to examine the overall effect of flipped learning on student achievement through systematic reviews of the empirical research. The findings enhance our understanding of this instructional approach. Using a meta-analytic approach, a small but significant difference in effect in favor of flipped learning over traditional learning was found in all three contexts (i.e., mathematics education, health professions, and engineering education). Most importantly, our moderator analyses provided quantitative support for a brief review and/or formative assessment of pre-class materials at the start of face-to-face lessons. The effect of flipped learning was further promoted when instructors provided such an assessment (for mathematics education and health professions) and/or review (for engineering education) in their flipped classrooms. These findings not only extend our understanding of flipped learning, but also inform future practice of flipped classrooms (e.g., offering a quiz on pre-class materials at the start of face-to-face lessons).
4 Reflections on Some Practical Issues of Conducting Systematic Reviews
The following sections cover some practical aspects of systematic reviews of flipped classroom research, including literature search, article selection, and research synthesis.
4.1 Literature Search
Abeysekera and Dawson (2015) shared their experiences of searching for articles on flipped classrooms. They performed their search using the term “flipped classroom” in the ERIC database. In June 2013, they found only two peer-reviewed articles on flipped learning. Although not much research had been published at that time, this scarcity of search outcome has prompted us to reflect on (1) the design of the search string and (2) the choice of databases when conducting a systematic review.
4.2 The Design of Search String
The search term “flipped classroom” is very specific in that it cannot include other terms used to describe this instructional approach, such as flipped learning, flipping classrooms, and inverted classrooms. From my observation, some authors use even more flexible wording. For example, Talbert (2014) entitled his article “Inverting the Linear Algebra Classroom” (p. 361). If certain keywords are not included in their title, abstract, and keywords, their articles might not be retrieved through a narrow database search.
Search strings used in systematic reviews of flipped classroom research
History education (Lo, 2017)
(flip* OR invert*)
(class* OR learn*)
K-12 education (Lo and Hew 2017)
(flip* OR invert*)
(class* OR learn*)
(K12 OR K-12 OR primary OR elementary OR secondary OR “high school” OR “middle school”)
Mathematics education (Lo et al. 2017)
(flip* OR invert*)
(class* OR learn*)
(math* OR algebra OR trigonometry OR geometry OR calculus OR statistics)
Our search strings comprised two parts: (1) The instructional approach, and (2) the context. In the first part, “(flip* OR invert*) AND (class* OR learn*)” allowed us to capture different combinations of terms about flipped learning. In the second part, we used various search terms to specify the research contexts (e.g., K12 OR K-12 OR primary OR elementary OR secondary OR “high school” OR “middle school”) or subject areas (e.g., math* OR algebra OR trigonometry OR geometry OR calculus OR statistics) that we wanted. As a result, we were able to reach research items that had seldom been downloaded and cited.
Improved search strings used in systematic reviews of flipped classroom research
Health professions (Hew and Lo 2018)
(flip* OR invert*)
(class* OR learn* OR instruction* OR course*)
(medic* OR nurs* OR pharmac* OR physiotherap* OR dental OR dentist* OR chiropract*)
Engineering education (Lo and Hew 2019)
(flip* OR invert*)
(class* OR learn* OR course*)
As a side note about the design of search strings, one researcher emailed me about our systematic review of flipped mathematics education (Lo et al. 2017). He told me that our review has missed his article, an experimental study of flipped mathematics learning. After careful checking, his study perfectly fulfilled all inclusive criteria for our systematic review. However, I could not find any variations of “mathematics” or other possible identifiers of subject areas (e.g., algebra, calculus, and statistics) in his title, abstract, and keywords. That is why we were unable to retrieve his article through database searching using our search string.
At this point, I still believe that the context part of our search string of flipped mathematics education (i.e., math* OR algebra OR trigonometry OR geometry OR calculus OR statistics) is broad enough to capture the flipped classroom research conducted in mathematics education. However, this search string cannot capture studies that do not describe their subject domain at all. Without this information, other readers would have no idea about where the work is situated within the broader field of flipped learning if they only scan the title, abstract, and keywords. Most importantly, this valuable piece of work cannot be retrieved in a database search. Other snowballing strategies, such as tracking the reference lists of reviewed studies (see Lo 2017; Wohlin 2014 for a review), should be applied to find these articles in future systematic reviews.
4.3 The Choice of Databases
Apart from the aforementioned databases, other researchers (e.g., Lundin et al. 2018; O’Flaherty and Phillips 2015; Ramnanan and Pound 2017) have used the following databases in their systematic reviews of flipped learning: Cochrane library, EMBASE, Joanna Briggs Institute, Scopus, and Web of Science. In future systematic reviews, relevant databases need to be consulted. Researchers can follow existing reviews in their research field or consult librarians for advice on which databases to use.
4.4 Article Selection
A few controversial criteria for article selection
Citation (Lundin et al. 2018)
Studies are cited at least 15 times in Scopus
Studies are cited less than 15 times in Scopus
Why “15 times” is set, instead of 10 or other possibilities? Will this criterion become a threshold for recently published studies?
Ethics clearance (O’Flaherty and Phillips 2015)
Studies with approved ethics notification
Studies without approved ethics notification
Is it common practice in the field of flipped learning to explicitly acknowledge ethical approval in journal writing?
Taking a recent systematic review by Lundin et al. (2018) as an example, they reviewed the most-cited publications on flipped learning. They only included publications that were cited at least 15 times in the Scopus database. With such a constraint, 493 out of 530 documents were excluded in the early stage of their review. Only 31 articles were ultimately included in their synthesis. This particular criterion could block the inclusion of recently published articles because it takes time to accumulate a number of citations. The majority of the articles that they included were published in 2012 (n = 6), 2013 (n = 16), and 2014 (n = 5), with only a scattering of articles from 2000 (n = 1), 2008 (n = 1), and 2015 (n = 2). No documents after 2016 were included in their systematic review. The authors argued that citation frequency is “an indicator of which texts are widely used in this emerging field of research” (p. 4). However, further justification may help highlight the value of examining this particular set of documents instead of a more comprehensive one. They also have to provide a strong rationale for their 15+ citation threshold (as opposed to 10+ or other possibilities).
In our systematic reviews, we also added a controversial criterion for article selection, the definition of the flipped classroom approach. In my own conceptualization, “Inverting the classroom means that events that have traditionally taken place inside the classroom now take place outside the classroom and vice versa” (Lage et al. 2000, p. 32). What traditionally takes place inside the classroom is instructor lecturing. Therefore, I agree with the definition of Bishop and Verleger (2013) that instructional videos (or other forms of multimedia materials) must be provided for students’ class preparation. For me, the use of pre-class videos is a necessary element for flipped learning, although it is not the whole story. Merely asking students to read text-based materials on their own before class is not a method of flipping. As one student of Wu et al. (2017) said, “Sometimes I couldn’t get the meanings by reading alone. But the instructional videos helped me understand the overall meaning” (p. 150). Using instructional videos, instructors of flipped classrooms still deliver lectures and explain concepts for their students (Bishop and Verleger 2013). Most importantly, this instructional medium can “closely mimic what students in a traditional setting would experience” (Love et al. 2015, p. 749).
However, a number of researchers have challenged the definition provided by Bishop and Verleger (2013). For example, He et al. (2016) asserted that “qualifying instructional medium is unnecessary and unjustified” (p. 61). During the peer-review process, reviewers have also questioned our systematic reviews and disagreed with the use of this definition. In response to the reviewers’ concern, we added a section discussing our rationale for using the definition by Bishop and Verleger (2013). We also acknowledged that our systematic review “focused specifically on a set of flipped classroom studies in which pre-class instructional videos were provided prior to face-to-face class meetings” (Lo et al. 2017, p. 50). Without a doubt, if instructors insist on “flipping” their courses using pre-class text-based materials only, they will not find our review very useful. Therefore, in addition to explaining the criteria for article selection, future systematic reviews should detail their review scope and acknowledge the limitations of reviewing only a particular set of articles.
4.5 Research Synthesis
The difficulty of the research synthesis is somewhat correlated to the number of studies to be analyzed. My research synthesis of flipped history education (Lo 2017) was not difficult. In this systematic review, I found only five empirical studies at the time of writing (June 2016). I first extracted the data on learning activities, learning outcomes, benefits, and challenges reported in the reviewed studies. These data were then organized and presented in a logical sequence (e.g., from pre-class to in-class). Similarly, Betihavas et al. (2016) also reviewed and identified themes from only five empirical studies of flipped nursing education. They focused on study characteristics, academic performance outcomes, student satisfaction, and challenges in implementing flipped classrooms. With a limited number of studies, Betihavas et al. (2016) were able to discuss the findings of each reviewed study in detail.
In contrast, synthesizing the findings of a large number of studies is challenging and time-consuming. In our systematic review of flipped mathematics education (Lo et al. 2017), we included and analyzed 61 empirical studies. We read through all of the texts, focusing particularly on the results/findings and discussion sections. One of our research objectives was to understand how the flipped classroom approach benefits student learning, and the challenges of flipping mathematics courses. Codes were assigned to pieces of data (i.e., the benefits and challenges reported in the reviewed studies). Thanks to previous efforts in flipped classroom research, we were able to adopt the frameworks by Kuiper et al. (2015) and Betihavas et al. (2016) as our initial analytic frameworks for benefits and challenges, respectively. Despite the large amount of data to be analyzed, these established frameworks made our research synthesis easier.
Thematic analysis of the challenges of flipped mathematics education. (Lo et al. 2017, p. 61)
• Unfamiliarity with flipped learning (n = 26)
• Unpreparedness for pre-class learning tasks (n = 14)
• Unable to ask questions during out-of-class learning (n = 13)
• Unable to understand video content (n = 11)
• Increased workload (n = 9)
• Disengaged from watching videos (n = 3)
• Significant start-up effort (n = 21)
• Not accustomed to flipping (n = 10)
• Ineffectiveness of using others’ videos (n = 4)
• Instructors’ lacking IT skills (n = 3)
• Students’ lacking IT resources (n = 3)
Furthermore, we quantified our thematic analysis by counting the number of studies that contributed to a theme. In this way, our findings could be more specific. Most importantly, such an analysis provided a foundation to develop our design principles to address these challenges. For example, the most-reported student-related challenge was students’ unfamiliarity with flipped learning. Therefore, our Principle 1 was to manage their transition to the flipped classroom. We recommended that instructors introduce students to (1) the rationale for flipped learning, (2) the potential benefits and challenges of this instructional approach, (3) the logistics of their flipped course, and (4) the tasks that students need to do (Lo et al. 2017).
Recommendations for future systematic reviews
Rationale for conducting systematic reviews
• Understand the current state of the literature
• Identify research gaps
Contribution of systematic reviews
• Inform future practice, for instance by establishing design principles or instructional models
• Examine the overall effect of instructional strategies
• Use the asterisk and Boolean operators to increase the flexibility of search strings
• Perform literature search across relevant databases, instead of relying on only a few databases
• Provide strong rationale for inclusion and exclusion criteria
• Acknowledge the limitations resulted from confining the scope of review
• Adopt established frameworks as initial analytic frameworks, especially when a large number of studies are being reviewed
• Quantify the thematic analysis by counting the number of studies that contribute to a theme
This chapter discussed several practical aspects of systematic reviews such as literature search, article selection, and research synthesis. To identify relevant documents, researchers should design more flexible search strings using the asterisk and Boolean operators. Moreover, relevant databases should be consulted in the literature search. Researchers should also provide strong rationales for inclusion and exclusion criteria for article selection. Meanwhile, they should acknowledge any possible limitations of their review scope. For the research synthesis, researchers can adopt established frameworks as initial analytic frameworks. Finally, the thematic analysis can be quantified by counting the number of studies that contribute to a theme. Taking these recommendations into account, the quality of future systematic reviews can be improved.
- Bergmann, J., & Sams, A. (2008). Remixing chemistry class. Learning and Leading with Technology, 36(4), 24–27.Google Scholar
- Bishop, J. L., & Verleger, M. A. (2013). The flipped classroom: A survey of the research. In 120th ASEE national conference and exposition, Atlanta, GA (paper ID 6219). Washington, DC: American Society for Engineering Education.Google Scholar
- Chen, T. L., & Chen, L. (2018). Utilizing wikis and a LINE messaging app in flipped classrooms. EURASIA Journal of Mathematics, Science and Technology Education, 14(3), 1063–1074.Google Scholar
- Hew, K. F., & Lo, C. K. (2018). Flipped classroom improves student learning in health professions education: A meta-analysis. BMC Medical Education, 18(1), article 38.Google Scholar
- Kennedy, E., Beaudrie, B., Ernst, D. C., & St. Laurent, R. (2015). Inverted pedagogy in second semester calculus. PRIMUS, 25(9–10), 892–906.Google Scholar
- Lo, C. K. (2017). Toward a flipped classroom instructional model for history education: A call for research. International Journal of Culture and History, 3(1), 36–43.Google Scholar
- Lo, C. K., & Hew, K. F. (2017). A critical review of flipped classroom challenges in K-12 education: possible solutions and recommendations for future research. Research and Practice in Technology Enhanced Learning, 12(1), article 4.Google Scholar
- Lo, C. K., & Hew, K. F. (2019). The impact of flipped classrooms on student achievement in engineering education: A meta-analysis of 10 years of research. Journal of Engineering Education. Advance online publication. https://doi.org/10.1002/jee.20293.
- Lundin, M., Bergviken Rensfeldt, A., Hillman, T., Lantz-Andersson, A., & Peterson, L. (2018). Higher education dominance and siloed knowledge: A systematic review of flipped classroom research. International Journal of Educational Technology in Higher Education, 15, article 20.Google Scholar
- Tseng, M. F., Lin, C. H., & Chen, H. (2018). An immersive flipped classroom for learning Mandarin Chinese: Design, implementation, and outcomes. Computer Assisted Language Learning, 31(7), 714–733.Google Scholar
- Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In 18th international conference on evaluation and assessment in software engineering, London, England, United Kingdom (article no.: 38). New York, NY: ACM.Google Scholar
- Wu, W. C. V., Chen Hsieh, J. S., & Yang J. C. (2017). Creating an online learning community in a flipped classroom to enhance EFL learners’ oral proficiency. Educational Technology & Society, 20(2), 142–157.Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.