Abstract
Between 2010 and 2019, ETR&D experienced increased publication of a specific type of research that does not provide useful knowledge to the instructional design field. This type of research is research to prove, which entails pitting an incumbent, “traditional” learning experience against a new, innovative learning experience that lacks maturity. Additionally, under closer inspection, these new, innovative learning experiences show significant gaps of good design judgment, in terms of their alignment with the instructional theory framework. This type of research robs the instructional design field of important and useful data associated with effectiveness, efficiency, and appeal outcomes. To provide evidence for our claims, we reviewed 39 ETR&D articles between 1980 and 2019 and 41 articles in non-ETR&D journals between 2009 and 2018 that represented traditional instruction comparisons. Our conclusion is that a change in ETR&D editorial policies around 2010, such as reviewers having more power than editors in determining which papers get published, led to the unintended consequences this paper reports. We provide recommendations for addressing this situation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Acker, S. R., & Klein, E. L. (1986). Visualizing spatial tasks: A comparison of computer graphic and full-band video displays. ECTJ, 34, 21–30. https://doi.org/10.1007/BF02768359.
Al-Samarraie, H., Shamsuddin, A., & Alzahrani, A. I. (2019). A flipped classroom model in higher education: A review of the evidence across disciplines. Educational Technology Research and Development. https://doi.org/10.1007/s11423-019-09718-8.
Al-Zahrani, A. M. (2015). From passive to active: The impact of the flipped classroom through social learning platforms on higher education students’ creative thinking. British Journal of Educational Technology, 46(6), 1133–1148.
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1), 1–14.
Belfi, L. M., Bartolotta, R. J., Giambrone, A. E., Davi, C., & Min, R. J. (2015). “Flipping” the introductory clerkship in radiology: Impact on medical student performance and perceptions. Academic Radiology, 22(6), 794–801.
Blair, E., Maharaj, C., & Primus, S. (2016). Performance and perception in the flipped classroom. Education and Information Technologies, 21(6), 1465–1482.
Boling, E., et al. (2017). Core judgments of instructional designers in practice. Performance Improvement Quarterly, 30(3), 119–219. https://doi.org/10.1002/piq.21250.
Bonneau, C., & Bourdeau, S. (2019). Computer-supported collaboration: Simulation-based training using LEGO®. Educational Technology Research and Development, 67, 1507–1527. https://doi.org/10.1007/s11423-019-09689-w.
Branson, R. K. (1987). Why the schools can’t improve: The upper limit hypothesis. Journal of Instructional Development, 10(4), 15–26.
Briggs, L. J. (1984). Trying to straddle four research cultures. Educational Technology, 24(8), 33–34.
Brooks, A. W. (2014). Information literacy and the flipped classroom: Examining the impact of a one-shot flipped class on student learning and perceptions. Communications in Information Literacy, 8(2), 225–235.
Burbach, M. E., Matkin, G. S., & Fritz, S. M. (2004). Teaching critical thinking in an introductory leadership course utilizing active learning strategies: A confirmatory study. College Student Journal, 38(3), 482–493.
Cabi, E. (2018). The impact of the flipped classroom model on students’ academic achievement. The International Review of Research in Open and Distributed Learning, 19(3), 202–221.
Chang, C. Y., Kao, C. H., Hwang, G. J., et al. (2019). From experiencing to critical thinking: A contextual game-based learning approach to improving nursing students’ performance in Electrocardiogram training. Educational Technology Research and Development, 68, 1225–1245. https://doi.org/10.1007/s11423-019-09723-x.
Chevalier, R. (2003). Updating the behavior engineering model. Performance Improvement, 42(5), 9–14.
Chien, C.-F., & Hsieh, L.-H.C. (2018). Exploring university students’ achievement, motivation, and receptivity of flipped learning in an engineering mathematics course. International Journal of Online Pedagogy and Course Design (IJOPCD), 8(4), 22–37.
Choi, J. I., & Hannafin, M. (1997). The effects of instructional context and reasoning complexity on mathematics problem-solving. ETR&D, 45, 43–55. https://doi.org/10.1007/BF02299728.
Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445–459.
Clark, R. E. (1985). Evidence for confounding in computer-based instruction studies: Analyzing the meta-analyses. Educational Communications and Technology Journal, 33(4), 249–262.
Clark, R. E. (1986). Absolutes and angst in educational technology research: A reply to Don Cunningham. Educational Communications and Technology Journal, 34(1), 8–10.
Clark, R. E. (1989). Current progress and future directions for research in instructional technology. Educational Technology Research and Development, 37(1), 57–66.
Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and Development, 42(2), 21–29.
Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.
Cobb, T. (1997). Cognitive efficiency: Toward a revised theory of media. Educational Technology Research and Development, 45(4), 21–35.
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. The Journal of the Learning Sciences, 13(1), 15–42. https://doi.org/10.1207/s15327809jls1301_2.
Cunningham, D. J. (1986). Good guys and bad guys. Educational Communications and Technology Journal, 34(1), 3–7.
Czeropski, S., & Pembroke, C. (2017). E-learning ain’t performance: Revising HPT in an era of agile and lean. Performance Improvement, 56(8), 37–47. https://doi.org/10.1002/pfi.21728.
Danielson, J. A., Bender, H. S., Mills, E. M., et al. (2003). A tool for helping veterinary students learn diagnostic problem solving. ETR&D, 51, 63–81. https://doi.org/10.1007/BF02504553.
Danker, B. (2015). Using flipped classroom approach to explore deep learning in large classrooms. IAFOR Journal of Education, 3(1), 171–186.
Davies, R. S., Dean, D. L., & Ball, N. (2013). Flipping the classroom and instructional technology integration in a college-level information systems spreadsheet course. Educational Technology Research and Development, 61(4), 563–580.
Demiral-Uzan, M. (2015). Instructional design students’ design judgment in action. Performance Improvement Quarterly, 28(3), 7–23.
Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8.
Driscoll, M. P., & Dick, W. (1999). New research paradigms in instructional technology: An inquiry. Educational Technology Research and Development, 47(2), 7–18.
Efron, S. E., & Ravid, R. (2020). Action research in education: A practical guide. (2nd ed.). The Guilford Press.
Efstathiou, C., Hovardas, T., Xenofontos, N. A., et al. (2018). Providing guidance in virtual lab experimentation: The case of an experiment design tool. Educational Technology Research and Development, 66, 767–791. https://doi.org/10.1007/s11423-018-9576-z.
Eftekhari, M., Sotoudehnama, E., & Marandi, S. S. (2016). Computer-aided argument mapping in an EFL setting: Does technology precede traditional paper and pencil approach in developing critical thinking? Educational Technology Research and Development, 64, 339–357. https://doi.org/10.1007/s11423-016-9431-z.
English, R. E., & Reigeluth, C. M. (1996). Formative research on sequencing instruction with the elaboration theory. Educational Technology Research and Development, 44, 23–42. https://doi.org/10.1007/BF02300324.
Farley, F. H. (1982). The future of educational research. Educational Researcher, 11(8), 11–19.
Fabian, K., Topping, K. J., & Barron, I. G. (2018). Using mobile technologies for mathematics: Effects on student attitudes and achievement. Educational Technology Research and Development, 66, 1119–1139. https://doi.org/10.1007/s11423-018-9580-3.
Fiorella, L., Vogel-Walcutt, J. J., & Schatz, S. (2012). Applying the modality principle to real-time feedback and the acquisition of higher-order cognitive skills. Educational Technology Research and Development, 60, 223–238. https://doi.org/10.1007/s11423-011-9218-1.
Foldnes, N. (2016). The flipped classroom and cooperative learning: Evidence from a randomised experiment. Active Learning in Higher Education, 17(1), 39–49.
Fraga, L. M., & Harmon, J. (2014). The flipped classroom model of learning in higher education: An investigation of preservice teachers’ perspectives and achievement. Journal of Digital Learning in Teacher Education, 31(1), 18–27.
Gibbons, A. S., & Rogers, P. C. (2009). The architecture of instructional theory. In C. M. Reigeluth & A. Carr-Chellman (Eds.), Instructional-design theories and models: Building a common knowledge base. (Vol. III, pp. 305–326). Lawrence Erlbaum Associates.
Gilbert, T. F. (1996). Human competence: Engineering worthy performance. . HRD Press.
Gross, D., Pietri, E. S., Anderson, G., Moyano-Camihort, K., & Graham, M. J. (2015). Increased preclass preparation underlies student outcome improvement in the flipped classroom. CBE-Life Sciences Education. https://doi.org/10.1187/cbe.15-02-0040.
Han, J., Jo, M., Hyun, E., et al. (2015). Examining young children’s perception toward augmented reality-infused dramatic play. Educational Technology Research and Development, 63, 455–474. https://doi.org/10.1007/s11423-015-9374-9.
Hancock-Niemic, M. A., Lin, L., Atkinson, R. K., et al. (2016). Example-based learning: Exploring the use of matrices and problem variability. Educational Technology Research and Development, 64, 115–136. https://doi.org/10.1007/s11423-015-9403-8.
Harrington, S. A., Bosch, M. V., Schoofs, N., Beel-Bates, C., & Anderson, K. (2015). Quantitative outcomes for nursing students in a flipped classroom. Nursing Education Perspectives, 36(3), 179–181.
Honebein, P. C. (1994). The effects of a problem-based learning curriculum for diabetes management and care in a large medical school (Doctoral dissertation). Retrieved from https://search-proquest-com.proxyiub.uits.iu.edu/pqdtglobal/docview/304102555/fulltextPDF/784056FA4C0F49F2PQ/2?accountid=11620. Accessed 17 Dec 2020.
Honebein, P. C. (2018). Specifying human performance solutions through well-formed business requirements. Performance Improvement, 57(7), 45–49. https://doi.org/10.1002/pfi.21788.
Honebein, P. C., & Honebein, C. H. (2015). Effectiveness, efficiency, and appeal: Pick any two? The influence of learning domains and learning outcomes on designer judgments of useful instructional methods. Educational Technology Research and Development, 63(6), 937–955. https://doi.org/10.1007/s11423-015-9396-3.
Honebein, P. C., & Reigeluth, C. M. (2020). The instructional theory framework appears lost. Isn’t it time we find it again? Revista de Educación a Distancia. https://doi.org/10.6018/red.405871.
Honebein, P., & Reigeluth, C. M. (2021). Making good design judgments via the instructional theory framework. In J. K. McDonald & R. E. West (Eds.), Design for learning: Principles, processes, and praxis. EdTech Books.
Hotle, S. L., & Garrow, L. A. (2015). Effects of the traditional and flipped classrooms on undergraduate student opinions and success. Journal of Professional Issues in Engineering Education and Practice, 142(1), 05015005.
Huang, Y. M., & Huang, Y. M. (2015). A scaffolding strategy to develop handheld sensor-based vocabulary games for improving students’ learning motivation and performance. Educational Technology Research and Development, 63, 691–708. https://doi.org/10.1007/s11423-015-9382-9.
Huang, Y. M., Shadiev, R., Sun, A., et al. (2017). A study of the cognitive diffusion model: Facilitating students’ high level cognitive processes with authentic support. Educational Technology Research and Development, 65, 505–531. https://doi.org/10.1007/s11423-016-9475-0.
Hung, H.-T. (2015). Flipping the classroom for English language learners to foster active learning. Computer Assisted Language Learning, 28(1), 81–96.
Hwang, G. J., Chang, S. C., Chen, P. Y., et al. (2018). Effects of integrating an active learning-promoting mechanism into location-based real-world learning environments on students’ learning performances and behaviors. Educational Technology Research and Development, 66, 451–474. https://doi.org/10.1007/s11423-017-9567-5.
Hwang, G. J., Hung, C. M., & Chen, N. S. (2014). Improving learning achievements, motivations and problem-solving skills through a peer assessment-based game development approach. Educational Technology Research and Development, 62, 129–145. https://doi.org/10.1007/s11423-013-9320-7.
Hwang, W. Y., Zhao, L., Shadiev, R., et al. (2019). Exploring the effects of ubiquitous geometry learning in real situations. Educational Technology Research and Development, 68, 1121–1147. https://doi.org/10.1007/s11423-019-09730-y.
Jayasinghe, M. G., Morrison, G. R., & Ross, S. M. (1997). The effect of distance learning classroom design on student perceptions. ETR&D, 45, 5–19. https://doi.org/10.1007/BF02299680.
Jensen, J. L., Kummer, T. A., & Godoy, P. D. M. (2015). Improvements from a flipped classroom may simply be the fruits of active learning. CBE-Life Sciences Education. https://doi.org/10.1187/cbe.14-08-0129.
Jonassen, D. H., Campbell, J. P., & Davidson, M. E. (1994). Learning with media: Restructuring the debate. Educational Technology Research and Development, 42(2), 31–39. https://doi.org/10.1007/BF02299089.
Jungić, V., Kaur, H., Mulholland, J., & Xin, C. (2015). On flipping the classroom in large first year calculus courses. International Journal of Mathematical Education in Science and Technology, 46(4), 508–520.
Kakos-Kraft, S., Honebein, P. C., Prince, M. J., & Marrero, D. G. (1997). The SOCRATES curriculum: An innovative integration of technology and theory in medical education. Journal of Audiovisual Media in Medicine, 20(4), 166–171. https://doi.org/10.3109/17453059709063100.
Ke, F. (2008). Computer games application within alternative classroom goal structures: Cognitive, metacognitive, and affective evaluation. Educational Technology Research and Development, 56, 539–556. https://doi.org/10.1007/s11423-008-9086-5.
Koo, C. L., Demps, E. L., Farris, C., Bowman, J. D., Panahi, L., & Boyle, P. (2016). Impact of flipped classroom design on student performance and perceptions in a pharmacotherapy course. American Journal of Pharmaceutical Education, 80(2), 33.
Kozma, R. B. (1994). Will media influence learning? Reframing the debate. Educational Technology Research and Development, 42, 7–19. https://doi.org/10.1007/BF02299087.
Kozma, R. (2000). Reflections on the state of educational technology research and development. Educational Technology Research and Development, 48(1), 5–15.
Kuo, M. L. A., & Hooper, S. (2004). The effects of visual and verbal coding mnemonics on learning Chinese characters in computer-based instruction. Educational Technology Research and Development, 52(3), 23–38.
Lan, Y. J., Fang, S. Y., Legault, J., et al. (2015). Second language acquisition of Mandarin Chinese vocabulary: Context of learning effects. Educational Technology Research and Development, 63, 671–690. https://doi.org/10.1007/s11423-015-9380-y.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
Lee, G., & Wallace, A. (2018). Flipped learning in the english as a foreign language classroom: Outcomes and perceptions. TESOL Quarterly, 52(1), 62–84.
Lee, J., & Reigeluth, C. M. (2003). Formative research on the heuristic task analysis process. Educational Technology Research and Development, 51, 5–17. https://doi.org/10.1007/BF02504541.
Lee, Y. M., Jahnke, I., & Austin, L. (2021). Mobile microlearning design and effects on learning efficacy and learner experience. Educational Technology Research and Development. https://doi.org/10.1007/s11423-020-09931-w.
Lee, V. R., & Thomas, J. M. (2011). Integrating physical activity data technologies into elementary school classrooms. Educational Technology Research and Development, 59, 865–884. https://doi.org/10.1007/s11423-011-9210-9.
LeMahieu, P. G., Edwards, A. R., & Gomez, L. M. (2015). At the nexus of improvement science and teaching: Introduction to a special section of the Journal of Teacher Education. Journal of Teacher Education, 66(5), 446–449.
Lewis, C. (2015). What is improvement science? Do we need it in education? Educational Researcher, 44(1), 54–61.
Liebert, C. A., Lin, D. T., Mazer, L. M., Bereknyei, S., & Lau, J. N. (2016). Effectiveness of the surgery core clerkship flipped classroom: A prospective cohort trial. The American Journal of Surgery, 211(2), 451.e1-457.e1.
Liefeld, J. P., & Herrmann, T. F. (1990). Learning consequences for university students using computerized mastery testing. ETR&D, 38, 19–25. https://doi.org/10.1007/BF02298266.
Lindsey, L., & Berger, N. (2009). Experiential approach to instructions. In C. M. Reigeluth & A. Carr-Chellman (Eds.), Instructional-design theories and models: Building a common knowledge base. (Vol. III, pp. 117–142). Lawrence Erlbaum Associates.
Lin-Siegler, X., Shaenfield, D., & Elder, A. D. (2015). Contrasting case instruction can improve self-assessment of writing. Educational Technology Research and Development, 63, 517–537. https://doi.org/10.1007/s11423-015-9390-9.
Liou, W. K., Bhagat, K. K., & Chang, C. Y. (2018). The design, implementation, and evaluation of a digital interactive globe system integrated into an earth science course. Educational Technology Research and Development, 66, 545–561. https://doi.org/10.1007/s11423-018-9573-2.
Lockee, B. B., Burton, J. K., & Cross, L. H. (1999). No comparison: Distance education finds a new use for ‘no significant difference.’ Educational Technology Research and Development, 47(3), 33–42.
Lubin, I. A., & Ge, X. (2012). Investigating the influences of a LEAPS model on preservice teachers’ problem solving, metacognition, and motivation in an educational technology course. Educational Technology Research and Development, 60, 239–270. https://doi.org/10.1007/s11423-011-9224-3.
Mager, R. F. (1984). Preparing instructional objectives. . Lake Publishing.
Malinverni, L., Schaper, M. M., & Pares, N. (2016). An evaluation-driven design approach to develop learning environments based on full-body interaction. Educational Technology Research and Development, 64, 1337–1360. https://doi.org/10.1007/s11423-016-9468-z.
Mason, G. S., Shuman, T. R., & Cook, K. E. (2013). Comparing the effectiveness of an inverted classroom to a traditional classroom in an upper-division engineering course. IEEE Transactions on Education, 56(4), 430–435.
Mattis, K. V. (2015). Flipped classroom versus traditional textbook instruction: Assessing accuracy and mental effort at different levels of mathematical complexity. Technology, Knowledge and Learning, 20(2), 231–248.
Mavridis, A., Katmada, A., & Tsiatsos, T. (2017). Impact of online flexible games on students’ attitude towards mathematics. Educational Technology Research and Development, 65, 1451–1470. https://doi.org/10.1007/s11423-017-9522-5.
McCallum, S., Schultz, J., Sellke, K., & Spartz, J. (2015). An examination of the flipped classroom approach on college student academic involvement. International Journal of Teaching and Learning in Higher Education, 27(1), 42–55.
McLaughlin, J. E., Griffin, L. M., Esserman, D. A., Davidson, C. A., Glatt, D. M., Roth, M. T., & Mumper, R. J. (2013). Pharmacy student engagement, performance, and perception in a flipped satellite classroom. American Journal of Pharmaceutical Education, 77(9), 196.
McLaughlin, J. E., Roth, M. T., Glatt, D. M., Gharkholonarehe, N., Davidson, C. A., Griffin, L. M., & Mumper, R. J. (2014). The flipped classroom: A course redesign to foster learning and engagement in a health professions school. Academic Medicine, 89(2), 236–243.
Merrill, M. D. (1983). Component display theory. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status. (pp. 279–333). Lawrence Erlbaum Associates.
Missildine, K., Fountain, R., Summers, L., & Gosselin, K. (2013). Flipping the classroom to improve student performance and satisfaction. Journal of Nursing Education, 52(10), 597–599.
Moffett, J., & Mill, A. C. (2014). Evaluation of the flipped classroom approach in a veterinary professional skills course. Advances in Medical Education and Practice, 5, 415.
Morrison, G. R. (1994). The media effects question: “Unresolvable” or asking the right question. Educational Technology Research and Development, 42, 41–44. https://doi.org/10.1007/BF02299090.
Morrison, G. R., Ross, S. M., & O’Dell, J. K. (1988). Text density level as a design variable in instructional displays. ECTJ, 36, 103–115. https://doi.org/10.1007/BF02766618.
Murray, L., McCallum, C., & Petrosino, C. (2014). Flipping the classroom experience: A comparison of online learning to traditional lecture. Journal of Physical Therapy Education, 28(3), 35–41.
Nelson, H. G., & Stolterman, E. (2012). The design way: Intentional change in an unpredictable world. (2nd ed.). MIT Press.
Ojennus, D. D. (2016). Assessment of learning gains in a flipped biochemistry classroom. Biochemistry and Molecular Biology Education, 44(1), 20–27.
Parkhurst, P. E. (1992). Criticisms of a report on teaching with interactive videodiscs. Academic Medicine, 67(10), 657–658.
Peterson, D. J. (2016). The flipped classroom improves student achievement and course satisfaction in a statistics course: A quasi-experimental study. Teaching of Psychology, 43(1), 10–15.
Phillips, R., Kennedy, G., & McNaught, C. (2012). The role of theory in learning technology and evaluation research. Australasian Journal of Educational Technology, 28(7), 1103–1118.
Pierce, R., & Fox, J. (2012). Vodcasts and active-learning exercises in a “flipped classroom” model of a renal pharmacotherapy module. American Journal of Pharmaceutical Education, 76(10), 196.
Ponce, H. R., Mayer, R. E., & Lopez, M. J. (2013). A computer-based spatial learning strategy approach that improves reading comprehension and writing. Educational Technology Research and Development, 61, 819–840. https://doi.org/10.1007/s11423-013-9310-9.
Porcaro, P. A., Jackson, D. E., McLaughlin, P. M., & O’Malley, C. J. (2016). Curriculum design of a flipped classroom to enhance haematology learning. Journal of Science Education and Technology, 25(3), 345–357.
Prashar, A. (2015). Assessing the flipped classroom in operations management: A pilot study. Journal of Education for Business, 90(3), 126–138.
Proske, A., Roscoe, R. D., & McNamara, D. S. (2014). Game-based practice versus traditional practice in computer-based writing strategy training: Effects on motivation and achievement. Educational Technology Research and Development, 62, 481–505. https://doi.org/10.1007/s11423-014-9349-2.
Raja, F. U., & Najmonnisa. (2018). Comparing traditional teaching method and experiential teaching method using experimental research. Journal of Education and Educational Development, 5(2), 276–288.
Reeves, T. C., & Oh, E. (2017). The goals and methods of educational technology research over a quarter century (1989–2014). Educational Technology Research and Development, 65(2), 325–339.
Reigeluth, C. M. (1983). Instructional design: What is it and why is it? In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status. (pp. 3–36). Lawrence Erlbaum Associates.
Reigeluth, C. M. (1989). Educational technology at the crossroads: New mindsets and new directions. Educational Technology Research and Development, 37(1), 67–80.
Reigeluth, C. M. (1999). What is instructional-design theory and how is it changing? In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory. (Vol. II, pp. 5–29). Lawrence Erlbaum Associates.
Reigeluth, C. M., & An, Y. (2009). Theory building. In C. M. Reigeluth & A. Carr-Chellman (Eds.), Instructional-design theories and models: Building a common knowledge base. (Vol. III, pp. 365–386). Lawrence Erlbaum Associates.
Reigeluth, C. M., & An, Y. (2021). Merging the instructional design process with learner-centered theory: The holistic 4D model. . Routledge.
Reigeluth, C. M., & Carr-Chellman, A. (2009). Understanding instructional theory. In C. M. Reigeluth & A. Carr-Chellman (Eds.), Instructional-design theories and models: Building a common knowledge base. (Vol. III, pp. 3–26). Lawrence Erlbaum Associates.
Reigeluth, C. M., & Frick, T. W. (1999). Formative research: A methodology for creating and improving design theories. In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory. (Vol. II, pp. 633–651). Lawrence Erlbaum Associates.
Reigeluth, C. M., & Schwartz, E. (1989). An instructional theory for the design of computer-based simulations. Journal of Computer-Based Instruction, 16(1), 1–10.
Reiser, R. A. (1994). Clark’s invitation to the dance: An instructional designer’s response. Educational Technology Research and Development, 42, 45–48. https://doi.org/10.1007/BF02299091.
Richey, R. (1998). The pursuit of useable knowledge in instructional technology. Educational Technology Research and Development, 46(4), 7–22.
Ritzhaupt, A., & Sommer, M. (2018). Impact of the flipped classroom on learner achievement and satisfaction in an undergraduate technology literacy course. Journal of Information Technology Education: Research, 17(1), 159–182.
Rogers, E. M. (2003). Diffusion of innovations. (5th ed.). Free Press.
Ronimus, M., Eklund, K., Pesu, L., et al. (2019). Supporting struggling readers with digital game-based learning. Educational Technology Research and Development, 67, 639–663. https://doi.org/10.1007/s11423-019-09658-3.
Ross, S. M. (1994). Delivery trucks or groceries? More food for thought on whether media (will, may, can’t) influence learning. Educational Technology Research and Development, 42, 5–6. https://doi.org/10.1007/BF02299086.
Ross, S. M., & Anand, P. G. (1987). A computer-based strategy for personalizing verbal problems in teaching mathematics. ECTJ, 35, 151–162. https://doi.org/10.1007/BF02793843.
Ryan, M. D., & Reid, S. A. (2015). Impact of the flipped classroom on student performance and retention: A parallel controlled study in general chemistry. Journal of Chemical Education, 93(1), 13–23.
Sáez-López, J. M., Sevillano-García, M. L., & Vazquez-Cano, E. (2019). The effect of programming on primary school students’ mathematical and scientific understanding: Educational use of mBot. Educational Technology Research and Development, 67, 1405–1425. https://doi.org/10.1007/s11423-019-09648-5.
Salomon, G. (1991). Transcending the qualitative-quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher, 20(6), 10–18.
Schweickert, R., & Boruff, B. (1986). Short-term memory capacity: Magic number or magic spell? Journal of Experimental Psychology: Learning, Memory, and Cognition, 12(3), 419–425. https://doi.org/10.1037/0278-7393.12.3.419.
Shadiev, R., Hwang, W. Y., & Liu, T. Y. (2018). Investigating the effectiveness of a learning activity supported by a mobile multimedia learning system to enhance autonomous EFL learning in authentic contexts. Educational Technology Research and Development, 66, 893–912. https://doi.org/10.1007/s11423-018-9590-1.
Shrock, S. A. (1994). The media influence debate: Read the fine print, but don’t lose sight of the big picture. Educational Technology Research and Development, 42, 49–53. https://doi.org/10.1007/BF02299092.
Simpson, V., & Richards, E. (2015). Flipping the classroom to teach population health: Increasing the relevance. Nurse Education in Practice, 15(3), 162–167.
Smith, K. M., & Boling, E. (2009). What do we make of design? Design as a concept in educational technology. Educational Technology, 49(4), 3–17.
Sohrabi, B., & Iraj, H. (2016). Implementing flipped classroom using digital media: A comparison of two demographically different groups perceptions. Computers in Human Behavior, 60, 514–524.
Spector, J. M. (2017). Reflections on educational technology research and development. Educational Technology Research and Development, 66(4), 893–912.
Staker, H., & Horn, M. B. (2012). Classifying K-12 blended learning. Retrieved from https://www.christenseninstitute.org/wp-content/uploads/2013/04/Classifying-K-12-blended-learning.pdf. Accessed 17 Dec 2020.
Stebbins, R. A. (2001). Exploratory research in the social sciences. . Sage.
Stokes, D. E. (1997). Pasteur’s quadrant: Basic science and technological innovation. . The Brookings Institution.
Stolterman, E., & Nelson, H. (2000). The guarantor of design. In L. Svensson, U. Snis, C. Sørensen, H. Fägerlind, T. Lindroth, M. Magnusson, & C. Östlund (Eds.), Proceedings of IRIS 23. Laboratorium for Interaction Technology, University of Trollhättan Uddevalla.
Strayer, J. F. (2012). How learning in an inverted classroom influences cooperation, innovation and task orientation. Learning Environments Research, 15(2), 171–193.
Stringer, E. T. (2008). Action research in education. (2nd ed.). Pearson Prentice Hall.
Stringer, E. T., & Aragon, A. O. (2021). Action research. (5th ed.). Sage Publications.
Tanner, D. (1998). The social consequences of bad research. Phi Delta Kappan, 79(5), 345–349.
Tanner, M., & Scott, E. (2015). A flipped classroom approach to teaching systems analysis, design and implementation. Journal of Information Technology Education: Research, 14, 219–241.
Tennyson, R. D. (1994). The big wrench vs. integrated approaches: The great media debate. Educational Technology Research and Development, 42(3), 15–28.
Tessmer, M., & Driscoll, M. P. (1986). Effects of a diagrammatic display of coordinate concept definitions on concept classification performance. ECTJ, 34, 195–205. https://doi.org/10.1007/BF02767401.
The Performance Thinking Network. (2012). Six boxes model. Retrieved from https://www.sixboxes.com/Six-Boxes-Model.html. Accessed 17 Dec 2020.
Tune, J. D., Sturek, M., & Basile, D. P. (2013). Flipped classroom model improves graduate student performance in cardiovascular, respiratory, and renal physiology. Advances in Physiology Education, 37(4), 316–320.
Ullmer, E. J. (1994). Media and learning: Are there two kinds of truth? Educational Technology Research and Development, 42(1), 21–32.
Uribe, D., Klein, J. D., & Sullivan, H. (2003). The effect of computer-mediated collaborative learning on solving III-defined problems. ETR&D, 51, 5–19. https://doi.org/10.1007/BF02504514.
Van Eck, R., & Dempsey, J. (2002). The effect of competition and contextualized advisement on the transfer of mathematics skills a computer-based instructional simulation game. Educational Technology Research and Development, 50, 23–41. https://doi.org/10.1007/BF02505023.
Van Vliet, E., Winnips, J., & Brouwer, N. (2015). Flipped-class pedagogy enhances student metacognition and collaborative-learning strategies in higher education but effect does not persist. CBE-Life Sciences Education. https://doi.org/10.1187/cbe.14-09-0141.
Velegol, S. B., Zappe, S. E., & Mahoney, E. (2015). The evolution of a flipped classroom: Evidence-based recommendations. Advances in Engineering Education, 4(3), 1–37.
Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53, 5–23. https://doi.org/10.1007/BF02504682.
Whillier, S., & Lystad, R. P. (2015). No differences in grades or level of satisfaction in a flipped classroom for neuroanatomy. Journal of Chiropractic Education, 29(2), 127–133.
Wong, T. H., Ip, E. J., Lopes, I., & Rajagopalan, V. (2014). Pharmacy students’ performance and perceptions in a flipped teaching pilot on cardiac arrhythmias. American Journal of Pharmaceutical Education, 78(10), 185.
Wilson, S. G. (2013). The flipped class: A method to address the challenges of an undergraduate statistics course. Teaching of Psychology, 40(3), 193–199.
Yeh, Y. L., & Lan, Y. J. (2018). Fostering student autonomy in English learning through creations in a 3D virtual world. Educational Technology Research and Development, 66, 693–708. https://doi.org/10.1007/s11423-017-9566-6.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1
Coding categories for ETR&D articles between 1980 and 2019
Element/column | Criteria |
|---|---|
Order number | Sequential order in which the article was collected and reviewed |
Date | Year of publication |
Document | File name of the article’s PDF |
Region | Region where the study was conducted or where first author is based |
Subjects | Description of the subjects for the study |
Research questions/hypotheses | Indicates presence of research questions, hypothesis, objective, derived, or some mixture of these four concepts. Cell comment provides data from the paper. “Derived” means that no specific research questions or hypotheses were specified, but could be derived from statements made in the manuscript |
Classification | Classifies studies based upon the available data: 1 = Research to prove—for descriptive theory 2 = Research to prove—for design theory 3 = Research to improve 4 = Research to describe 5 = Research to test feasibility |
Instructional theory/model | The type instructional theory/model described in the literature section of the paper (macro, meso, micro) |
Traditional method | The primary instructional method employed by the perceived inferior solution |
Traditional media | The primary media method employed by the perceived inferior solution |
New method | The primary instructional method employed by the perceived superior solution |
New media | The primary media method employed by the perceived superior solution |
Independent variable | The primary focus of the independent variable: method, media, or mixed |
Other method | Other instructional methods received by all participants |
Treatments | Number of treatments in the study |
Research method | As specified by author or implied by researcher: experimental, quasi-experimental, case study, etc |
Task | Description of the primary task of the learner |
Conditions | Yes or no depending upon whether the author provided at least one condition, signaled by words such as “given,” “using,” or similar |
Behavior | Yes or no depending upon if the author specified at least one logical statement that includes an observable action verb followed by a behavioral statement |
Criterion specification | Yes or no or partial depending upon a specification of a quantified, target criterion for at least one of effectiveness, efficiency, or appeal |
Formative evaluation | Yes or no or partial depending upon whether an instructional design was pilot tested prior to the running of the experiment. Search terms: formative, pilot |
Effectiveness | Yes or No for specific measure present in the Method section for assessing the effectiveness of the instructional designs |
Efficiency | Yes or No for specific measure present in the Method section for assessing the efficiency of the instructional designs |
Appeal | Yes or no for specific measure present in the Method section for assessing appeal (liking, satisfaction, motivation) of the instructional designs |
Other measures | Other measures beyond effectiveness, efficiency, and appeal |
Result | Identification of the treatment that performed “best” |
Source | Link to the research paper |
Title | Title of the research paper |
Appendix 2
Study 1 articles—from ETR&D 1980–2019
Author | Region | Research type |
|---|---|---|
Tessmer and Driscoll (1986) | North America | Research to prove for design |
Acker and Klein (1986) | North America | Research to prove for design |
Ross and Anand (1987) | North America | Research to prove for design |
Morrison et al. (1988) | North America | Research to prove for design |
Liefeld and Herrmann (1990) | North America | Research to prove for design |
Jayasinghe et al. (1997) | North America | Research to prove for design |
Choi and Hannafin (1997) | Asia | Research to prove for design |
Van Eck and Dempsey (2002) | North America | Research to prove for design |
Uribe et al. (2003) | North America | Research to prove for design |
Danielson et al. (2003) | North America | Research to prove for design |
Kuo and Hooper (2004) | North America | Research to prove for design |
Ke (2008) | North America | Research to prove for design |
Lee and Thomas (2011) | North America | Research to test feasibility |
Lubin and Ge (2012) | North America | Research to describe |
Fiorella et al. (2012) | North America | Research to prove for design |
Ponce et al. (2013) | South America | Research to prove for design |
Davies et al. (2013) | North America | Research to prove for design |
Hwang et al. (2014) | Asia | Research to prove for design |
Proske et al. (2014) | Europe | Research to prove for design |
Lin-Siegler et al. (2015) | North America | Research to prove for design |
Huang and Huang (2015) | Asia | Research to prove for design |
Lan et al. (2015) | Asia | Research to prove for design |
Han et al. (2015) | Asia | Research to prove for design |
Malinverni et al. (2016) | Europe | Research to prove for design |
Eftekhari et al. (2016) | Middle East | Research to prove for design |
Hancock-Niemic et al. (2016) | North America | Research to prove for design |
Hwang et al. (2018) | Asia | Research to prove for design |
Mavridis et al. (2017) | Europe | Research to prove for design |
Huang et al. (2017) | Asia | Research to prove for design |
Fabian et al. (2018) | Europe | Research to prove for design |
Shadiev et al. (2018) | Asia | Research to prove for design |
Efstathiou et al. (2018) | Europe | Research to prove for design |
Liou et al. (2018) | Asia | Research to prove for design |
Yeh and Lan (2018) | Asia | Research to describe |
Chang et al. (2019) | Asia | Research to prove for design |
Ronimus et al. (2019) | Europe | Research to prove for design |
Hwang et al. (2019) | Asia | Research to prove for design |
Sáez-López et al. (2019) | Europe | Research to prove for design |
Bonneau and Bourdeau (2019) | North America | Research to test feasibility |
Appendix 3
Study 2 articles – see Al-Samarraie et al. (2019)
Author | Region | Discipline | Research type |
|---|---|---|---|
Pierce and Fox (2012) | North America | Health | Research to prove for design |
Strayer (2012) | North America | Mathematics | Research to prove for design |
Tune et al. (2013) | North America | Health | Research to prove for design |
Wilson (2013) | North America | Mathematics | Research to prove for design |
McLaughlin et al. (2013) | North America | Health | Research to prove for design |
Missildine et al. (2013) | North America | Health | Research to prove for design |
Mason et al. (2013) | North America | Engineering | Research to prove for design |
McLaughlin et al. (2014) | North America | Health | Research to prove for design |
Wong et al. (2014) | North America | Health | Research to prove for design |
Brooks (2014) | North America | Social science | Research to prove for design |
Murray et al. (2014) | North America | Health | Research to prove for design |
Moffett and Mill (2014) | Europe | Health | Research to prove for design |
Whillier and Lystad (2015) | Oceania | Health | Research to prove for design |
Harrington et al. (2015) | North America | Health | Research to prove for design |
McCallum et al. (2015) | North America | Mathematics | Research to describe |
Prashar (2015) | Asia | Social science | Research to prove for design |
Ryan and Reid (2015) | North America | Natural sciences | Research to prove for design |
Velegol et al. (2015) | North America | Engineering | Research to improve |
Gross et al. (2015) | North America | Natural sciences | Research to prove for design |
Tanner and Scott (2015) | Africa | Social science | Research to improve |
Mattis (2015) | North America | Mathematics | Research to prove for design |
Van Vliet et al. (2015) | Europe | Natural sciences | Research to prove for design |
Jensen et al. (2015) | North America | Natural sciences | Research to prove for design |
Hotle and Garrow (2015) | North America | Engineering | Research to prove for design |
Al-Zahrani (2015) | Middle East | Education | Research to improve |
Hung (2015) | Asia | Social science | Research to prove for design |
Jungić et al. (2015) | North America | Mathematics | Research to describe |
Danker (2015) | Asia | Arts | Research to describe |
Belfi et al. (2015) | North America | Health | Research to prove for design |
Porcaro et al. (2016) | Oceania | Health | Research to prove for design |
Koo et al. (2016) | North America | Health | Research to prove for design |
Peterson (2016) | North America | Mathematics | Research to prove for design |
Foldnes (2016) | Europe | Social science | Research to prove for design |
Blair et al. (2016) | South America | Engineering | Research to prove for design |
Liebert et al. (2016) | North America | Health | Research to prove for design |
Ojennus (2016) | North America | Natural sciences | Research to prove for design |
Sohrabi and Iraj (2016) | Middle East | Social science | Research to describe |
Chien and Hsieh (2018) | Asia | Social science | Research to prove for design |
Cabi (2018) | Europe | Social science | Research to prove for design |
Lee and Wallace (2018) | Asia | Social science | Research to prove for design |
Ritzhaupt and Sommer (2018) | North America | Education | Research to prove for design |
Rights and permissions
About this article
Cite this article
Honebein, P.C., Reigeluth, C.M. To prove or improve, that is the question: the resurgence of comparative, confounded research between 2010 and 2019. Education Tech Research Dev 69, 465–496 (2021). https://doi.org/10.1007/s11423-021-09988-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-021-09988-1