What works may hurt: Side effects in education
Medical research is held as a field for education to emulate. Education researchers have been urged to adopt randomized controlled trials, a more “scientific” research method believed to have resulted in the advances in medicine. But a much more important lesson education needs to borrow from medicine has been ignored. That is the study of side effects. Medical research is required to investigate both the intended effects of any medical interventions and their unintended adverse effects, or side effects. In contrast, educational research tends to focus only on proving the effectiveness of practices and policies in pursuit of “what works.” It has generally ignored the potential harms that can result from what works. This article presents evidence that shows side effects are inseparable from effects. Both are the outcomes of the same intervention. This article further argues that studying and reporting side effects as part of studying effects will help advance education by settling long fought battles over practices and policies and move beyond the vicious cycle of pendulum swings in education.
KeywordsEducational research Methodology RCT Direct instruction International assessment Side effects PISA Educational policy Educational reform
Ken Frank of Michigan State University, James Basham and Jason Travers of University of Kansas, and Yechen Zhao of Stanford University read drafts of the manuscript and provided invaluable suggestions.
- Adams, G. L., & Engelmann, S. (1996). Research on direct instruction: 25 years beyond DISTAR. Seattle, WA: Educational Achievement Systems.Google Scholar
- Barker, B. (2010). The pendulum swings: Transforming school reform. Sterling, VA: Trentham Books.Google Scholar
- Brown, E. (2016, April 27). U.S. high school seniors slip in math and show no improvement in reading. The Washington Post. Retrieved from https://www.washingtonpost.com/local/education/us-high-school-seniors-slip-in-math-and-show-no-improvement-in-reading/2016/04/26/9b8f033e-0bc8-11e6-8ab8-9ad050f76d7d_story.html?utm_term=.7a9d458243cd.
- Bryk, A. S. (2015). Learning to improve: How America’s schools can get better at getting better. Cambridge, MA: Harvard Education Press.Google Scholar
- Campbell, D. T. (1976). Assessing the impact of planned social change. Hanover New Hampshire, USA. Retrieved from https://www.globalhivmeinfo.org/CapacityBuilding/OccasionalPapers/08AssessingtheImpactofPlannedSocialChange.pdf.
- Common Core State Standards Initiative. (2011). Common core state standards initiative. Retrieved from http://www.corestandards.org/.
- Darling-Hammond, L., & Lieberman, A. (Eds.). (2012). Teacher education around the world. New York: Routledge.Google Scholar
- Figazzolo, L. (2009). Impact of PISA 2006 on the education policy debate. Retrieved from http://download.ei-ie.org/docs/IRISDocuments/ResearchWebsiteDocuments/2009-00036-01-E.pdf.
- Ginsberg, R., & Kingston, N. (2014). Caught in a vise: The challenges facing teacher preparation in an era of accountability. Teachers College Record, 116(1), n1.Google Scholar
- Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London, New York: Routledge.Google Scholar
- Hempenstall, K. (2012, November 11). Reviews supporting direct instruction program effectiveness. Retrieved from https://www.nifdi.org/news/hempenstall-blog/403-reviews-supporting-direct-instruction-program-effectiveness.
- Hempenstall, K. (2013, October 10). Why does direct instruction evoke such rancour? Retrieved from https://www.nifdi.org/news-latest-2/blog-hempenstall/389-why-does-direct-instruction-evoke-such-rancour.
- Hess, F. M. (2011). Our achievement-gap mania. National Affairs, Fall 2011 (Number 9), 113–129.Google Scholar
- Hout, M., & Elliott, S. W. (Eds.). (2011). Incentives and test-based accountability in education. Washington, DC: National Academies Press.Google Scholar
- Jensen, B. (2012). Catching up: Learning from the best school systems in East Asia. Retrieved from Melbourne: http://www.grattan.edu.au/publications/129_report_learning_from_the_best_main.pdf.
- Kaestle, C. (1985). Education reform and the swinging pendulum. Phi Delta Kappan, 66(6), 422–423.Google Scholar
- Kaplan, K. (2016, December 9). Cooling cap helps cancer patients preserve their hair during chemotherapy, clinical trial shows. The Los Angeles Times. Retrieved from http://www.latimes.com/science/sciencenow/la-sci-sn-cooling-scalp-chemotherapy-20161209-story.html.
- Kreiner, S., & Christensen, K. B. (2014). Analyses of model fit and robustness. A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika. doi: 10.1007/s11336-013-9347-z.
- Loveless, T. (2006). How well are American students learning? Retrieved from Washington, DC: http://www.brookings.edu/~/media/Files/rc/reports/2006/10education_loveless/10education_loveless.pdf.
- Loveless, T. (2014). PISA’s China problem continues: A response to Schleicher, Zhang, and Tucker. Retrieved from http://www.brookings.edu/research/papers/2014/01/08-shanghai-pisa-loveless.
- McMurrer, J. (2007). Choices, changes, and challenges: Curriculum and instruction in the NCLB era. Retrieved from Washington, DC: http://www.cep-dc.org/displayDocument.cfm?DocumentID=312.
- Morrison, H. (2013, December 1). Pisa 2012 major flaw exposed. Retrieved from https://paceni.wordpress.com/2013/12/01/pisa-2012-major-flaw-exposed/.
- Mullis, I. V. S., Martin, M. O., & Foy, P. (with Olson, J.F., Preuschoff, C., Erberber, E., Arora, A., & Galia, J.). (2008). TIMSS 2007 international mathematics report: Findings from IEA’s trends in international mathematics and science study at the fourth and eighth grades. Retrieved from Chestnut Hill, MA.Google Scholar
- Mullis, I. V. S., Martin, M. O., Foy, P., & Arora, A. (2012). TIMSS 2011 international results in mathematics. Retrieved from Chestnut Hill, MA.Google Scholar
- National Institute for Direct Instruction. (2015). Writings on direct instruction: A bibliography. Retrieved from Eugene, OR.Google Scholar
- National Research Council. (1999). Global perspectives for local action: Using TIMSS to improve U.S. mathematics and science education. Washington, D.C.: National Academy Press.Google Scholar
- Nichols, S. L., & Berliner, D. C. (2007). Collateral damage: How high-stakes testing corrupts America’s schools. Cambridge, MA: Harvard Education Press.Google Scholar
- No Child Left Behind Act of 2001, 107–110, Congress (2002).Google Scholar
- OECD. (2011). Strong performers and successful reformers in education: Lessons from PISA for the United States. Retrieved from Paris: http://dx.doi.org/10.1787/9789264096660-en.
- OECD. (2013a). Programme for international student assessment (PISA). Retrieved from http://www.oecd.org/pisa/aboutpisa/.
- OECD. (2013b). Ready to learn: Students engagement, drive, and self-beliefs. Retrieved from Paris: http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-III.pdf.
- OECD. (2014). PISA 2012 results: What students know and can do: Student performance in mathematics, reading and science (Volume I) [Revised edition February 2014]. Retrieved from Paris: http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-i.htm.
- OECD. (2016). PISA 2015 results (volume I): Excellence and equity in education. Retrieved from Paris: http://dx.doi.org/10.1787/9789264266490-en.
- Peterson, P. L. (1979). Direct instruction: Effective for what and for whom. Educational Leadership, 37(1), 46–48.Google Scholar
- Ravitch, D. (2013). Reign of error: The hoax of the privatization movement and the danger to America’s public schools. New York: Knopf.Google Scholar
- Roehler, L. R., & Duffy, G. G. (1982). Matching direct instruction to reading outcomes. Language Arts, 59(5), 476–480.Google Scholar
- Rosenshine, B. (2008). Five meanings of direct instruction. Retrieved from Lincoln, IL: http://www.centerii.org/search/Resources%5CFiveDirectInstruct.pdf.
- Sahlberg, P. (2011). Finnish lessons: What can the world learn from educational change in Finland?. New York: Teachers College Press.Google Scholar
- Sanchez, C. (Writer). (2013). El Paso schools cheating scandal: Who’s accountable? Washington DC: NPR.Google Scholar
- Schleicher, A. (2013, December 3). What we learn from the PISA 2012 results. Retrieved from http://oecdeducationtoday.blogspot.com/2013/12/what-we-learn-from-pisa-2012-results.html.
- Schmidt, W. H. (1999). Facing the consequences: Using TIMSS for a closer look at U.S. mathematics and science education. Dordrecht; Boston: Kluwer Academic Publishers.Google Scholar
- Schwerdt, G., & Wuppermann, A. C. (2011). Sage on the stage: Is lecturing really all that bad? Education Next, 11(3), 63–67.Google Scholar
- Shakeel, M. D., Anderson, K. P., & Wolf, P. J. (2016). The participant effects of private school vouchers across the globe: A meta analytic and systematic review. Retrieved from Little Rock: http://www.uaedreform.org/downloads/2016/05/the-participant-effects-of-private-school-vouchers-across-the-globe-a-meta-analytic-and-systematic-review-2.pdf.
- Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education. Washington DC: National Academy Press.Google Scholar
- Sjøberg, S. (2012). PISA: Politics, fundamental problems and intriguing results Recherches en Education, 14. Retrieved from http://www.recherches-en-education.net/spip.php?article140.
- Slavin, R. E. (1989). PET and the pendulum: Faddism in education and how to stop it. Phi Delta Kappan, 70(10), 752–758.Google Scholar
- Stewart, W. (2013, December 3rd). Is Pisa fundamentally flawed? Retrieved from http://www.tes.co.uk/article.aspx?storycode=6344672.
- Tarver, S. G. (1998). Myths and truths about direct instruction. Effective School Practices, 17(1), 18–22.Google Scholar
- Telegraph Reporters. (2016, July 23). Half of primary school pupils to follow Chinese style of learning maths with focus on whole-class teaching. Retrieved from http://www.telegraph.co.uk/education/2016/07/12/half-of-primary-school-pupils-to-follow-chinese-style-of-learnin/.
- Tienken, C. H., & Zhao, Y. (2013). How common standards and standardized testing widen the opportunity gap. In P. L. Carter & K. G. Welner (Eds.), Closing the opportunity gap: What America must do to give every child an even chance (pp. 113–122). New York: Oxford University Press.Google Scholar
- Toppo, G., Amos, D., Gillum, J., & Upton, J. (2011, March 17). When test scores seem too good to believe. USA TODAY. Retrieved from http://www.usatoday.com/news/education/2011-03-06-school-testing_N.htm.
- Tucker, M. (Ed.). (2011). Surpassing Shanghai: An agenda for American education built on the world’s leading systems. Boston: Harvard Education Press.Google Scholar
- Tucker, M. (2014). Chinese lessons: Shanghai’s rise to the top of the PISA league tables. Retrieved from Washington, DC: http://www.ncee.org/wp-content/uploads/2013/10/ChineseLessonsWeb.pdf.
- U.S. Department of Health and Human Services (Food and Drug Administration Center for Drug Evaluation and Research (CDER) Center for Biologics Evaluation and Research (CBER)). (2012). Guidance for industry and investigators: Safety reporting requirements for INDs and BA/BE studies. Retrieved from Silver Spring, MD: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM227351.pdf.
- Vogell, H. (2011, July 6, 2011). Investigation into APS cheating finds unethical behavior across every level. The Atlanta Journal-Constitution. Retrieved from http://www.ajc.com/news/investigation-into-aps-cheating-1001375.html.
- What Works Clearinghouse. (2006). Reading mastery. Retrieved from http://ies.ed.gov/ncee/wwc/interventionreport.aspx?sid=417.
- What Works Clearinghouse. (2007). Direct instruction. Retrieved from http://ies.ed.gov/ncee/wwc/interventionreport.aspx?sid=139.
- What Works Clearinghouse. (2014). What works clearinghouse: Procedures and standards handbook version 3.0 Retrieved from Washington, DC.Google Scholar
- What Works Clearinghouse. (2015). Frequently asked questions. Retrieved from http://ies.ed.gov/ncee/wwc/Document.aspx?sid=15-wwc.
- Zhao, Y. (2012a, December 11). Numbers can lie: What TIMSS and PISA truly tell us, if anything? Retrieved from http://zhaolearning.com/2012/12/11/numbers-can-lie-what-timss-and-pisa-truly-tell-us-if-anything/.
- Zhao, Y. (2012b). World class learners: Educating creative and entrepreneurial students. Thousand Oaks, CA: Corwin.Google Scholar
- Zhao, Y. (2014). Who’s afraid of the big bad dragon: Why China has the best (and worst) education system in the world. San Francisco: Jossey-Bass.Google Scholar
- Zhao, Y. (2015). Lessons that matter: What we should learn from Asian school systems. Retrieved from Melbourne: http://www.mitchellinstitute.org.au/reports/lessons-that-matter-what-should-we-learn-from-asias-school-systems/.
- Zhao, Y. (2016a). Counting what counts: Reframing education outcomes. Bloomington, IN: Solution Tree Press.Google Scholar
- Zhao, Y. (2016c, November 29). It must be the chopsticks: The less reported findings of 2015 TIMSS and explaining the East Asian outstanding performance. Retrieved from http://zhaolearning.com/2016/11/29/it-must-be-chopsticks-the-less-reported-findings-of-2015-timss-and-explaining-the-east-asian-outstanding-performance/.
- Zhao, Y. (2016d, November 29). Stop copying others: TIMSS lessons for America. Retrieved from http://zhaolearning.com/2016/11/30/stop-copying-others-timss-lessons-for-america/.