Meta-analyses are used to make educational decisions in policy and practice. Publication bias refers to the extent to which published literature is more likely to have statistically significant results and larger sample sizes than studies that do not make it through the publication process. The purpose of the present study is to estimate the extent to which publication bias is present in a broad set of education and special education journals. We reviewed 222 meta-analyses to describe the prevalence of publication bias tests, and further identified 29 that met inclusion criteria for effect size extraction. Descriptive data reveal that 58% of meta-analyses (n = 128) documented no effort to test for possible publication bias, and analyses of 72 difference statistics revealed that published studies were associated with significantly larger effect sizes than unpublished studies (d = 0.64). Exploratory moderator analyses revealed that effect size metric was a significant predictor of the difference between published and unpublished studies.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
References preceded by asterisks were included in the meta-review.
*Adesope, O. O., & Nesbit, J. C. (2012). Verbal redundancy in multimedia learning environments: a meta-analysis. Journal of Educational Psychology, 104(1), 250. https://doi.org/10.1037/a0026147.
*Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. https://doi.org/10.1037/a0021017.
*Alfieri, L., Nokes-Malach, T. J., & Schunn, C. D. (2013). Learning through case comparisons: a meta-analytic review. Educational Psychologist, 48(2), 87–113. https://doi.org/10.1080/00461520.2013.775712.
*Aloe, A. M., Amo, L. C., & Shanahan, M. E. (2014). Classroom management self-efficacy and burnout: a multivariate meta-analysis. Educational Psychology Review, 26(1), 101–126. https://doi.org/10.1007/s10648-013-9244-0.
Banda, D. R., & Therrien, W. J. (2008). A teacher’s guide to meta-analysis. Teaching Exceptional Children, 41(2), 66–71. https://doi.org/10.1177/004005990804100208.
Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Medicine, 7(9), e1000326.
Becker, B. (2005). Failsafe N or file-drawer number. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 35–48). Chichester: Wiley.
Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 1088–1101. https://doi.org/10.2307/2533446.
Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester: Wiley.
*Carbonneau, K. J., Marley, S. C., & Selig, J. P. (2013). A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives. Journal of Educational Psychology, 105(2), 380–400. https://doi.org/10.1037/a0031084.
Cheung, A. C., & Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283–292.
Cook, B. G., & Therrien, W. J. (2017). Null effects and publication bias in special education research. Behavioral Disorders. https://doi.org/10.1177/0198742917709473.
Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., et al. (1993). Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA, 269(21), 2749–2753.
Cooper, H., & Koenka, A. C. (2012). The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. American Psychologist, 67(6), 446.
*Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: a meta-analysis. Educational Psychology Review, 28(3), 425–474. https://doi.org/10.1007/s10648-015-9320-8.
Dickersin, K. (2005). Publication bias: recognizing the problem, understanding its origin and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Boren- stein (Eds.), Publication bias in metaanalysis: Prevention, assessment and adjustments (pp.11–33). Chichester: England, Wiley.
*Dunst, C. J. (2014). Meta-analysis of the effects of puppet shows on attitudes toward and knowledge of individuals with disabilities. Exceptional Children, 80(2), 136–148. https://doi.org/10.1177/001440291408000201.
Duval, S., & Tweedie, R. (2000). Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. https://doi.org/10.1111/j.0006-341x.2000.00455.x.
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634. https://doi.org/10.1136/bmj.315.7109.629.
Every Student Succeeds Act of 2015 (2015). Pub. L. No. 114-95 § 114 Stat. 1177.
Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128. https://doi.org/10.1037/a0024445.
Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6), 555–561. https://doi.org/10.1177/1745691612459059.
*Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: a meta-analysis. Exceptional Children, 53(3), 199–208. https://doi.org/10.1177/004005998401600318.
*Gajda, A., Karwowski, M., & Beghetto, R. A. (2016). Creativity and academic achievement: a meta-analysis. Journal of Educational Psychology. Advance online publication: https://doi.org/10.1037/edu0000133.
*Ginns, P., Martin, A. J., & Marsh, H. W. (2013). Designing instructional text in a conversational style: a meta-analysis. Educational Psychology Review, 25(4), 445–472. https://doi.org/10.1007/s10648-013-9228-0.
Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203.
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65.
Higgins, JPT, Green S (editors). (2011). Cochrane handbook for systematic reviews of interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org.
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. The Cochrane Library.
Hopewell, S., Clarke, M., & Mallett, S. (2005). Grey literature and systematic reviews. Publication bias in meta-analysis: prevention, assessment and adjustments, 48–72.
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. The Cochrane Library.
*Huang, C. (2012). Discriminant and criterion-related validity of achievement goals in predicting academic achievement: a meta-analysis. Journal of Educational Psychology, 104(1), 48–73. https://doi.org/10.1037/a0026223.
Individuals with Disabilities Education Act (IDEA) of 2004 (2004) 20 U.S.C. § 1401 et seq.
Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
Ioannidis, J. P., Munafo, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences, 18(5), 235–241.
*Klassen, R. M., Tze, V. M., & Hannok, W. (2011). Internalizing problems of adults with learning disabilities: a meta-analysis. Journal of Learning Disabilities, 46(4), 317–327. https://doi.org/10.1177/0022219411422260.
Kromrey, J. D., & Rendina-Gobioff, G. (2006). On knowing what we do not know: an empirical comparison of methods to detect publication bias in meta-analysis. Educational and Psychological Measurement, 66(3), 357–373.
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. M. K., Hammerstrøm, K., & Sathe, N. (2017). Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: The Campbell Collaboration. https://doi.org/10.4073/cmg.2016.1.
*Kulik, J. A., Bangert, R. L., & Williams, G. W. (1983). Effects of computer-based teaching on secondary school students. Journal of Educational Psychology, 75(1), 19–26. https://doi.org/10.1037/0022-06126.96.36.199.
*Lee, J., & Yoon, S. Y. (2015). The effects of repeated reading on reading fluency for students with reading disabilities: a meta-analysis. Journal of Learning Disabilities. https://doi.org/10.1177/0022219415605194.
*Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review, 22(3), 215–243. https://doi.org/10.1007/s10648-010-9125-8.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks: Sage.
*Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. Journal of Educational Psychology, 106(4), 901–918. https://doi.org/10.1037/a0037222.
*Maeda, Y., & Yoon, S. Y. (2013). A meta-analysis on gender differences in mental rotation ability measured by the Purdue spatial visualization tests: visualization of rotations (PSVT: R). Educational Psychology Review, 25(1), 69–94. https://doi.org/10.1007/s10648-012-9215-x.
McLeod, B. D., & Weisz, J. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72, 235–251. https://doi.org/10.1037/0022-006x.72.2.235.
*Murawski, W. W., & Swanson, H. L. (2001). A meta-analysis of co-teaching research where are the data? Remedial and Special Education, 22(5), 258–267. https://doi.org/10.1177/074193250102200501.
*Nelson, J. M., & Harwood, H. (2011). Learning disabilities and anxiety: a meta-analysis. Journal of Learning Disabilities, 44(1), 3–17. https://doi.org/10.1177/0022219409359939.
*Noltemeyer, A. L., Ward, R. M., & Mcloughlin, C. (2015). Relationship between school suspension and student outcomes: a meta-analysis. School Psychology Review, 44(2), 224–240. https://doi.org/10.17105/spr-14-0008.1.
Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8(2), 157–159. https://doi.org/10.2307/1164923.
*Owen, K. B., Parker, P. D., Van Zanden, B., MacMillan, F., Astell-Burt, T., & Lonsdale, C. (2016). Physical activity and school engagement in youth: a systematic review and meta-analysis. Educational Psychologist, 51(2), 129–145. https://doi.org/10.1080/00461520.2016.1151793.
*Pinquart, M. (2016). Associations of parenting styles and dimensions with academic achievement in children and adolescents: a meta-analysis. Educational Psychology Review, 28(3), 475–493. https://doi.org/10.1007/s10648-015-9338-y.
Polanin, J. R., Maynard, B. R., & Dell, N. A. (2017). Overviews in education research: a systematic review and analysis. Review of Educational Research, 87(1), 172–203.
Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the difference between published and unpublished effect sizes: a meta-review. Review of Educational Research, 86(1), 207–236. https://doi.org/10.3102/0034654315582067.
Rosenthal, R. (1979). The file drawer problem and tolerance for null result. Psychological Bulletin, 86, 638–641. https://doi.org/10.1037/0033-2909.86.3.638.
Rothstein, H. R., Bornstein, M., & Sutton, A. J. (2006). Publication bias in meta-analysis. Prevention, assessment, and adjustments. Chichester: Wiley.
*Schwinger, M., Wirthwein, L., Lemmer, G., & Steinmayr, R. (2014). Academic self-handicapping and achievement: a meta-analysis. Journal of Educational Psychology, 106(3), 744–761. https://doi.org/10.1037/a0035832.
Spaulding, L. S. (2009). Best practices and interventions in special education: how do we know what works? Teaching Exceptional Children Plus, 5(3), 1–13.
*Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970–987. https://doi.org/10.1037/a0032447.
Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 145–174). Chichester: Wiley.
Sterne, J. A., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology, 54(10), 1046–1055.
*Swanson, H. L. (1999). Reading research for students with LD: a meta-analysis of intervention outcomes. Journal of Learning Disabilities, 32(6), 504–532. https://doi.org/10.1177/002221949903200605.
Tanner-Smith, E. E., & Tipton, E. (2014). Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 4, 13–30. https://doi.org/10.1002/jrsm.1091.
*Tenenbaum, H. R., & Ruck, M. D. (2007). Are teachers’ expectations different for racial minority than for European American students? A meta-analysis. Journal of Educational Psychology, 99(2), 253–273. https://doi.org/10.1037/0022-06188.8.131.52.
*Theule, J., Wiener, J., Tannock, R., & Jenkins, J. M. (2012). Parenting stress in families of children with ADHD a meta-analysis. Journal of Emotional and Behavioral Disorders, 21(1), 3–17. https://doi.org/10.1177/1063426610387433.
Torgerson, C. J. (2006). Publication bias: the Achilles’ heel of systematic reviews? British Journal of Educational Studies, 54(1), 89–102.
Torgerson, C., Porthouse, J., & Brooks, G. (2003). A systematic review and meta-analysis of randomised controlled trials evaluating interventions in adult literacy and numeracy. Journal of Research in Reading, 26(3), 234–255.
*Vasquez, A. C., Patall, E. A., Fong, C. J., Corrigan, A. S., & Pine, L. (2016). Parent autonomy support, academic achievement, and psychosocial functioning: a meta-analysis of research. Educational Psychology Review, 28(3), 605–644. https://doi.org/10.1007/s10648-015-9329-z.
Wilson, D. (2009). Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews. Journal of Experimental Criminology, 5, 429–440. https://doi.org/10.1007/s11292-009-9085-5.
*Wouters, P., Van Nimwegen, C., Van Oostendorp, H., & Van Der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265. https://doi.org/10.1037/a0031311.
About this article
Cite this article
Chow, J.C., Ekholm, E. Do Published Studies Yield Larger Effect Sizes than Unpublished Studies in Education and Special Education? A Meta-review. Educ Psychol Rev 30, 727–744 (2018). https://doi.org/10.1007/s10648-018-9437-7
- Publication bias