Do Published Studies Yield Larger Effect Sizes than Unpublished Studies in Education and Special Education? A Meta-review

Abstract

Meta-analyses are used to make educational decisions in policy and practice. Publication bias refers to the extent to which published literature is more likely to have statistically significant results and larger sample sizes than studies that do not make it through the publication process. The purpose of the present study is to estimate the extent to which publication bias is present in a broad set of education and special education journals. We reviewed 222 meta-analyses to describe the prevalence of publication bias tests, and further identified 29 that met inclusion criteria for effect size extraction. Descriptive data reveal that 58% of meta-analyses (n = 128) documented no effort to test for possible publication bias, and analyses of 72 difference statistics revealed that published studies were associated with significantly larger effect sizes than unpublished studies (d = 0.64). Exploratory moderator analyses revealed that effect size metric was a significant predictor of the difference between published and unpublished studies.

This is a preview of subscription content, log in to check access.

Fig. 1

References

References preceded by asterisks were included in the meta-review.

  1. *Adesope, O. O., & Nesbit, J. C. (2012). Verbal redundancy in multimedia learning environments: a meta-analysis. Journal of Educational Psychology, 104(1), 250. https://doi.org/10.1037/a0026147.

  2. *Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. https://doi.org/10.1037/a0021017.

  3. *Alfieri, L., Nokes-Malach, T. J., & Schunn, C. D. (2013). Learning through case comparisons: a meta-analytic review. Educational Psychologist, 48(2), 87–113. https://doi.org/10.1080/00461520.2013.775712.

  4. *Aloe, A. M., Amo, L. C., & Shanahan, M. E. (2014). Classroom management self-efficacy and burnout: a multivariate meta-analysis. Educational Psychology Review, 26(1), 101–126. https://doi.org/10.1007/s10648-013-9244-0.

  5. Banda, D. R., & Therrien, W. J. (2008). A teacher’s guide to meta-analysis. Teaching Exceptional Children, 41(2), 66–71. https://doi.org/10.1177/004005990804100208.

    Article  Google Scholar 

  6. Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Medicine, 7(9), e1000326.

    Article  Google Scholar 

  7. Becker, B. (2005). Failsafe N or file-drawer number. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 35–48). Chichester: Wiley.

    Google Scholar 

  8. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 1088–1101. https://doi.org/10.2307/2533446.

  9. Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester: Wiley.

    Google Scholar 

  10. *Carbonneau, K. J., Marley, S. C., & Selig, J. P. (2013). A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives. Journal of Educational Psychology, 105(2), 380–400. https://doi.org/10.1037/a0031084.

  11. Cheung, A. C., & Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283–292.

    Article  Google Scholar 

  12. Cook, B. G., & Therrien, W. J. (2017). Null effects and publication bias in special education research. Behavioral Disorders. https://doi.org/10.1177/0198742917709473.

  13. Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., et al. (1993). Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA, 269(21), 2749–2753.

    Article  Google Scholar 

  14. Cooper, H., & Koenka, A. C. (2012). The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. American Psychologist, 67(6), 446.

    Article  Google Scholar 

  15. *Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: a meta-analysis. Educational Psychology Review, 28(3), 425–474. https://doi.org/10.1007/s10648-015-9320-8.

  16. Dickersin, K. (2005). Publication bias: recognizing the problem, understanding its origin and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Boren- stein (Eds.), Publication bias in metaanalysis: Prevention, assessment and adjustments (pp.11–33). Chichester: England, Wiley.

  17. *Dunst, C. J. (2014). Meta-analysis of the effects of puppet shows on attitudes toward and knowledge of individuals with disabilities. Exceptional Children, 80(2), 136–148. https://doi.org/10.1177/001440291408000201.

  18. Duval, S., & Tweedie, R. (2000). Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. https://doi.org/10.1111/j.0006-341x.2000.00455.x.

    Article  Google Scholar 

  19. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634. https://doi.org/10.1136/bmj.315.7109.629.

    Article  Google Scholar 

  20. Every Student Succeeds Act of 2015 (2015). Pub. L. No. 114-95 § 114 Stat. 1177.

  21. Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128. https://doi.org/10.1037/a0024445.

    Article  Google Scholar 

  22. Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6), 555–561. https://doi.org/10.1177/1745691612459059.

    Article  Google Scholar 

  23. *Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: a meta-analysis. Exceptional Children, 53(3), 199–208. https://doi.org/10.1177/004005998401600318.

  24. *Gajda, A., Karwowski, M., & Beghetto, R. A. (2016). Creativity and academic achievement: a meta-analysis. Journal of Educational Psychology. Advance online publication: https://doi.org/10.1037/edu0000133.

  25. *Ginns, P., Martin, A. J., & Marsh, H. W. (2013). Designing instructional text in a conversational style: a meta-analysis. Educational Psychology Review, 25(4), 445–472. https://doi.org/10.1007/s10648-013-9228-0.

  26. Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203.

    Article  Google Scholar 

  27. Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65.

    Article  Google Scholar 

  28. Higgins, JPT, Green S (editors). (2011). Cochrane handbook for systematic reviews of interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org.

  29. Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. The Cochrane Library.

  30. Hopewell, S., Clarke, M., & Mallett, S. (2005). Grey literature and systematic reviews. Publication bias in meta-analysis: prevention, assessment and adjustments, 48–72.

  31. Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. The Cochrane Library.

  32. *Huang, C. (2012). Discriminant and criterion-related validity of achievement goals in predicting academic achievement: a meta-analysis. Journal of Educational Psychology, 104(1), 48–73. https://doi.org/10.1037/a0026223.

  33. Individuals with Disabilities Education Act (IDEA) of 2004 (2004) 20 U.S.C. § 1401 et seq.

  34. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.

    Article  Google Scholar 

  35. Ioannidis, J. P., Munafo, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences, 18(5), 235–241.

    Article  Google Scholar 

  36. *Klassen, R. M., Tze, V. M., & Hannok, W. (2011). Internalizing problems of adults with learning disabilities: a meta-analysis. Journal of Learning Disabilities, 46(4), 317–327. https://doi.org/10.1177/0022219411422260.

  37. Kromrey, J. D., & Rendina-Gobioff, G. (2006). On knowing what we do not know: an empirical comparison of methods to detect publication bias in meta-analysis. Educational and Psychological Measurement, 66(3), 357–373.

  38. Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. M. K., Hammerstrøm, K., & Sathe, N. (2017). Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: The Campbell Collaboration. https://doi.org/10.4073/cmg.2016.1.

  39. *Kulik, J. A., Bangert, R. L., & Williams, G. W. (1983). Effects of computer-based teaching on secondary school students. Journal of Educational Psychology, 75(1), 19–26. https://doi.org/10.1037/0022-0663.75.1.19.

  40. *Lee, J., & Yoon, S. Y. (2015). The effects of repeated reading on reading fluency for students with reading disabilities: a meta-analysis. Journal of Learning Disabilities. https://doi.org/10.1177/0022219415605194.

  41. *Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review, 22(3), 215–243. https://doi.org/10.1007/s10648-010-9125-8.

  42. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks: Sage.

    Google Scholar 

  43. *Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. Journal of Educational Psychology, 106(4), 901–918. https://doi.org/10.1037/a0037222.

  44. *Maeda, Y., & Yoon, S. Y. (2013). A meta-analysis on gender differences in mental rotation ability measured by the Purdue spatial visualization tests: visualization of rotations (PSVT: R). Educational Psychology Review, 25(1), 69–94. https://doi.org/10.1007/s10648-012-9215-x.

  45. McLeod, B. D., & Weisz, J. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72, 235–251. https://doi.org/10.1037/0022-006x.72.2.235.

    Article  Google Scholar 

  46. *Murawski, W. W., & Swanson, H. L. (2001). A meta-analysis of co-teaching research where are the data? Remedial and Special Education, 22(5), 258–267. https://doi.org/10.1177/074193250102200501.

  47. *Nelson, J. M., & Harwood, H. (2011). Learning disabilities and anxiety: a meta-analysis. Journal of Learning Disabilities, 44(1), 3–17. https://doi.org/10.1177/0022219409359939.

  48. *Noltemeyer, A. L., Ward, R. M., & Mcloughlin, C. (2015). Relationship between school suspension and student outcomes: a meta-analysis. School Psychology Review, 44(2), 224–240. https://doi.org/10.17105/spr-14-0008.1.

  49. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8(2), 157–159. https://doi.org/10.2307/1164923.

    Google Scholar 

  50. *Owen, K. B., Parker, P. D., Van Zanden, B., MacMillan, F., Astell-Burt, T., & Lonsdale, C. (2016). Physical activity and school engagement in youth: a systematic review and meta-analysis. Educational Psychologist, 51(2), 129–145. https://doi.org/10.1080/00461520.2016.1151793.

  51. *Pinquart, M. (2016). Associations of parenting styles and dimensions with academic achievement in children and adolescents: a meta-analysis. Educational Psychology Review, 28(3), 475–493. https://doi.org/10.1007/s10648-015-9338-y.

  52. Polanin, J. R., Maynard, B. R., & Dell, N. A. (2017). Overviews in education research: a systematic review and analysis. Review of Educational Research, 87(1), 172–203.

    Article  Google Scholar 

  53. Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the difference between published and unpublished effect sizes: a meta-review. Review of Educational Research, 86(1), 207–236. https://doi.org/10.3102/0034654315582067.

    Article  Google Scholar 

  54. Rosenthal, R. (1979). The file drawer problem and tolerance for null result. Psychological Bulletin, 86, 638–641. https://doi.org/10.1037/0033-2909.86.3.638.

    Article  Google Scholar 

  55. Rothstein, H. R., Bornstein, M., & Sutton, A. J. (2006). Publication bias in meta-analysis. Prevention, assessment, and adjustments. Chichester: Wiley.

    Google Scholar 

  56. *Schwinger, M., Wirthwein, L., Lemmer, G., & Steinmayr, R. (2014). Academic self-handicapping and achievement: a meta-analysis. Journal of Educational Psychology, 106(3), 744–761. https://doi.org/10.1037/a0035832.

  57. Spaulding, L. S. (2009). Best practices and interventions in special education: how do we know what works? Teaching Exceptional Children Plus, 5(3), 1–13.

    Google Scholar 

  58. *Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970–987. https://doi.org/10.1037/a0032447.

  59. Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 145–174). Chichester: Wiley.

    Google Scholar 

  60. Sterne, J. A., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology, 54(10), 1046–1055.

  61. *Swanson, H. L. (1999). Reading research for students with LD: a meta-analysis of intervention outcomes. Journal of Learning Disabilities, 32(6), 504–532. https://doi.org/10.1177/002221949903200605.

  62. Tanner-Smith, E. E., & Tipton, E. (2014). Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 4, 13–30. https://doi.org/10.1002/jrsm.1091.

    Article  Google Scholar 

  63. *Tenenbaum, H. R., & Ruck, M. D. (2007). Are teachers’ expectations different for racial minority than for European American students? A meta-analysis. Journal of Educational Psychology, 99(2), 253–273. https://doi.org/10.1037/0022-0663.99.2.253.

  64. *Theule, J., Wiener, J., Tannock, R., & Jenkins, J. M. (2012). Parenting stress in families of children with ADHD a meta-analysis. Journal of Emotional and Behavioral Disorders, 21(1), 3–17. https://doi.org/10.1177/1063426610387433.

  65. Torgerson, C. J. (2006). Publication bias: the Achilles’ heel of systematic reviews? British Journal of Educational Studies, 54(1), 89–102.

    Article  Google Scholar 

  66. Torgerson, C., Porthouse, J., & Brooks, G. (2003). A systematic review and meta-analysis of randomised controlled trials evaluating interventions in adult literacy and numeracy. Journal of Research in Reading, 26(3), 234–255.

    Article  Google Scholar 

  67. *Vasquez, A. C., Patall, E. A., Fong, C. J., Corrigan, A. S., & Pine, L. (2016). Parent autonomy support, academic achievement, and psychosocial functioning: a meta-analysis of research. Educational Psychology Review, 28(3), 605–644. https://doi.org/10.1007/s10648-015-9329-z.

  68. Wilson, D. (2009). Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews. Journal of Experimental Criminology, 5, 429–440. https://doi.org/10.1007/s11292-009-9085-5.

    Article  Google Scholar 

  69. *Wouters, P., Van Nimwegen, C., Van Oostendorp, H., & Van Der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265. https://doi.org/10.1037/a0031311.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jason C. Chow.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chow, J.C., Ekholm, E. Do Published Studies Yield Larger Effect Sizes than Unpublished Studies in Education and Special Education? A Meta-review. Educ Psychol Rev 30, 727–744 (2018). https://doi.org/10.1007/s10648-018-9437-7

Download citation

Keywords

  • Meta-review
  • Meta-analysis
  • Publication bias