Do Published Studies Yield Larger Effect Sizes than Unpublished Studies in Education and Special Education? A Meta-review

Meta-Analysis

Abstract

Meta-analyses are used to make educational decisions in policy and practice. Publication bias refers to the extent to which published literature is more likely to have statistically significant results and larger sample sizes than studies that do not make it through the publication process. The purpose of the present study is to estimate the extent to which publication bias is present in a broad set of education and special education journals. We reviewed 222 meta-analyses to describe the prevalence of publication bias tests, and further identified 29 that met inclusion criteria for effect size extraction. Descriptive data reveal that 58% of meta-analyses (n = 128) documented no effort to test for possible publication bias, and analyses of 72 difference statistics revealed that published studies were associated with significantly larger effect sizes than unpublished studies (d = 0.64). Exploratory moderator analyses revealed that effect size metric was a significant predictor of the difference between published and unpublished studies.

Keywords

Meta-review Meta-analysis Publication bias 

References

References preceded by asterisks were included in the meta-review.

  1. *Adesope, O. O., & Nesbit, J. C. (2012). Verbal redundancy in multimedia learning environments: a meta-analysis. Journal of Educational Psychology, 104(1), 250.  https://doi.org/10.1037/a0026147.
  2. *Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18.  https://doi.org/10.1037/a0021017.
  3. *Alfieri, L., Nokes-Malach, T. J., & Schunn, C. D. (2013). Learning through case comparisons: a meta-analytic review. Educational Psychologist, 48(2), 87–113.  https://doi.org/10.1080/00461520.2013.775712.
  4. *Aloe, A. M., Amo, L. C., & Shanahan, M. E. (2014). Classroom management self-efficacy and burnout: a multivariate meta-analysis. Educational Psychology Review, 26(1), 101–126.  https://doi.org/10.1007/s10648-013-9244-0.
  5. Banda, D. R., & Therrien, W. J. (2008). A teacher’s guide to meta-analysis. Teaching Exceptional Children, 41(2), 66–71.  https://doi.org/10.1177/004005990804100208.CrossRefGoogle Scholar
  6. Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Medicine, 7(9), e1000326.CrossRefGoogle Scholar
  7. Becker, B. (2005). Failsafe N or file-drawer number. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 35–48). Chichester: Wiley.Google Scholar
  8. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 1088–1101.  https://doi.org/10.2307/2533446.
  9. Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester: Wiley.CrossRefGoogle Scholar
  10. *Carbonneau, K. J., Marley, S. C., & Selig, J. P. (2013). A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives. Journal of Educational Psychology, 105(2), 380–400.  https://doi.org/10.1037/a0031084.
  11. Cheung, A. C., & Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283–292.CrossRefGoogle Scholar
  12. Cook, B. G., & Therrien, W. J. (2017). Null effects and publication bias in special education research. Behavioral Disorders.  https://doi.org/10.1177/0198742917709473.
  13. Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., et al. (1993). Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA, 269(21), 2749–2753.CrossRefGoogle Scholar
  14. Cooper, H., & Koenka, A. C. (2012). The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. American Psychologist, 67(6), 446.CrossRefGoogle Scholar
  15. *Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: a meta-analysis. Educational Psychology Review, 28(3), 425–474.  https://doi.org/10.1007/s10648-015-9320-8.
  16. Dickersin, K. (2005). Publication bias: recognizing the problem, understanding its origin and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Boren- stein (Eds.), Publication bias in metaanalysis: Prevention, assessment and adjustments (pp.11–33). Chichester: England, Wiley.Google Scholar
  17. *Dunst, C. J. (2014). Meta-analysis of the effects of puppet shows on attitudes toward and knowledge of individuals with disabilities. Exceptional Children, 80(2), 136–148.  https://doi.org/10.1177/001440291408000201.
  18. Duval, S., & Tweedie, R. (2000). Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463.  https://doi.org/10.1111/j.0006-341x.2000.00455.x.CrossRefGoogle Scholar
  19. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.  https://doi.org/10.1136/bmj.315.7109.629.CrossRefGoogle Scholar
  20. Every Student Succeeds Act of 2015 (2015). Pub. L. No. 114-95 § 114 Stat. 1177.Google Scholar
  21. Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128.  https://doi.org/10.1037/a0024445.CrossRefGoogle Scholar
  22. Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6), 555–561.  https://doi.org/10.1177/1745691612459059.CrossRefGoogle Scholar
  23. *Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: a meta-analysis. Exceptional Children, 53(3), 199–208.  https://doi.org/10.1177/004005998401600318.
  24. *Gajda, A., Karwowski, M., & Beghetto, R. A. (2016). Creativity and academic achievement: a meta-analysis. Journal of Educational Psychology. Advance online publication:  https://doi.org/10.1037/edu0000133.
  25. *Ginns, P., Martin, A. J., & Marsh, H. W. (2013). Designing instructional text in a conversational style: a meta-analysis. Educational Psychology Review, 25(4), 445–472.  https://doi.org/10.1007/s10648-013-9228-0.
  26. Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203.CrossRefGoogle Scholar
  27. Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65.CrossRefGoogle Scholar
  28. Higgins, JPT, Green S (editors). (2011). Cochrane handbook for systematic reviews of interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org.
  29. Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. The Cochrane Library.Google Scholar
  30. Hopewell, S., Clarke, M., & Mallett, S. (2005). Grey literature and systematic reviews. Publication bias in meta-analysis: prevention, assessment and adjustments, 48–72.Google Scholar
  31. Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. The Cochrane Library.Google Scholar
  32. *Huang, C. (2012). Discriminant and criterion-related validity of achievement goals in predicting academic achievement: a meta-analysis. Journal of Educational Psychology, 104(1), 48–73.  https://doi.org/10.1037/a0026223.
  33. Individuals with Disabilities Education Act (IDEA) of 2004 (2004) 20 U.S.C. § 1401 et seq.Google Scholar
  34. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.CrossRefGoogle Scholar
  35. Ioannidis, J. P., Munafo, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences, 18(5), 235–241.CrossRefGoogle Scholar
  36. *Klassen, R. M., Tze, V. M., & Hannok, W. (2011). Internalizing problems of adults with learning disabilities: a meta-analysis. Journal of Learning Disabilities, 46(4), 317–327.  https://doi.org/10.1177/0022219411422260.
  37. Kromrey, J. D., & Rendina-Gobioff, G. (2006). On knowing what we do not know: an empirical comparison of methods to detect publication bias in meta-analysis. Educational and Psychological Measurement, 66(3), 357–373.Google Scholar
  38. Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. M. K., Hammerstrøm, K., & Sathe, N. (2017). Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: The Campbell Collaboration.  https://doi.org/10.4073/cmg.2016.1.
  39. *Kulik, J. A., Bangert, R. L., & Williams, G. W. (1983). Effects of computer-based teaching on secondary school students. Journal of Educational Psychology, 75(1), 19–26.  https://doi.org/10.1037/0022-0663.75.1.19.
  40. *Lee, J., & Yoon, S. Y. (2015). The effects of repeated reading on reading fluency for students with reading disabilities: a meta-analysis. Journal of Learning Disabilities.  https://doi.org/10.1177/0022219415605194.
  41. *Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review, 22(3), 215–243.  https://doi.org/10.1007/s10648-010-9125-8.
  42. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks: Sage.Google Scholar
  43. *Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. Journal of Educational Psychology, 106(4), 901–918.  https://doi.org/10.1037/a0037222.
  44. *Maeda, Y., & Yoon, S. Y. (2013). A meta-analysis on gender differences in mental rotation ability measured by the Purdue spatial visualization tests: visualization of rotations (PSVT: R). Educational Psychology Review, 25(1), 69–94.  https://doi.org/10.1007/s10648-012-9215-x.
  45. McLeod, B. D., & Weisz, J. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72, 235–251.  https://doi.org/10.1037/0022-006x.72.2.235.CrossRefGoogle Scholar
  46. *Murawski, W. W., & Swanson, H. L. (2001). A meta-analysis of co-teaching research where are the data? Remedial and Special Education, 22(5), 258–267.  https://doi.org/10.1177/074193250102200501.
  47. *Nelson, J. M., & Harwood, H. (2011). Learning disabilities and anxiety: a meta-analysis. Journal of Learning Disabilities, 44(1), 3–17.  https://doi.org/10.1177/0022219409359939.
  48. *Noltemeyer, A. L., Ward, R. M., & Mcloughlin, C. (2015). Relationship between school suspension and student outcomes: a meta-analysis. School Psychology Review, 44(2), 224–240.  https://doi.org/10.17105/spr-14-0008.1.
  49. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8(2), 157–159.  https://doi.org/10.2307/1164923.Google Scholar
  50. *Owen, K. B., Parker, P. D., Van Zanden, B., MacMillan, F., Astell-Burt, T., & Lonsdale, C. (2016). Physical activity and school engagement in youth: a systematic review and meta-analysis. Educational Psychologist, 51(2), 129–145.  https://doi.org/10.1080/00461520.2016.1151793.
  51. *Pinquart, M. (2016). Associations of parenting styles and dimensions with academic achievement in children and adolescents: a meta-analysis. Educational Psychology Review, 28(3), 475–493.  https://doi.org/10.1007/s10648-015-9338-y.
  52. Polanin, J. R., Maynard, B. R., & Dell, N. A. (2017). Overviews in education research: a systematic review and analysis. Review of Educational Research, 87(1), 172–203.CrossRefGoogle Scholar
  53. Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the difference between published and unpublished effect sizes: a meta-review. Review of Educational Research, 86(1), 207–236.  https://doi.org/10.3102/0034654315582067.CrossRefGoogle Scholar
  54. Rosenthal, R. (1979). The file drawer problem and tolerance for null result. Psychological Bulletin, 86, 638–641.  https://doi.org/10.1037/0033-2909.86.3.638.CrossRefGoogle Scholar
  55. Rothstein, H. R., Bornstein, M., & Sutton, A. J. (2006). Publication bias in meta-analysis. Prevention, assessment, and adjustments. Chichester: Wiley.Google Scholar
  56. *Schwinger, M., Wirthwein, L., Lemmer, G., & Steinmayr, R. (2014). Academic self-handicapping and achievement: a meta-analysis. Journal of Educational Psychology, 106(3), 744–761.  https://doi.org/10.1037/a0035832.
  57. Spaulding, L. S. (2009). Best practices and interventions in special education: how do we know what works? Teaching Exceptional Children Plus, 5(3), 1–13.Google Scholar
  58. *Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970–987.  https://doi.org/10.1037/a0032447.
  59. Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, M. Bornstein, & A. J. Sutton (Eds.), Publication bias in meta-analysis: prevention, assessment, and adjustments (pp. 145–174). Chichester: Wiley.Google Scholar
  60. Sterne, J. A., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology, 54(10), 1046–1055.Google Scholar
  61. *Swanson, H. L. (1999). Reading research for students with LD: a meta-analysis of intervention outcomes. Journal of Learning Disabilities, 32(6), 504–532.  https://doi.org/10.1177/002221949903200605.
  62. Tanner-Smith, E. E., & Tipton, E. (2014). Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 4, 13–30.  https://doi.org/10.1002/jrsm.1091.CrossRefGoogle Scholar
  63. *Tenenbaum, H. R., & Ruck, M. D. (2007). Are teachers’ expectations different for racial minority than for European American students? A meta-analysis. Journal of Educational Psychology, 99(2), 253–273.  https://doi.org/10.1037/0022-0663.99.2.253.
  64. *Theule, J., Wiener, J., Tannock, R., & Jenkins, J. M. (2012). Parenting stress in families of children with ADHD a meta-analysis. Journal of Emotional and Behavioral Disorders, 21(1), 3–17.  https://doi.org/10.1177/1063426610387433.
  65. Torgerson, C. J. (2006). Publication bias: the Achilles’ heel of systematic reviews? British Journal of Educational Studies, 54(1), 89–102.CrossRefGoogle Scholar
  66. Torgerson, C., Porthouse, J., & Brooks, G. (2003). A systematic review and meta-analysis of randomised controlled trials evaluating interventions in adult literacy and numeracy. Journal of Research in Reading, 26(3), 234–255.CrossRefGoogle Scholar
  67. *Vasquez, A. C., Patall, E. A., Fong, C. J., Corrigan, A. S., & Pine, L. (2016). Parent autonomy support, academic achievement, and psychosocial functioning: a meta-analysis of research. Educational Psychology Review, 28(3), 605–644.  https://doi.org/10.1007/s10648-015-9329-z.
  68. Wilson, D. (2009). Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews. Journal of Experimental Criminology, 5, 429–440.  https://doi.org/10.1007/s11292-009-9085-5.CrossRefGoogle Scholar
  69. *Wouters, P., Van Nimwegen, C., Van Oostendorp, H., & Van Der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265.  https://doi.org/10.1037/a0031311.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Virginia Commonwealth UniversityRichmondUSA

Personalised recommendations