A general linear model for estimating effect size in the presence of publication bias

Abstract

When the process of publication favors studies with smallp-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailedp-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.

This is a preview of subscription content, log in to check access.

References

  1. Begg, C. B. (1994). Publication bias. In H. Cooper & L. V. Hedges,The handbook of research synthesis (pp. 399–409). New York: Russell Sage Foundation.

    Google Scholar 

  2. Begg, C. B., & Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data (with discussion).Journal of the Royal Statistical Society, Series A, 151, 419–463.

    Google Scholar 

  3. Bozarth, J. D., & Roberts, R. R. (1972). Signifying significant significance.American Psychologist, 27, 774–775.

    Google Scholar 

  4. Cooper, H., & Hedges, L. V. (1994).The handbook of research synthesis. New York: Russell Sage Foundation.

    Google Scholar 

  5. Coursol, A., & Wagner, E. E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias.Professional Psychology, 17, 136–137.

    Google Scholar 

  6. Dawes, R. M., Landman, J., & Williams, M. (1984). Discussion on meta-analysis and selective publication bias.American Psychologist, 39, 75–78.

    Google Scholar 

  7. Dear, K. B. G., & Begg, C. B. (1992). An approach for assessing publication bias prior to performing a meta-analysis.Statistical Science, 7, 237–245.

    Google Scholar 

  8. Dickersin, K., Min, Y-I, & Meinert, C. L. (1991). The fate of controlled trials funded by the NIH in 1979.Controlled Clinical Trials, 12, 634.

    Google Scholar 

  9. Dickersin, K., Min, Y-I, & Meinert, C. L. (1992). Factors influencing the publication of research results: Followup of applications submitted to two institutional review boards.Journal of the American Medical Association, 267, 374–378.

    Google Scholar 

  10. Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publication bias in clinical research.Lancet, 337, 867–872.

    Google Scholar 

  11. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis.Psychological Bulletin, 82, 1–20.

    Google Scholar 

  12. Hedges, L. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences.Journal of Educational Statistics, 9, 61–85.

    Google Scholar 

  13. Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis.Statistical Science, 7, 246–255.

    Google Scholar 

  14. Hedges, L. V., & Olkin, I. (1985).Statistical methods for meta-analysis. New York: Academic Press.

    Google Scholar 

  15. Hedges, L. V., & Vevea, J. L. (1993).Estimating effect size under publication bias: Small sample properties and robustness of a selection model. Manuscript submitted for publication.

  16. Iyengar, S., & Greenhouse, J. B. (1988). Selection models and the file drawer problem.Statistical Science, 3, 109–135.

    Google Scholar 

  17. Kendall, M., & Stuart, A. (1979).The advanced theory of statistics. Volume 2, Inference and relationship (4th ed.). London and High Wycombe: Charles Griffin and Company.

    Google Scholar 

  18. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions.British Journal of Mathematical and Statistical Psychology, 31, 107–112.

    Google Scholar 

  19. Light, R. J., & Pillemer, D. B. (1984).Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.

    Google Scholar 

  20. Melton, A. W. (1962). Editorial.Journal of Experimental Psychology, 64, 553–557.

    Google Scholar 

  21. National Research Council (1992).Combining information: Statistical issues and research opportunities. Washington, DC: National Academy Press.

    Google Scholar 

  22. Nelson, N., Rosenthal, R., & Rosnow, R. L. (1986). Interpretation of significance levels by psychological researchers.American Psychologist, 41, 1299–1301.

    Google Scholar 

  23. Rosenthal, R., & Gaito, J. (1963). The interpretation of levels of significance by psychological researchers.Journal of Psychology, 55, 33–38.

    Google Scholar 

  24. Rosenthal, R., & Gaito, J. (1964). Further evidence for the cliff effect in the interpretation of levels of significance.Psychological Reports, 4, 570.

    Google Scholar 

  25. Smith, M. L. (1980). Publication bias in meta-analysis.Evaluation in Education, 4, 22–24.

    Google Scholar 

  26. Smith, M. L., Glass, G. V., & Miller, T. I. (1980).The benefits of psychotherapy. Baltimore: The Johns Hopkins University Press.

    Google Scholar 

  27. Sterling, T. C. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vice versa.Journal of the American Statistical Association, 54, 30–34.

    Google Scholar 

  28. Vevea, J. L., Clements, N. C., & Hedges, L. V. (1993). Assessing the effects of selection bias on validity data for the general aptitude test battery.Journal of Applied Psychology, 78, 981–987.

    Google Scholar 

  29. White, K. R. (1982). The relation between socioeconomic status and achievement.Psychological Bulletin, 31, 461–481.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jack L. Vevea.

Additional information

Authors' note: The contributions of the authors are considered equal, and the order of authorship was chosen to be reverse-alphabetical.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Vevea, J.L., Hedges, L.V. A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60, 419–435 (1995). https://doi.org/10.1007/BF02294384

Download citation

Key words

  • meta-analysis
  • research synthesis
  • publication bias
  • effect size
  • mixed models
  • selection models