Psychometrika

, Volume 60, Issue 3, pp 419–435 | Cite as

A general linear model for estimating effect size in the presence of publication bias

  • Jack L. Vevea
  • Larry V. Hedges
Article

Abstract

When the process of publication favors studies with smallp-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailedp-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.

Key words

meta-analysis research synthesis publication bias effect size mixed models selection models 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Begg, C. B. (1994). Publication bias. In H. Cooper & L. V. Hedges,The handbook of research synthesis (pp. 399–409). New York: Russell Sage Foundation.Google Scholar
  2. Begg, C. B., & Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data (with discussion).Journal of the Royal Statistical Society, Series A, 151, 419–463.Google Scholar
  3. Bozarth, J. D., & Roberts, R. R. (1972). Signifying significant significance.American Psychologist, 27, 774–775.Google Scholar
  4. Cooper, H., & Hedges, L. V. (1994).The handbook of research synthesis. New York: Russell Sage Foundation.Google Scholar
  5. Coursol, A., & Wagner, E. E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias.Professional Psychology, 17, 136–137.Google Scholar
  6. Dawes, R. M., Landman, J., & Williams, M. (1984). Discussion on meta-analysis and selective publication bias.American Psychologist, 39, 75–78.Google Scholar
  7. Dear, K. B. G., & Begg, C. B. (1992). An approach for assessing publication bias prior to performing a meta-analysis.Statistical Science, 7, 237–245.Google Scholar
  8. Dickersin, K., Min, Y-I, & Meinert, C. L. (1991). The fate of controlled trials funded by the NIH in 1979.Controlled Clinical Trials, 12, 634.Google Scholar
  9. Dickersin, K., Min, Y-I, & Meinert, C. L. (1992). Factors influencing the publication of research results: Followup of applications submitted to two institutional review boards.Journal of the American Medical Association, 267, 374–378.Google Scholar
  10. Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publication bias in clinical research.Lancet, 337, 867–872.Google Scholar
  11. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis.Psychological Bulletin, 82, 1–20.Google Scholar
  12. Hedges, L. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences.Journal of Educational Statistics, 9, 61–85.Google Scholar
  13. Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis.Statistical Science, 7, 246–255.Google Scholar
  14. Hedges, L. V., & Olkin, I. (1985).Statistical methods for meta-analysis. New York: Academic Press.Google Scholar
  15. Hedges, L. V., & Vevea, J. L. (1993).Estimating effect size under publication bias: Small sample properties and robustness of a selection model. Manuscript submitted for publication.Google Scholar
  16. Iyengar, S., & Greenhouse, J. B. (1988). Selection models and the file drawer problem.Statistical Science, 3, 109–135.Google Scholar
  17. Kendall, M., & Stuart, A. (1979).The advanced theory of statistics. Volume 2, Inference and relationship (4th ed.). London and High Wycombe: Charles Griffin and Company.Google Scholar
  18. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions.British Journal of Mathematical and Statistical Psychology, 31, 107–112.Google Scholar
  19. Light, R. J., & Pillemer, D. B. (1984).Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.Google Scholar
  20. Melton, A. W. (1962). Editorial.Journal of Experimental Psychology, 64, 553–557.Google Scholar
  21. National Research Council (1992).Combining information: Statistical issues and research opportunities. Washington, DC: National Academy Press.Google Scholar
  22. Nelson, N., Rosenthal, R., & Rosnow, R. L. (1986). Interpretation of significance levels by psychological researchers.American Psychologist, 41, 1299–1301.Google Scholar
  23. Rosenthal, R., & Gaito, J. (1963). The interpretation of levels of significance by psychological researchers.Journal of Psychology, 55, 33–38.Google Scholar
  24. Rosenthal, R., & Gaito, J. (1964). Further evidence for the cliff effect in the interpretation of levels of significance.Psychological Reports, 4, 570.Google Scholar
  25. Smith, M. L. (1980). Publication bias in meta-analysis.Evaluation in Education, 4, 22–24.Google Scholar
  26. Smith, M. L., Glass, G. V., & Miller, T. I. (1980).The benefits of psychotherapy. Baltimore: The Johns Hopkins University Press.Google Scholar
  27. Sterling, T. C. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vice versa.Journal of the American Statistical Association, 54, 30–34.Google Scholar
  28. Vevea, J. L., Clements, N. C., & Hedges, L. V. (1993). Assessing the effects of selection bias on validity data for the general aptitude test battery.Journal of Applied Psychology, 78, 981–987.Google Scholar
  29. White, K. R. (1982). The relation between socioeconomic status and achievement.Psychological Bulletin, 31, 461–481.Google Scholar

Copyright information

© The Psychometric Society 1995

Authors and Affiliations

  • Jack L. Vevea
    • 1
  • Larry V. Hedges
    • 1
  1. 1.University of ChicagoUSA
  2. 2.University of North Carolina at Chapel Hill

Personalised recommendations