Journal of Experimental Criminology

, Volume 4, Issue 1, pp 61–81

Publication bias as a threat to the validity of meta-analytic results

Article
  • 356 Downloads

Abstract

This paper reviews the evidence in support of the contention that publication bias is a potential threat to the validity of meta-analytic results in criminology and similar fields. It then provides a critique of the traditional file drawer or failsafe N method for examining publication bias, and an overview of four newer methods that can be used to detect publication bias. These include two (trim and fill and cumulative meta-analysis) that enable the researcher to estimate the magnitude of the influence of publication bias on the overall mean effect size. Advantages and limitations of both traditional and newer methods are examined. The methods reviewed are illustrated through their application to a meta-analysis of the effects of drug courts on recidivism by Wilson et al. (Journal of Experimental Criminology, 2, 459–487, 2006).

Keywords

Failsafe N Funnel plot Meta-analysis Publication bias Trim and fill Validity 

References

  1. Begg, C. B. (1994). Publication Bias. In H. Cooper & L. Hedges (Eds.), The Handbook of Research Synthesis, NY: Russell Sage.Google Scholar
  2. Begg, C. B., & Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society A, 151, 419–463.CrossRefGoogle Scholar
  3. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.CrossRefGoogle Scholar
  4. Borenstein, M. (2005). Software for publication bias. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.Google Scholar
  5. Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2005). Comprehensive Meta Analysis, Version 2. Biostat, Englewood, NJ.Google Scholar
  6. Braga, A. (2005). Hot Spots Policing and Crime Prevention: A Systematic Review of Randomized Controlled Trials. Journal of Experimental Criminology, 1, 3.Google Scholar
  7. Chan, A-W., & Altman, D. G. (2005). Outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ, 330, 753.CrossRefGoogle Scholar
  8. Chan, A-W., Hrobjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. Journal of the American Medical Association, 291, 2457–2465.CrossRefGoogle Scholar
  9. Cooper, H. M. (1979). Statistically combining independent studies: A meta-analysis of sex differences in conformity research. Journal of Personality and Social Psychology, 37, 131–146.CrossRefGoogle Scholar
  10. Copas, J. B. (1999). What works? Selectivity models and meta-analysis. Journal of the Royal Statistical Association, 162, 95–109.CrossRefGoogle Scholar
  11. Copas, J. B., & Shi, J. Q. (2001). A sensitivity analysis for publication bias in systematic reviews. Statistical Methods in Medical Research, 10, 251–265.CrossRefGoogle Scholar
  12. Deffenbacher, K., Bornstein, B., Penrod, S., & McGorty, K. (2004). Meta-Analytic Review of the Effects of High Stress on Eyewitness Memory. Law and Human Behavior, 28, 687–706.CrossRefGoogle Scholar
  13. Del Vecchio, T., & O’Leary, K. D. (2004). The effectiveness of anger treatments for specific anger problems: A meta-analytic review. Clinical Psychology Review, 24, 15–34.CrossRefGoogle Scholar
  14. Dickersin, K. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments (pp. 11–34). Chichester, UK: Wiley.Google Scholar
  15. Dreznick, M. (2003). Heterosocial competence of rapists and child molesters: a meta-analysis. Journal of Sex Research, 40, 170–178.CrossRefGoogle Scholar
  16. Duval, S. (2005). The “trim and fill” method. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments (pp. 127–144). Chichester, UK: Wiley.Google Scholar
  17. Duval, S. J., & Tweedie, R. L. (2000a). A non-parametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89–98.CrossRefGoogle Scholar
  18. Duval, S. J., & Tweedie, R. L. (2000b). Trim and Fill: A simple funnel plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 276–284.CrossRefGoogle Scholar
  19. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315, 629–634.Google Scholar
  20. Egger, M., Davey Smith, G., & Altman, D. G. (2000). Systematic reviews in health care: Meta-analysis in context. London: BMJ Books.Google Scholar
  21. Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis. Statistical Science, 7, 246–255.CrossRefGoogle Scholar
  22. Hedges, L. V., & Vevea, J. L. (1996). Estimating effect size under publication bias: Small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics, 21, 299–332.CrossRefGoogle Scholar
  23. Hedges, L., & Vevea, J. (2005). The selection model approach to publication bias. In H. Rothstein, A.J. Sutton, & M. Borenstein (Eds.). Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.Google Scholar
  24. Illescas, S., Sanchez-Meca, J., & Garrido, V. (2001). Treatment of offenders and recidivism: Assessment of the effectiveness of programmes applied in Europe. Psychology in Spain, 5, 47–62.Google Scholar
  25. Ioannidis, J. (2005). Differentiating biases from genuine heterogeneity. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.). Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.Google Scholar
  26. Krleža-Jeric, K., Chan, A-W., Dickersin, K., Sim, I., Grimshaw, J., Gluud, C., for the Ottawa Group. (2005) Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa Statement (part 1). BMJ, 330, 956–958.Google Scholar
  27. Landenberger, N., & Lipsey, M. W. (2005). The positive effects of cognitive-behavioral programs for offenders: A meta-analysis of factors associated with associated with effective treatment. Journal of Experimental Criminology, 1, 451–476.CrossRefGoogle Scholar
  28. Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992). Cumulative meta-analysis of therapeutic trials for myocardial infarction. New England Journal of Medicine, 327, 248–254.CrossRefGoogle Scholar
  29. Light, R., & Pillemer, D. (1984). Summing Up: the Science of Reviewing Research. Cambridge, MA: Harvard University Press.Google Scholar
  30. Light, R. J., Singer, J. D., & Willett, J. B. (1994). The visual presentation and interpretation of meta-analyses. In H. M. Cooper, & L. V. Hedges (Eds.), The Handbook of Research Synthesis. New York: Russell Sage Foundation.Google Scholar
  31. Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: Good, bad, and ugly. Annals of the American Academy of Political and Social Science, 587, 69–81.CrossRefGoogle Scholar
  32. Losel, F., & Schmucker, M. (2005). The effectiveness of treatment for sexual offenders: A comprehensive meta-analysis.Google Scholar
  33. McCord, J. (2003). Cures that harm: Unanticipated outcomes of crime prevention programs. Annals of the American Academy of Political and Social Science, 587, 16–30.CrossRefGoogle Scholar
  34. Mullen, B., Muellerleile, P., Bryant, B. (2001). Cumulative meta-analysis: A consideration of indicators of sufficiency and stability. Personality and Social Psychology Bulletin, 27, 1450–1462.CrossRefGoogle Scholar
  35. NY vs GlaxoSmithKline. Filing to the Supreme Court of the State of New York. June 2, 2004.Google Scholar
  36. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157–159.CrossRefGoogle Scholar
  37. Pratt, T., Cullen, C., Blevins, K., Daigle, L., & Unnver, J. (2002). The relationship of attention deficit hyperactivity disorder to crime and delinquency: A meta-analysis. International Journal of Police Science and Management, 4, 344–360.CrossRefGoogle Scholar
  38. Rhoades, L., & Eisenberger, R. (2002). Perceived organizational support: A review of the literature. Journal of Applied Psychology, 87, 698–714.CrossRefGoogle Scholar
  39. Richy, F., & Reginster, J. (2006). A Simple Method for Detecting and Adjusting Meta-Analyses for Publication Bias. The Internet Journal of Epidemiology, 3, 2.Google Scholar
  40. Rosenberg, M. S., Adams, D. C., & Gurevitch, J. (2000). MetaWin: Statistical Software for Meta-Analysis, Version 2.0. Sunderland, MA: Sinauer Associates.Google Scholar
  41. Rosenthal, R. (1979).The ‘File Drawer Problem’ and Tolerance for Null Results. Psychological Bulletin, 86, 638–641.Google Scholar
  42. Sterne, J. A. C., Bradburn, M. J., & Egger, M. (2001a). Meta-Analysis in Stata. In M. Egger, G. Davey Smith, & D. G. Altman (Eds.), Systematic Reviews in Health Care: Meta Analysis in Context (pp. 347–369). London: BMJ.Google Scholar
  43. Sterne, J. A. C., Egger, M., & Davey Smith, G. (2001b). Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis. BMJ, 323, 101–105.CrossRefGoogle Scholar
  44. Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. . In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.Google Scholar
  45. Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology, 54, 1046–1055.CrossRefGoogle Scholar
  46. Sterne, J. A. C., & Egger, M. (2005). Regression methods to detect publication and other bias in meta-analysis. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments. Chichester, UK: Wiley.Google Scholar
  47. Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000). Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology, 53, 1119–1129.CrossRefGoogle Scholar
  48. Sterne, J. A. C., Harris, R., Harbord, R., & Steichen, T. (2007). What meta-analysis features are available in Stata? Retrieved from http://www.stata.com/support/faqs/stat/meta.html on 21 June 2007.
  49. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113–2126.CrossRefGoogle Scholar
  50. Wall Street Journal, (2004, November 1). E-Mails Suggest Merck Knew Vioxx’s Dangers at Early Stage.Google Scholar
  51. Weisburd, D., Petrosino, A., & Mason, G. (1993). Design sensitivity in criminal justice experiments. Crime and Justice, 17, 337–379.CrossRefGoogle Scholar
  52. Wilson, D. B., Mitchell, O., & MacKenzie, D. (2006). A systematic review of drug court effects on recidivism. Journal of Experimental Criminology, 2, 459–487.CrossRefGoogle Scholar
  53. Zhang, J., Ding, E. L., & Song, Y. (2006). Adverse effects of cyclooxygenase 2 inhibitors on renal and arrhythmia events: meta-analysis of randomized trials. JAMA, 296, 1619–1632.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2007

Authors and Affiliations

  1. 1.Department of Management Zicklin School of BusinessBaruch College—CUNYNew YorkUSA

Personalised recommendations