Advertisement

Journal of Business and Psychology

, Volume 29, Issue 2, pp 183–203 | Cite as

Avoiding Bias in Publication Bias Research: The Value of “Null” Findings

Article

Abstract

Meta-analytic reviews are an important tool for advancing science and guiding evidence-based practice. Publication bias is one of the greatest threats to meta-analytic reviews. This paper assesses the degree of publication bias in four previously published meta-analytic datasets from various fields of study in the organizational sciences. Of these datasets, one appears to be relatively unaffected by publication bias while the others seem to be noticeably influenced by this bias. Our “null” result (i.e., a prior meta-analytic estimate is unlikely to have been affected by publication bias) increases our confidence in the accuracy of our cumulative knowledge. Yet, our other findings suggest the presence of publication bias and point to the need for caution and further research.

Keywords

Meta-analysis Systematic reviews Publication bias Dissemination bias Null results 

References

  1. Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R., & Dalton, C. M. (2011). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14, 306–331. doi: 10.1177/1094428110375720.CrossRefGoogle Scholar
  2. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.Google Scholar
  3. Banks, G. C., Batchelor, J. H., & McDaniel, M. A. (2010). Smarter people are (a bit) more symmetrical: A meta-analysis of the relationship between intelligence and fluctuating asymmetry. Intelligence, 38, 393–401. doi: 10.1016/j.intell.2010.04.003.CrossRefGoogle Scholar
  4. Banks, G. C., Kepes, S., & Banks, K. P. (2012a). Publication bias: The antagonist of meta-analytic reviews and effective policy making. Educational Evaluation and Policy Analysis, 34, 259–277. doi: 10.3102/0162373712446144.CrossRefGoogle Scholar
  5. Banks, G. C., Kepes, S., & McDaniel, M. A. (2012b). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20, 182–196. doi: 10.1111/j.1468-2389.2012.00591.x.CrossRefGoogle Scholar
  6. Banks, G. C., & McDaniel, M. A. (2011). The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 40–44. doi: 10.1111/j.1754-9434.2010.01292.x.CrossRefGoogle Scholar
  7. Becker, B. J. (1994). Combining significance levels. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 215–230). New York: Russell Sage Foundation.Google Scholar
  8. Becker, B. J. (2005). The failsafe N or file-drawer number. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 111–126). West Sussex: Wiley.Google Scholar
  9. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101. doi: 10.2307/2533446.PubMedCrossRefGoogle Scholar
  10. Berlin, J. A., & Ghersi, D. (2005). Preventing publication bias: Registries and prospective meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 35–48). West Sussex: Wiley.Google Scholar
  11. Bettencourt, B. A., & Miller, N. (1996). Gender differences in aggression as a function of provocation: A meta-analysis. Psychological Bulletin, 119, 422–447. doi: 10.1037/0033-2909.119.3.422.PubMedCrossRefGoogle Scholar
  12. Blackwell, S. C., Thompson, L., & Refuerzo, J. (2009). Full publication of clinical trials presented at a national maternal-fetal medicine meeting: Is there a publication bias? American Journal of Perinatology, 26, 679–682. doi: 10.1055/s-0029-1220786.PubMedCrossRefGoogle Scholar
  13. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2005). Comprehensive meta-analysis (Version 2). Englewood: Biostat.Google Scholar
  14. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester: Wiley.CrossRefGoogle Scholar
  15. Briner, R. B., & Rousseau, D. M. (2011). Evidence-based I-O psychology: Not there yet. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 3–22. doi: 10.1111/j.1754-9434.2010.01287.x.CrossRefGoogle Scholar
  16. Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. Journal of the American Medical Association, 291, 2457–2465. doi: 10.1001/jama.291.20.2457.PubMedCrossRefGoogle Scholar
  17. Cooper, H. (2003). Editorial. Psychological Bulletin, 129, 3–9. doi: 10.1037/0033-2909.129.1.3.CrossRefGoogle Scholar
  18. Curfman, G. D., Morrissey, S., & Drazen, J. M. (2006). Expression of concern reaffirmed. New England Journal of Medicine, 354, 1193. doi: 10.1056/NEJMe068054.PubMedCrossRefGoogle Scholar
  19. Davis, M. S. (1971). That’s interesting! Towards a phenomenology of sociology and a sociology of phenomenology. Philosophy of the Social Sciences, 1, 309–344. doi: 10.1177/004839317100100211.CrossRefGoogle Scholar
  20. Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association, 263, 1385–1389. doi: 10.1001/jama.263.10.1385.PubMedCrossRefGoogle Scholar
  21. Dickersin, K. (2005). Publication bias: Recognizing the problem, understandings its origins and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 11–34). West Sussex: Wiley.Google Scholar
  22. Duval, S. J. (2005). The “trim and fill” method. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 127–144). West Sussex: Wiley.Google Scholar
  23. Duval, S. J., & Tweedie, R. L. (2000a). A nonparametric “Trim and Fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89–98. doi: 10.2307/2669529.Google Scholar
  24. Duval, S. J., & Tweedie, R. L. (2000b). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. doi: 10.1111/j.0006-341X.2000.00455.x.PubMedCrossRefGoogle Scholar
  25. Dwan, K., et al. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE, 3, e3081. doi: 10.1371/journal.pone.00030.PubMedCentralPubMedCrossRefGoogle Scholar
  26. Eagly, A. H., Johannesen-Schmidt, M. C., & van Engen, M. L. (2003). Transformational, transactional, and laissez-faire leadership styles: A meta-analysis comparing women and men. Psychological Bulletin, 129, 569–591. doi: 10.1037/0033-2909.129.4.569.PubMedCrossRefGoogle Scholar
  27. Egger, M., & Smith, G. D. (1998). Bias in location and selection of studies. British Medical Journal, 316, 61–66. doi: 10.1136/bmj.316.7124.61.PubMedCentralPubMedCrossRefGoogle Scholar
  28. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634. doi: 10.1136/bmj.315.7109.629.PubMedCentralPubMedCrossRefGoogle Scholar
  29. Evangelou, E., Trikalinos, T. A., & Ioannidis, J. P. (2005). Unavailability of online supplementary scientific information from articles published in major journals. The FASEB Journal, 19, 1943–1944. doi: 10.1096/fj.05-4784lsf.CrossRefGoogle Scholar
  30. Evans, S. (1996). Statistician’s comment (to Misleading meta-analysis: “Fail safe N” is a useful mathematical measure of the stability of results by R. Persaud). British Medical Journal, 312, 125.PubMedCentralCrossRefGoogle Scholar
  31. Ferguson, C. J., & Brannick, M. T. (2011). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128. doi: 10.1037/a0024445.PubMedCrossRefGoogle Scholar
  32. Field, A. P., & Gillett, R. (2010). How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology, 63, 665–694. doi: 10.1348/000711010X502733.PubMedCrossRefGoogle Scholar
  33. Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587–606. doi: 10.1016/j.socec.2004.09.033.CrossRefGoogle Scholar
  34. Greenhouse, J. B., & Iyengar, S. (2009). Sensitivity analysis and diagnostics. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 417–433). New York: Russell Sage Foundation.Google Scholar
  35. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20. doi: 10.1037/h0076157.CrossRefGoogle Scholar
  36. Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis. Statistical Science, 7, 246–255. doi: 10.1214/ss/1177011364.CrossRefGoogle Scholar
  37. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.Google Scholar
  38. Hedges, L. V., & Vevea, J. L. (2005). Selection methods approaches. In H. R. Rothstein, A. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 145–174). West Sussex: Wiley.Google Scholar
  39. Hermelin, E., Lievens, F., & Robertson, I. T. (2007). The validity of assessment centers for the prediction of supervisory performance ratings: A meta-analysis. International Journal of Selection and Assessment, 15, 405–411. doi: 10.1111/j.1468-2389.2007.00399.x.CrossRefGoogle Scholar
  40. Higgins, J. P. T., & Green, S. (Eds.). (2009). Cochrane handbook for systematic reviews of interventions; Version 5.0.2 [updated September 2009]: The Cochrane Collaboration. www.cochrane-handbook.org.
  41. Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558. doi: 10.1002/sim.1186.PubMedCrossRefGoogle Scholar
  42. Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557–560. doi: 10.1136/bmj.327.7414.557.PubMedCentralPubMedCrossRefGoogle Scholar
  43. Hopewell, S., Clarke, M., & Mallett, S. (2005). Grey literature and systematic reviews. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 48–72). West Sussex: Wiley.Google Scholar
  44. Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews. doi: 10.1002/14651858.MR000006.pub3.Google Scholar
  45. Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews. doi: 10.1002/14651858.MR000006.pub3.Google Scholar
  46. Hubbard, R., & Armstrong, J. S. (1997). Publication bias against null results. Psychological Reports, 80, 337–338. doi: 10.2466/PR0.80.1.337-338.CrossRefGoogle Scholar
  47. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park: Sage.Google Scholar
  48. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2, e124. doi: 10.1371/journal.pmed.0020124.PubMedCentralPubMedCrossRefGoogle Scholar
  49. Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602–611. doi: 10.2307/2392366.CrossRefGoogle Scholar
  50. Kepes, S., Banks, G. C., McDaniel, M., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods,. doi: 10.1177/1094428112452760.Google Scholar
  51. Kisamore, J. L., & Brannick, M. T. (2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11, 35–53. doi: 10.1177/1094428106287393.CrossRefGoogle Scholar
  52. Le, H., Oh, I.-S., Shaffer, J., & Schmidt, F. L. (2007). Implications of methodological advances for the practice of personnel selection: How practitioners benefit from meta-analysis. Academy of Management Perspectives, 21, 6–15.CrossRefGoogle Scholar
  53. McDaniel, M. A. (2009, April). Cumulative meta-analysis as a publication bias method. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.Google Scholar
  54. McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927–953. doi: 10.1111/j.1744-6570.2006.00059.x.CrossRefGoogle Scholar
  55. McNatt, D. B. (2000). Ancient Pygmalion joins contemporary management: A meta-analysis of the result. Journal of Applied Psychology, 85, 314–322. doi: 10.1037/0021-9010.85.2.314.PubMedCrossRefGoogle Scholar
  56. Moreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M., et al. (2009). Novel methods to deal with publication biases: Secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. British Medical Journal, 339, b2981. doi: 10.1136/bmj.b2981.PubMedCentralPubMedCrossRefGoogle Scholar
  57. Palmer, T. M., Peters, J. L., Sutton, A. J., & Moreno, S. G. (2008). Contour-enhanced funnel plots for meta-analysis. Stata Journal, 8, 242–254.Google Scholar
  58. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008). Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology, 61, 991–996. doi: 10.1016/j.jclinepi.2007.11.010.PubMedCrossRefGoogle Scholar
  59. Peters, J., Sutton, A., Jones, D. R., Abrams, K. R., Rushton, L., & Moreno, S. G. (2010). Assessing publication bias in meta-analyses in the presence of between-study heterogeneity. Journal of the Royal Statistical Society (Series A), 173, 575–591. doi: 10.1111/j.1467-985X.2009.00629.x.CrossRefGoogle Scholar
  60. Quiñones, M. A., Ford, J. K., & Teachout, M. S. (1995). The relationship between work experience and job performance: A conceptual and meta-analytic review. Personnel Psychology, 48, 887–910. doi: 10.1111/j.1744-6570.1995.tb01785.x.CrossRefGoogle Scholar
  61. Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870–881.Google Scholar
  62. Robbins, S., Oh, I.-S., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path-analysis. Journal of Applied Psychology, 94, 1163–1184. doi: 10.1037/a0015738.PubMedCrossRefGoogle Scholar
  63. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641. doi: 10.1037/0033-2909.86.3.638.CrossRefGoogle Scholar
  64. Rothstein, H. R. (2012). Accessing relevant literature. In H. Cooper (Ed.), APA handbook of research methods in psychology: Vol. 1. Foundations, planning, measures, and psychometrics (pp. 133–144). Washington, DC: American Psychological Association.Google Scholar
  65. Rothstein, H. R., & Hopewell, S. (2009). Grey literature. In H. M. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 103–126). New York: Russell Sage Foundation.Google Scholar
  66. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005a). Publication bias in meta-analyses. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 1–7). West Sussex: Wiley.CrossRefGoogle Scholar
  67. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005b). Publication bias in meta-analysis: Prevention, assessment, and adjustments. West Sussex: Wiley.CrossRefGoogle Scholar
  68. Sackett, P. R., & Larson, J. R. (1990). Research strategies and tactics in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 419–489). Palo Alto, CA: Consulting Psychologists Press.Google Scholar
  69. Scandura, T. A., & Williams, E. A. (2000). Research methodology in management: Current practices, trends, and implications for future research. The Academy of Management Journal, 43, 1248–1264. doi: 10.2307/1556348.CrossRefGoogle Scholar
  70. Schmidt, F. L., & Le, H. (2005). Hunter & Schmidt’s Meta-analysis programs (Version 1.1). The University of Iowa, IA.Google Scholar
  71. Schmidt, F. L., Oh, I.-S., & Hayes, T. (2009). Fixed versus random effects models in meta-analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62, 97–128. doi: 10.1348/000711007X255327.PubMedCrossRefGoogle Scholar
  72. Schopfel, J. (2006). Observations on the future of grey literature. Grey Journal, 2, 67–76.Google Scholar
  73. Schulze, R. (2004). Meta-analysis: A comparison of approaches. Cambridge: Hogrefe & Huber.Google Scholar
  74. Smith, G. D., & Egger, M. (1994). Who benefits from medical interventions? British Medical Journal, 308, 72–74. doi: 10.1136/bmj.308.6921.72.PubMedCentralPubMedCrossRefGoogle Scholar
  75. Song, F., Easterwood, A., Gilbody, S., Duley, L., & Sutton, A. J. (2000). Publication and other selection biases in systematic reviews. Health Technology, 4, 1–115. doi: 10.3310/hta4100.Google Scholar
  76. Song, F., et al. (2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14, 1–220. doi: 10.3310/hta14080.Google Scholar
  77. Sterling, T. D., & Rosenbaum, W. L. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108–112. doi:http://www.tandfonline.com/doi/abs/10.1080/00031305.1995.10476125.Google Scholar
  78. Sterne, J. A., & Egger, M. (2005). Regression methods to detect publication bias and other bias in meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 99–110). West Sussex: Wiley.Google Scholar
  79. Sterne, J. A., Gavaghan, D., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 75–98). West Sussex: Wiley.Google Scholar
  80. Sterne, J. A., et al. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. British Medical Journal, 342, d4002–d4010. doi: 10.1136/bmj.d4002.CrossRefGoogle Scholar
  81. Sutton, A. J. (2005). Evidence concerning the consequences of publication and related biases. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 175–192). West Sussex: Wiley.Google Scholar
  82. Sutton, A. J. (2009). Publication bias. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435–452). New York: Russell Sage Foundation.Google Scholar
  83. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113–2126. doi: 10.1002/sim.1461.PubMedCrossRefGoogle Scholar
  84. Torgerson, C. J. (2006). Publication bias: The Achilles’ heel of systematic reviews? British Journal of Educational Studies, 54, 89–102. doi:http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8527.2006.00332.x/abstract.CrossRefGoogle Scholar
  85. Trikalinos, T. A., & Ioannidis, J. P. A. (2005). Assessing the evolution of effect sizes over time. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 241–259). West Sussex: Wiley.Google Scholar
  86. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358, 252–260. doi: 10.1056/NEJMsa065779.PubMedCrossRefGoogle Scholar
  87. Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428–443. doi: 10.1037/1082-989X.10.4.428.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  1. 1.Department of Management, School of BusinessVirginia Commonwealth UniversityRichmondUSA
  2. 2.College of Business and EconomicsLongwood UniversityFarmvilleUSA
  3. 3.Fox School of BusinessTemple UniversityPhiladelphiaUSA

Personalised recommendations