Skip to main content
Log in

Publication bias in the German social sciences: an application of the caliper test to three top-tier German social science journals

  • Published:
Quality & Quantity Aims and scope Submit manuscript

An Erratum to this article was published on 01 July 2015

Abstract

Systematic research reviews have become essential in all empirical sciences. However, the validity of research syntheses is threatened if the preparation, submission or publication of research findings depends on the statistical significance of these findings. The present study investigates publication bias in three top-tier journals in the German social sciences, utilizing the caliper test. For the period between 2001 and 2010, we have collected 156 articles that appeared in the Kölner Zeitschrift für Soziologie und Sozialpsychologie (KZfSS), the Zeitschrift für Soziologie (ZfS) and the Politische Vierteljahresschrift (PVS). In all three journals, we found empirical evidence for the existence of publication bias at the 10 % significance level. We also investigated possible causes linked to this bias, including single versus multiple authorship as well as academic degree. We found only weak support for the relationships between individual author characteristics and publication bias.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Engl. Cologne Journal of Sociology and Social Psychology.

  2. Engl. Journal of Sociology.

  3. Journal of the German Association for Political Science.

  4. We collected the reported significance levels of three volumes (53; 57; 62) for the KZfSS and find that around 43 % of all articles that report statistical significance introduce the 10 % significance threshold.

  5. Hypothesizing after the results are known.

  6. Journal Impact Factor = average citations for an article published in one journal for the last 2 years.

  7. 5-Year Impact Factor: KZfSS = 1.308; ZfS = 0.952; PVS = 0.798.

  8. Book reviews, obituaries and comments are not part of the gross sample.

  9. “Signifikan” accounts for the German word for significance (Signifikanz) and significant (signifikant).

  10. The character “*” is not used as a wildcard character.

  11. The habilitation is the highest qualification a researcher can obtain. It is the necessary requirement for a professorship in Germany.

  12. Note that relevant to our theoretical assumption is the academic degree at the time of submission, not the academic degree at the time of publication. Since the actual date of submission was not available to us, we used a conservative measure with a one year time lag. To test this measure for robustness we reran our analyses with the academic degree at the time of publication with substantively identical results.

  13. uc = under caliper and oc = over caliper.

  14. These result patterns disappear when we include the four publications which reported more than 94 effects. In total, these four publication contributed 484 effects to the sample, having in mind that these are all hypotheses testing coefficients and no control variables. This means: 2.5 % of all articles provide 17.1 % of all effects. Beyond this numerical mismatch, it is reasonable to believe that the pressure to present significant findings negatively correlates with the number of findings within a paper. Our findings presented in Table 6 generally support this assumption. To our knowledge there is no statistical package available that allows weighting for a binominal distribution tests.

References

  • Aickin, M., Gensler, H.: Adjusting for multiple testing when reporting research results: the Bonferroni versus Holm methods. Am. J. Public Health 86(5), 726–728 (1996)

    Article  Google Scholar 

  • ASA: Code of Ethics and Policies and Procedures of the ASA Committee in Professional Ethics. American Sociological Association, Washington (1999)

    Google Scholar 

  • Auspurg, K., Hinz, T.: What fuels publication bias? theoretical and empirical analyses of risk factors using the caliper test. Jahrbücher für Nationalökonomie und Statistik 231(5–6), 636–660 (2011)

    Google Scholar 

  • Auspurg, K., Hinz, T., Schneck, A.: Ausmaß und Risikofaktoren des Publication Bias in der deutschen Soziologie. Kölner Zeitschrift für Soziologie und Sozialpsychologie 66(4), 549–573 (2014)

    Article  Google Scholar 

  • Coursol, A., Wagner, E.E.: Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Prof. Psychol. 17(2), 136–137 (1986)

    Article  Google Scholar 

  • Dickersin, K.: Publication Bias: Recognizing the Problem, Understanding its Origins and Scope, and Preventing Harm. In: Rothstein, H., Sutton, A., Borenstein, M. (eds.) Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments, pp. 11–33. Wiley, Chichester (2005)

    Google Scholar 

  • Dwan, K., Altman, D.G., Arnaiz, J.A., Bloom, J., Chan, A.W., Cronin, E., Decullier, E., Easterbrook, P.J., Von Elm, E., Gamble, C., Ghersi, D., Ioannidis, J.P., Simes, J., Williamson, P.R.: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 3(8), e3081 (2008)

    Article  Google Scholar 

  • Fanelli, D.: Do pressures to publish increase scientists’ bias? an empirical support from US states data. PLoS One 5(4), e10271 (2010a)

    Article  Google Scholar 

  • Fanelli, D.: “Positive” results increase down the hierarchy of the sciences. PLoS One 5(4), e10068 (2010b)

    Article  Google Scholar 

  • Fanelli, D.: Negative results are disappearing from most disciplines and countries. Scientometrics 90(3), 891–904 (2011)

    Article  Google Scholar 

  • Federkeil, G., Buch, F.: Fünf Jahre Juniorprofessur—Zweite CHE Befragung zum Stand der Einführung. Arbeitspapier Nr. 90 (2007)

  • Fisher, R.A.: Statistical tests. Nature 136(3438), 474 (1935)

    Article  Google Scholar 

  • Formann, A.K.: Estimating the proportion of studies missing for meta analysis due to publication bias. Contemp. Clin. Trials 29(5), 732–739 (2008)

    Article  Google Scholar 

  • Freeman, R., Weinstein, E., Marincola, E., Rosenbaum, J., Solomon, F.: Competition and careers in biosciences. Science 294(5550), 2293–2294 (2001)

    Article  Google Scholar 

  • Gerber, A.S., Malhotra, N. (2006): Can political science literatures be believed? A study of publication bias in the APSR and the AJPS. Presented at the annual meeting of the Midwest Political Science Association, April 20–23, Chicago, IL.

  • Gerber, A.S., Malhotra, N.: Publication bias in empirical sociological research. Do arbitrary significance levels distort published results? Sociol. Methods Res. 37(1), 3–30 (2008)

    Article  Google Scholar 

  • Graber, M., Launov, A., Waelde, K.: Publish or perish? The increasing importance of publications for prospective economics professors in Austria, Germany and Switzerland. Ger. Econ. Rev. 9(4), 457–472 (2008)

    Article  Google Scholar 

  • Haller, H., Krauss, K.: Misinterpretations of significance: a problem students share with their teachers? Methods Psychol. Res. 7(1), 1–20 (2002)

    Google Scholar 

  • Hills, P.J.: Publish or Perish: A Guide to Academic Authors. Hyperion Books, Dereham (1987)

    Google Scholar 

  • Hirschhauer, S.: Peer review Verfahren auf dem Prüfstand. Zeitschrift für Soziologie 33(1), 62–83 (2004)

    Google Scholar 

  • Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6(2), 65–70 (1979)

    Google Scholar 

  • Journal Citation Reports®. Social Sciences Edition. Thomson Reuters, 2010. (2009)

  • Kerr, N.L.: HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. Rev. 2(3), 196–217 (1998)

    Article  Google Scholar 

  • Larsen, P.O., von Ins, M.: The rate of growth in scientific publication and the decline in coverage provided by science citation index. Scientometrics 84(3), 575–603 (2010)

    Article  Google Scholar 

  • Lawrence, P.: The politics of publication. Nature 422(6929), 259–261 (2003)

    Article  Google Scholar 

  • Lipsey, M.W., Wilson, D.: Practical Meta-Analysis. Sage Publications, New York (2001)

    Google Scholar 

  • Long, J.B. De, Lang, K.: Are all economic hypotheses false? J. Polit. Econ. 100(6), 1257–1272 (1992)

    Article  Google Scholar 

  • McNemar, Q.: At random: Sense and nonsense. Am. Psychol. 15(5), 295–300 (1960)

    Article  Google Scholar 

  • Mood, C.: Logistic regression: Why we cannot do what we think we can do, and what we can do about it. Eur. Sociol. Rev. 26(1), 67–82 (2010)

    Article  Google Scholar 

  • Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: part I. Biometrika 20A(1/2), 175–240 (1928a)

    Article  Google Scholar 

  • Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: part II. Biometrika 20A(3/4), 263–294 (1928b)

    Article  Google Scholar 

  • Neyman, J., Pearson, E.S.: On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. Lond. 231, 289–337 (1933)

    Article  Google Scholar 

  • Nosek, B.A., Spies, J.R., Motyl, M.: Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7(6), 615–631 (2012)

    Article  Google Scholar 

  • Oakes, M.: Statistical Inference: A Commentary for the Social and Behavioral Sciences. Wiley, New York (1986)

    Google Scholar 

  • Plümper, T., Schimmelfennig, F.: Wer wird Prof — und wann? Berufungsdeterminanten in der deutschen Politikwissenschaft. Politische Vierteljahresschrift 48(1), 97–117 (2007)

    Article  Google Scholar 

  • R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2015). Version 3.1.2

    Google Scholar 

  • Rosenthal, R.: The file drawer problem and tolerance for null results. Psychol. Bull. 86(3), 638–641 (1979)

    Article  Google Scholar 

  • Rothstein, H., Sutton, A., Borenstein, M.: Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments. Wiley, Chichester (2005)

    Book  Google Scholar 

  • Skipper, J.K., Guenther, A.L., Nass, G.: The Sacredness of 05: a note concerning the uses of statistical levels of significance in social science. Am. Sociol. 2(1), 16–18 (1967)

    Google Scholar 

  • Sterling, T.D., Rosenbaum, W.L., Weinkam, J.J.: Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. Am. Stat. 49(1), 108–112 (1995)

    Google Scholar 

  • Weiß, B., Wagner, M.: The identification and prevention of publication bias in the social sciences and economics. Jahrbücher für Nationalökonomie und Statistik 231(5–6), 661–684 (2011)

    Google Scholar 

Download references

Acknowledgments

We are deeply indebted to Christopher G. Thompson, Michael Wagner and William H. Yeaton, for their helpful and constructive comments on an earlier version of this paper.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Carl C. Berning.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Berning, C.C., Weiß, B. Publication bias in the German social sciences: an application of the caliper test to three top-tier German social science journals. Qual Quant 50, 901–917 (2016). https://doi.org/10.1007/s11135-015-0182-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-015-0182-4

Keywords

Navigation