Quality & Quantity

, Volume 50, Issue 2, pp 901–917 | Cite as

Publication bias in the German social sciences: an application of the caliper test to three top-tier German social science journals

  • Carl C. Berning
  • Bernd Weiß


Systematic research reviews have become essential in all empirical sciences. However, the validity of research syntheses is threatened if the preparation, submission or publication of research findings depends on the statistical significance of these findings. The present study investigates publication bias in three top-tier journals in the German social sciences, utilizing the caliper test. For the period between 2001 and 2010, we have collected 156 articles that appeared in the Kölner Zeitschrift für Soziologie und Sozialpsychologie (KZfSS), the Zeitschrift für Soziologie (ZfS) and the Politische Vierteljahresschrift (PVS). In all three journals, we found empirical evidence for the existence of publication bias at the 10 % significance level. We also investigated possible causes linked to this bias, including single versus multiple authorship as well as academic degree. We found only weak support for the relationships between individual author characteristics and publication bias.


Publication bias Multilevel analysis Statistical significance Caliper test Type I error 



We are deeply indebted to Christopher G. Thompson, Michael Wagner and William H. Yeaton, for their helpful and constructive comments on an earlier version of this paper.


  1. Aickin, M., Gensler, H.: Adjusting for multiple testing when reporting research results: the Bonferroni versus Holm methods. Am. J. Public Health 86(5), 726–728 (1996)CrossRefGoogle Scholar
  2. ASA: Code of Ethics and Policies and Procedures of the ASA Committee in Professional Ethics. American Sociological Association, Washington (1999)Google Scholar
  3. Auspurg, K., Hinz, T.: What fuels publication bias? theoretical and empirical analyses of risk factors using the caliper test. Jahrbücher für Nationalökonomie und Statistik 231(5–6), 636–660 (2011)Google Scholar
  4. Auspurg, K., Hinz, T., Schneck, A.: Ausmaß und Risikofaktoren des Publication Bias in der deutschen Soziologie. Kölner Zeitschrift für Soziologie und Sozialpsychologie 66(4), 549–573 (2014)CrossRefGoogle Scholar
  5. Coursol, A., Wagner, E.E.: Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Prof. Psychol. 17(2), 136–137 (1986)CrossRefGoogle Scholar
  6. Dickersin, K.: Publication Bias: Recognizing the Problem, Understanding its Origins and Scope, and Preventing Harm. In: Rothstein, H., Sutton, A., Borenstein, M. (eds.) Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments, pp. 11–33. Wiley, Chichester (2005)Google Scholar
  7. Dwan, K., Altman, D.G., Arnaiz, J.A., Bloom, J., Chan, A.W., Cronin, E., Decullier, E., Easterbrook, P.J., Von Elm, E., Gamble, C., Ghersi, D., Ioannidis, J.P., Simes, J., Williamson, P.R.: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 3(8), e3081 (2008)CrossRefGoogle Scholar
  8. Fanelli, D.: Do pressures to publish increase scientists’ bias? an empirical support from US states data. PLoS One 5(4), e10271 (2010a)CrossRefGoogle Scholar
  9. Fanelli, D.: “Positive” results increase down the hierarchy of the sciences. PLoS One 5(4), e10068 (2010b)CrossRefGoogle Scholar
  10. Fanelli, D.: Negative results are disappearing from most disciplines and countries. Scientometrics 90(3), 891–904 (2011)CrossRefGoogle Scholar
  11. Federkeil, G., Buch, F.: Fünf Jahre Juniorprofessur—Zweite CHE Befragung zum Stand der Einführung. Arbeitspapier Nr. 90 (2007)Google Scholar
  12. Fisher, R.A.: Statistical tests. Nature 136(3438), 474 (1935)CrossRefGoogle Scholar
  13. Formann, A.K.: Estimating the proportion of studies missing for meta analysis due to publication bias. Contemp. Clin. Trials 29(5), 732–739 (2008)CrossRefGoogle Scholar
  14. Freeman, R., Weinstein, E., Marincola, E., Rosenbaum, J., Solomon, F.: Competition and careers in biosciences. Science 294(5550), 2293–2294 (2001)CrossRefGoogle Scholar
  15. Gerber, A.S., Malhotra, N. (2006): Can political science literatures be believed? A study of publication bias in the APSR and the AJPS. Presented at the annual meeting of the Midwest Political Science Association, April 20–23, Chicago, IL. Google Scholar
  16. Gerber, A.S., Malhotra, N.: Publication bias in empirical sociological research. Do arbitrary significance levels distort published results? Sociol. Methods Res. 37(1), 3–30 (2008)CrossRefGoogle Scholar
  17. Graber, M., Launov, A., Waelde, K.: Publish or perish? The increasing importance of publications for prospective economics professors in Austria, Germany and Switzerland. Ger. Econ. Rev. 9(4), 457–472 (2008)CrossRefGoogle Scholar
  18. Haller, H., Krauss, K.: Misinterpretations of significance: a problem students share with their teachers? Methods Psychol. Res. 7(1), 1–20 (2002)Google Scholar
  19. Hills, P.J.: Publish or Perish: A Guide to Academic Authors. Hyperion Books, Dereham (1987)Google Scholar
  20. Hirschhauer, S.: Peer review Verfahren auf dem Prüfstand. Zeitschrift für Soziologie 33(1), 62–83 (2004)Google Scholar
  21. Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6(2), 65–70 (1979)Google Scholar
  22. Journal Citation Reports®. Social Sciences Edition. Thomson Reuters, 2010. (2009)Google Scholar
  23. Kerr, N.L.: HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. Rev. 2(3), 196–217 (1998)CrossRefGoogle Scholar
  24. Larsen, P.O., von Ins, M.: The rate of growth in scientific publication and the decline in coverage provided by science citation index. Scientometrics 84(3), 575–603 (2010)CrossRefGoogle Scholar
  25. Lawrence, P.: The politics of publication. Nature 422(6929), 259–261 (2003)CrossRefGoogle Scholar
  26. Lipsey, M.W., Wilson, D.: Practical Meta-Analysis. Sage Publications, New York (2001)Google Scholar
  27. Long, J.B. De, Lang, K.: Are all economic hypotheses false? J. Polit. Econ. 100(6), 1257–1272 (1992)CrossRefGoogle Scholar
  28. McNemar, Q.: At random: Sense and nonsense. Am. Psychol. 15(5), 295–300 (1960)CrossRefGoogle Scholar
  29. Mood, C.: Logistic regression: Why we cannot do what we think we can do, and what we can do about it. Eur. Sociol. Rev. 26(1), 67–82 (2010)CrossRefGoogle Scholar
  30. Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: part I. Biometrika 20A(1/2), 175–240 (1928a)CrossRefGoogle Scholar
  31. Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: part II. Biometrika 20A(3/4), 263–294 (1928b)CrossRefGoogle Scholar
  32. Neyman, J., Pearson, E.S.: On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. Lond. 231, 289–337 (1933)CrossRefGoogle Scholar
  33. Nosek, B.A., Spies, J.R., Motyl, M.: Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7(6), 615–631 (2012)CrossRefGoogle Scholar
  34. Oakes, M.: Statistical Inference: A Commentary for the Social and Behavioral Sciences. Wiley, New York (1986)Google Scholar
  35. Plümper, T., Schimmelfennig, F.: Wer wird Prof — und wann? Berufungsdeterminanten in der deutschen Politikwissenschaft. Politische Vierteljahresschrift 48(1), 97–117 (2007)CrossRefGoogle Scholar
  36. R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2015). Version 3.1.2Google Scholar
  37. Rosenthal, R.: The file drawer problem and tolerance for null results. Psychol. Bull. 86(3), 638–641 (1979)CrossRefGoogle Scholar
  38. Rothstein, H., Sutton, A., Borenstein, M.: Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments. Wiley, Chichester (2005)CrossRefGoogle Scholar
  39. Skipper, J.K., Guenther, A.L., Nass, G.: The Sacredness of 05: a note concerning the uses of statistical levels of significance in social science. Am. Sociol. 2(1), 16–18 (1967)Google Scholar
  40. Sterling, T.D., Rosenbaum, W.L., Weinkam, J.J.: Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. Am. Stat. 49(1), 108–112 (1995)Google Scholar
  41. Weiß, B., Wagner, M.: The identification and prevention of publication bias in the social sciences and economics. Jahrbücher für Nationalökonomie und Statistik 231(5–6), 661–684 (2011)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  1. 1.Department of Political ScienceJohannes Gutenberg University of MainzMainzGermany
  2. 2.Institute of Sociology and Social PsychologyUniversity of CologneCologneGermany

Personalised recommendations