Abstract
Systematic research reviews have become essential in all empirical sciences. However, the validity of research syntheses is threatened if the preparation, submission or publication of research findings depends on the statistical significance of these findings. The present study investigates publication bias in three top-tier journals in the German social sciences, utilizing the caliper test. For the period between 2001 and 2010, we have collected 156 articles that appeared in the Kölner Zeitschrift für Soziologie und Sozialpsychologie (KZfSS), the Zeitschrift für Soziologie (ZfS) and the Politische Vierteljahresschrift (PVS). In all three journals, we found empirical evidence for the existence of publication bias at the 10 % significance level. We also investigated possible causes linked to this bias, including single versus multiple authorship as well as academic degree. We found only weak support for the relationships between individual author characteristics and publication bias.
Similar content being viewed by others
Notes
Engl. Cologne Journal of Sociology and Social Psychology.
Engl. Journal of Sociology.
Journal of the German Association for Political Science.
We collected the reported significance levels of three volumes (53; 57; 62) for the KZfSS and find that around 43 % of all articles that report statistical significance introduce the 10 % significance threshold.
Hypothesizing after the results are known.
Journal Impact Factor = average citations for an article published in one journal for the last 2 years.
5-Year Impact Factor: KZfSS = 1.308; ZfS = 0.952; PVS = 0.798.
Book reviews, obituaries and comments are not part of the gross sample.
“Signifikan” accounts for the German word for significance (Signifikanz) and significant (signifikant).
The character “*” is not used as a wildcard character.
The habilitation is the highest qualification a researcher can obtain. It is the necessary requirement for a professorship in Germany.
Note that relevant to our theoretical assumption is the academic degree at the time of submission, not the academic degree at the time of publication. Since the actual date of submission was not available to us, we used a conservative measure with a one year time lag. To test this measure for robustness we reran our analyses with the academic degree at the time of publication with substantively identical results.
uc = under caliper and oc = over caliper.
These result patterns disappear when we include the four publications which reported more than 94 effects. In total, these four publication contributed 484 effects to the sample, having in mind that these are all hypotheses testing coefficients and no control variables. This means: 2.5 % of all articles provide 17.1 % of all effects. Beyond this numerical mismatch, it is reasonable to believe that the pressure to present significant findings negatively correlates with the number of findings within a paper. Our findings presented in Table 6 generally support this assumption. To our knowledge there is no statistical package available that allows weighting for a binominal distribution tests.
References
Aickin, M., Gensler, H.: Adjusting for multiple testing when reporting research results: the Bonferroni versus Holm methods. Am. J. Public Health 86(5), 726–728 (1996)
ASA: Code of Ethics and Policies and Procedures of the ASA Committee in Professional Ethics. American Sociological Association, Washington (1999)
Auspurg, K., Hinz, T.: What fuels publication bias? theoretical and empirical analyses of risk factors using the caliper test. Jahrbücher für Nationalökonomie und Statistik 231(5–6), 636–660 (2011)
Auspurg, K., Hinz, T., Schneck, A.: Ausmaß und Risikofaktoren des Publication Bias in der deutschen Soziologie. Kölner Zeitschrift für Soziologie und Sozialpsychologie 66(4), 549–573 (2014)
Coursol, A., Wagner, E.E.: Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Prof. Psychol. 17(2), 136–137 (1986)
Dickersin, K.: Publication Bias: Recognizing the Problem, Understanding its Origins and Scope, and Preventing Harm. In: Rothstein, H., Sutton, A., Borenstein, M. (eds.) Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments, pp. 11–33. Wiley, Chichester (2005)
Dwan, K., Altman, D.G., Arnaiz, J.A., Bloom, J., Chan, A.W., Cronin, E., Decullier, E., Easterbrook, P.J., Von Elm, E., Gamble, C., Ghersi, D., Ioannidis, J.P., Simes, J., Williamson, P.R.: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 3(8), e3081 (2008)
Fanelli, D.: Do pressures to publish increase scientists’ bias? an empirical support from US states data. PLoS One 5(4), e10271 (2010a)
Fanelli, D.: “Positive” results increase down the hierarchy of the sciences. PLoS One 5(4), e10068 (2010b)
Fanelli, D.: Negative results are disappearing from most disciplines and countries. Scientometrics 90(3), 891–904 (2011)
Federkeil, G., Buch, F.: Fünf Jahre Juniorprofessur—Zweite CHE Befragung zum Stand der Einführung. Arbeitspapier Nr. 90 (2007)
Fisher, R.A.: Statistical tests. Nature 136(3438), 474 (1935)
Formann, A.K.: Estimating the proportion of studies missing for meta analysis due to publication bias. Contemp. Clin. Trials 29(5), 732–739 (2008)
Freeman, R., Weinstein, E., Marincola, E., Rosenbaum, J., Solomon, F.: Competition and careers in biosciences. Science 294(5550), 2293–2294 (2001)
Gerber, A.S., Malhotra, N. (2006): Can political science literatures be believed? A study of publication bias in the APSR and the AJPS. Presented at the annual meeting of the Midwest Political Science Association, April 20–23, Chicago, IL.
Gerber, A.S., Malhotra, N.: Publication bias in empirical sociological research. Do arbitrary significance levels distort published results? Sociol. Methods Res. 37(1), 3–30 (2008)
Graber, M., Launov, A., Waelde, K.: Publish or perish? The increasing importance of publications for prospective economics professors in Austria, Germany and Switzerland. Ger. Econ. Rev. 9(4), 457–472 (2008)
Haller, H., Krauss, K.: Misinterpretations of significance: a problem students share with their teachers? Methods Psychol. Res. 7(1), 1–20 (2002)
Hills, P.J.: Publish or Perish: A Guide to Academic Authors. Hyperion Books, Dereham (1987)
Hirschhauer, S.: Peer review Verfahren auf dem Prüfstand. Zeitschrift für Soziologie 33(1), 62–83 (2004)
Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6(2), 65–70 (1979)
Journal Citation Reports®. Social Sciences Edition. Thomson Reuters, 2010. (2009)
Kerr, N.L.: HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. Rev. 2(3), 196–217 (1998)
Larsen, P.O., von Ins, M.: The rate of growth in scientific publication and the decline in coverage provided by science citation index. Scientometrics 84(3), 575–603 (2010)
Lawrence, P.: The politics of publication. Nature 422(6929), 259–261 (2003)
Lipsey, M.W., Wilson, D.: Practical Meta-Analysis. Sage Publications, New York (2001)
Long, J.B. De, Lang, K.: Are all economic hypotheses false? J. Polit. Econ. 100(6), 1257–1272 (1992)
McNemar, Q.: At random: Sense and nonsense. Am. Psychol. 15(5), 295–300 (1960)
Mood, C.: Logistic regression: Why we cannot do what we think we can do, and what we can do about it. Eur. Sociol. Rev. 26(1), 67–82 (2010)
Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: part I. Biometrika 20A(1/2), 175–240 (1928a)
Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: part II. Biometrika 20A(3/4), 263–294 (1928b)
Neyman, J., Pearson, E.S.: On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. Lond. 231, 289–337 (1933)
Nosek, B.A., Spies, J.R., Motyl, M.: Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7(6), 615–631 (2012)
Oakes, M.: Statistical Inference: A Commentary for the Social and Behavioral Sciences. Wiley, New York (1986)
Plümper, T., Schimmelfennig, F.: Wer wird Prof — und wann? Berufungsdeterminanten in der deutschen Politikwissenschaft. Politische Vierteljahresschrift 48(1), 97–117 (2007)
R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2015). Version 3.1.2
Rosenthal, R.: The file drawer problem and tolerance for null results. Psychol. Bull. 86(3), 638–641 (1979)
Rothstein, H., Sutton, A., Borenstein, M.: Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments. Wiley, Chichester (2005)
Skipper, J.K., Guenther, A.L., Nass, G.: The Sacredness of 05: a note concerning the uses of statistical levels of significance in social science. Am. Sociol. 2(1), 16–18 (1967)
Sterling, T.D., Rosenbaum, W.L., Weinkam, J.J.: Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. Am. Stat. 49(1), 108–112 (1995)
Weiß, B., Wagner, M.: The identification and prevention of publication bias in the social sciences and economics. Jahrbücher für Nationalökonomie und Statistik 231(5–6), 661–684 (2011)
Acknowledgments
We are deeply indebted to Christopher G. Thompson, Michael Wagner and William H. Yeaton, for their helpful and constructive comments on an earlier version of this paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Berning, C.C., Weiß, B. Publication bias in the German social sciences: an application of the caliper test to three top-tier German social science journals. Qual Quant 50, 901–917 (2016). https://doi.org/10.1007/s11135-015-0182-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-015-0182-4