Abstract
This paper investigates the extent to which staff editors’ evaluations of submitted manuscripts—that is, internal evaluations carried out before external peer reviewing—are valid. To answer this question we utilized data on the manuscript reviewing process at the journal Angewandte Chemie International Edition. The results of this study indicate that the initial internal evaluations are valid. Further, it appears that external review is indispensable for the decision on the publication worthiness of manuscripts: (1) For the majority of submitted manuscripts, staff editors are uncertain about publication worthiness; (2) there is a statistically significant proportional difference in “Rejection” between the editors' initial evaluation and the final editorial decision (after peer review); (3) three-quarters of the manuscripts that were rated negatively at the initial internal evaluation but accepted for publication after the peer review had far above-average citation counts.
This is a preview of subscription content, log in via an institution to check access.
References
Allison, P. D. (1980). Inequality and scientific productivity. Social Studies of Science, 10(2), 163–179.
Bornmann, L., & Daniel, H.-D. (2008a). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178.
Bornmann, L., & Daniel, H.-D. (2008b). Functional use of frequently and infrequently cited articles in citing publications. A content analysis of citations to articles with low and high citation counts. European Science Editing, 34(2), 35–38.
Bornmann, L., & Daniel, H.-D. (2008c). Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841–1852.
Bornmann, L., & Daniel, H.-D. (2008d). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.
Bornmann, L., & Daniel, H.-D. (2009). The luck of the referee draw: The effect of exchanging reviews. Learned Publishing, 22(2), 117–125.
Bornmann, L., & Daniel, H.-D. (2010). The manuscript reviewing process—empirical research on review requests, review sequences and decision rules in peer review. Library & Information Science Research, 32(1), 5–12.
Bornmann, L., Mutz, R., Marx, W., Schier, H., Daniel, H.-D. (submitted). A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high-impact journal select manuscripts that are highly cited after publication?
Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H.-D. (2008). Use of citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93–102.
Craig, I. D., Plume, A. M., McVeigh, M. E., Pringle, J., & Amin, M. (2007). Do open access articles have greater citation impact? A critical review of the literature. Journal of Informetrics, 1(3), 239–248.
Gölitz, P. (2005). Who is going to read all this? Angewandte Chemie International Edition, 44(35), 5538–5541.
Lokker, C., McKibbon, K. A., McKinlay, R. J., Wilczynski, N. L., & Haynes, R. B. (2008). Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study. British Medical Journal, 336(7645), 655–657.
McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12, 153–157.
StataCorp. (2007). Stata statistical software: release 10. College Station, TX, USA: Stata Corporation.
van Raan, A. F. J. (2004). Measuring science. Capita selecta of current main issues. In H. F. Moed, W. Glänzel, & U. Schmoch (Eds.), Handbook of quantitative science and technology research. The use of publication and patent statistics in studies of S&T systems (pp. 19–50). Dordrecht, The Netherlands: Kluwer Academic Publishers.
Wager, E., Parkin, E., & Tamber, P. (2006). Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study. BMC Medicine, 4(1), 13.
Acknowledgements
The broader research study, which is also investigating quality assurance of open access journals, is supported by a grant from the Max Planck Society. The authors would like to thank Dr. Christophe Weymuth (formerly at the Organic Chemistry Institute of the University of Zurich and now at BIOSYNTH AG, Switzerland) for investigation of the manuscripts rejected by Angewandte Chemie International Edition and published elsewhere. We also thank Dr. Werner Marx and Dr. Hermann Schier of the Central Information Service for the institutes of the Chemical Physical Technical (CPT) Section of the Max Planck Society (located at the Max Planck Institute for Solid State Research in Stuttgart, Germany) for conducting the search for citations of the accepted and rejected (but published elsewhere) manuscripts in the literature database Chemical Abstracts (CA). We thank Dr. Peter Gölitz, Editor-in-Chief of Angewandte Chemie, the Editorial Board of Angewandte Chemie, and the German Chemical Society (GDCh, Frankfurt am Main, Germany) for permission to conduct the evaluation of the selection process of the journal and thank the members of the editorial office for their generous support during the carrying out of the study.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bornmann, L., Daniel, HD. The validity of staff editors’ initial evaluations of manuscripts: a case study of Angewandte Chemie International Edition . Scientometrics 85, 681–687 (2010). https://doi.org/10.1007/s11192-010-0215-7
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-010-0215-7