By the Numbers: Track Record, Flawed Reviews, Journal Space, and the Fate of Talented Authors
We conducted a computer simulation of hundreds of competitions for limited journal space, varying (a) the correlation between the talent of authors and the quality of their manuscripts, (b) the correlation between manuscript quality and quality judged by peer reviewers, (c) the weights reviewers and editors gave judged quality versus number of previous publications (tract record), and (d) the proportion of manuscripts accepted for publication. The results show that even small decreases in the correlations, and small increases in the weight given to track record, quickly skew the outcomes of the peer review process, favouring authors who develop a track record of publications in the first cycles of journal publication while excluding many equally-talented or more-talented authors from publishing (the Matthew Effect; Merton, 1968). Implications for declines in the quality of published manuscripts and for wasting talent are discussed.
Keywordspeer review reputation track record competition
Unable to display preview. Download preview PDF.
- Allesina, S.: Modelling peer review: An agent-based approach. Ideas in Ecology and Evolution 5 (2012)Google Scholar
- APA: Summary report of journal operations (2011), http://www.apa.org/pubs/journals/features/2011-statistics.pdf
- Bourne, P.E., Barbour, V.: Ten simple rules for building and maintaining a scientific reputation. PLoS Computational Biology: a Peer Reviewed Open-Access Journal 7(6) (2011)Google Scholar
- Grimaldo, F., Paolucci, M.: A simulation of disagreement for control of rational cheating in peer review. Advances in Complex Systems (2013), doi:10.1142/S0219525913500045Google Scholar
- Howard, G.S., Cole, D.A., Maxwell, S.E.: Research productivity in psychology based on publication in the journals of the American Psychological Association. American Psychologist 42(11), 975–986 (1987)Google Scholar
- Kaufman, A.: IQ Testing 101. Springer Publishing, New York (2009)Google Scholar
- Morgan, D.R., Meier, K.J., Kearney, R.C., Hays, S.W., Birch, H.B.: Reputation and productivity among U. S. public administration and public affairs programs. Public Administration Review 41(6), 666–673 (1981)Google Scholar
- Natural Sciences and Engineering Research Council: Guidelines for the preparation and review of applications in engineering and the applied sciences (2012), http://www.nserc-crsng.gc.ca/NSERC-CRSNG/Policies-Politiques/prepEngAS-prepGenSA_eng.asp
- R Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria (2012) ISBN 3-900051-07-0, http://www.R-project.org/
- Thorngate, W., Dawes, R.M., Foddy, M.: Judging Merit. Psychology Press, Taylor & Francis Group, New York (2009)Google Scholar
- Thorngate, W., Hotta, M., McClintock, C.: Bingo! The case for cooperation revisited. In: Tolman, C.W., Cherry, F., Van Hezewijk, R., Lubek, I. (eds.) Problems of Theoretical Psychology, Captus Press Inc., Canada (1996)Google Scholar
- Thorngate, W., Liu, J., Chowdhury, W.: The Competition for Attention and the Evolution of Science. Journal of Artificial Societies and Social Simulation 14(4) (2011)Google Scholar