By the Numbers: Track Record, Flawed Reviews, Journal Space, and the Fate of Talented Authors

Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 229)

Abstract

We conducted a computer simulation of hundreds of competitions for limited journal space, varying (a) the correlation between the talent of authors and the quality of their manuscripts, (b) the correlation between manuscript quality and quality judged by peer reviewers, (c) the weights reviewers and editors gave judged quality versus number of previous publications (tract record), and (d) the proportion of manuscripts accepted for publication. The results show that even small decreases in the correlations, and small increases in the weight given to track record, quickly skew the outcomes of the peer review process, favouring authors who develop a track record of publications in the first cycles of journal publication while excluding many equally-talented or more-talented authors from publishing (the Matthew Effect; Merton, 1968). Implications for declines in the quality of published manuscripts and for wasting talent are discussed.

Keywords

peer review reputation track record competition 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Allesina, S.: Modelling peer review: An agent-based approach. Ideas in Ecology and Evolution 5 (2012)Google Scholar
  2. APA: Summary report of journal operations (2011), http://www.apa.org/pubs/journals/features/2011-statistics.pdf
  3. Bornmann, L., Daniel, H.D.: Gatekeepers of science – Effect of external reviewers’ attributes on the assessment of fellowship applications. Journal of Informetrics 1, 83–91 (2007)CrossRefGoogle Scholar
  4. Bourne, P.E., Barbour, V.: Ten simple rules for building and maintaining a scientific reputation. PLoS Computational Biology: a Peer Reviewed Open-Access Journal 7(6) (2011)Google Scholar
  5. Day, A., Peters, J.: Quality Indicators in Academic Publishing. Library Review 43(7), 4–72 (1994)CrossRefGoogle Scholar
  6. Dewett, T., Denisi, A.S.: Exploring scholarly reputation: It’s more than just productivity. Scientometrics 60(2), 249–272 (2004)CrossRefGoogle Scholar
  7. Ferber, M.A.: Citations: Are they an objective measure of scholarly merit? Signs 11(2), 381–389 (1986)CrossRefGoogle Scholar
  8. Grimaldo, F., Paolucci, M.: A simulation of disagreement for control of rational cheating in peer review. Advances in Complex Systems (2013), doi:10.1142/S0219525913500045Google Scholar
  9. Herron, D.: Is expert peer review obsolete? Surgical Endoscopy 26, 2275–2280 (2012)CrossRefGoogle Scholar
  10. Howard, G.S., Cole, D.A., Maxwell, S.E.: Research productivity in psychology based on publication in the journals of the American Psychological Association. American Psychologist 42(11), 975–986 (1987)Google Scholar
  11. Kaufman, A.: IQ Testing 101. Springer Publishing, New York (2009)Google Scholar
  12. Merton, R.K.: The Matthew Effect in Science. Science 159(3810), 56–63 (1968)CrossRefGoogle Scholar
  13. Morgan, D.R., Meier, K.J., Kearney, R.C., Hays, S.W., Birch, H.B.: Reputation and productivity among U. S. public administration and public affairs programs. Public Administration Review 41(6), 666–673 (1981)Google Scholar
  14. Natural Sciences and Engineering Research Council: Guidelines for the preparation and review of applications in engineering and the applied sciences (2012), http://www.nserc-crsng.gc.ca/NSERC-CRSNG/Policies-Politiques/prepEngAS-prepGenSA_eng.asp
  15. Ofori-Dankwa, J., Julian, S.: From thought to theory to school: The role of contextual factors in the evolution of schools of management thought. Organization Studies 26(9), 1307–1329 (2005)CrossRefGoogle Scholar
  16. Petty, R., Fleming, M., Fabrigar, L.: The review process at PSPB: Correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin 25(2), 188–203 (1999)CrossRefGoogle Scholar
  17. R Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria (2012) ISBN 3-900051-07-0, http://www.R-project.org/
  18. Squazzoni, F., Gandelli, C.: Saint Matthew strikes again: An agent-based model of peer review and he scientific community structure. Journal of Informetrics 6, 265–275 (2012)CrossRefGoogle Scholar
  19. Thorngate, W.: On the evolution of adjudicated contests and the principle of invidious selection. Journal of Behavioral Decision Making 1, 5–16 (1988)CrossRefGoogle Scholar
  20. Thorngate, W., Dawes, R.M., Foddy, M.: Judging Merit. Psychology Press, Taylor & Francis Group, New York (2009)Google Scholar
  21. Thorngate, W., Hotta, M.: Life and luck: Survival of the fattest. Simulation & Gaming 26, 5–16 (1995)CrossRefGoogle Scholar
  22. Thorngate, W., Hotta, M., McClintock, C.: Bingo! The case for cooperation revisited. In: Tolman, C.W., Cherry, F., Van Hezewijk, R., Lubek, I. (eds.) Problems of Theoretical Psychology, Captus Press Inc., Canada (1996)Google Scholar
  23. Thorngate, W., Liu, J., Chowdhury, W.: The Competition for Attention and the Evolution of Science. Journal of Artificial Societies and Social Simulation 14(4) (2011)Google Scholar
  24. Whitehurst, G.J.: Interrater agreement for journal manuscript reviews. American Psychologist 39(1), 22–28 (1984)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.Psychology Department and Institute of Cognitive ScienceCarleton UniversityOttawaCanada

Personalised recommendations