Skip to main content
SpringerLink
Log in

Building a Data-Driven Model of Peer Review: The Case of Science Foundation Ireland

  • Conference paper
  • First Online:
Advances in Social Simulation (ESSA 2019)

Part of the book series: Springer Proceedings in Complexity ((SPCOM))

Included in the following conference series:

  • 465 Accesses

Abstract

Research has long questioned the validity and reliability of peer review, the process for selecting manuscripts for publication and research proposals for funding. For example, scholars have shown that reviewers do not interpret evaluation criteria in the same way [1] and produce inconsistent ratings [2], and that peer review is subject to gender, ethnicity, seniority, and reputation biases [8, 11].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 219.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 279.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 279.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. H. Abdoul, C. Perrey, P. Amiel, F. Tubach, S. Gottot, I. Durand-Zaleski, C. Alberti, Peer review of grant applications: criteria used and qualitative study of reviewer practices. PLoS ONE 7(9), e46054 (2012)

    Article  Google Scholar 

  2. L. Bornmann, R. Mutz, H.-D. Daniel, A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE 5(12), e14331 (2010)

    Article  Google Scholar 

  3. S. Elsawah, J.H.A. Guillaume, T. Filatova, J. Rook, A.J. Jakeman, A methodology for eliciting, representing, and analysing stakeholder knowledge for decision making on complex socio-ecological systems: from cognitive maps to agent-based models. J. Environ. Manag. 151, 500–516 (2015)

    Article  Google Scholar 

  4. T. Feliciani, J. Luo, L. Ma, P. Lucas, F. Squazzoni, A. Marušić, K. Shankar, A scoping review of simulation models of peer review. Scientometrics (2019)

    Google Scholar 

  5. F. Grimaldo, M. Paolucci, J. Sabater-Mir, Reputation or peer review? The role of outliers. Scientometrics 116, 1421 (2018)

    Article  Google Scholar 

  6. S. Hassan, J. Pavón, L. Antunes, N. Gilbert, Injecting data into agent-based simulation, in Simulating Interacting Agents and Social Phenomena, eds. by K. Takadama, C. Cioffi-Revilla, G. Deffuant (2010), pp. 177–191

    Google Scholar 

  7. P. Hedström, G. Manzo, Recent trends in agent-based computational research: a brief introduction. Sociol. Methods Res. 44(2), 179–185 (2015)

    Article  MathSciNet  Google Scholar 

  8. C.J. Lee, D. Moher, Promote scientific integrity via journal peer review data. Science 357(6348), 256–257 (2017)

    Article  Google Scholar 

  9. C.J. Lee, C.R. Sugimoto, G. Zhang, B. Cronin, Bias in peer review. J. Am. Soc. Inform. Sci. Technol. 64(1), 2–17 (2013)

    Article  Google Scholar 

  10. J.D. Linton, Improving the peer review process: capturing more information and enabling high-risk/high-return research. Res. Policy 45(9), 1936–1938 (2016)

    Article  Google Scholar 

  11. Lyon A, Morreau M, The wisdom of collective grading and the effects of epistemic and semantic diversity, in Theory and Decision (2017), pp. 1–18

    Google Scholar 

  12. H.W. Marsh, U.W. Jayasinghe, N.W. Bond, Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. Am. Psychol. 63(3), 160–168 (2008)

    Article  Google Scholar 

  13. S. Righi, K. Takács, The miracle of peer review and development in science: an agent-based model. Scientometrics 113(1), 587–607 (2017)

    Article  Google Scholar 

  14. P.-O. Siebers, F. Klügl, What software engineering has to offer to agent-based social simulation, in Simulating Social Complexity, eds. by B. Edmonds, R. Meyer (2017), pp. 81–117

    Google Scholar 

  15. F. Squazzoni, E. Brezis, A. Marušić, Scientometrics of peer review. Scientometrics 113(1), 501–502 (2017)

    Article  Google Scholar 

  16. F. Squazzoni, K. Takács, Social simulation that “peers into peer review. J. Artif. Soc Soc. Simul. 14(4) (2011)

    Google Scholar 

  17. A.L. Stinchcombe, R. Ofshe, On journal editing as a probabilistic process. Am. Sociol. 4(2), 116–117 (1969)

    Google Scholar 

Download references

Acknowledgements

This material is based upon works supported by the Science Foundation Ireland under Grant No.17/SPR/5319.

Author information

Authors and Affiliations

  1. School of Sociology and Geary Institute for Public Policy, University College Dublin, Dublin, Ireland

    Thomas Feliciani & Pablo Lucas

  2. School of Information and Communication Studies and Geary Institute for Public Policy, University College Dublin, Dublin, Ireland

    Junwen Luo & Kalpana Shankar

Authors
  1. Thomas Feliciani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pablo Lucas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junwen Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kalpana Shankar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Feliciani .

Editor information

Editors and Affiliations

  1. Johannes Gutenberg University of Mainz, Mainz, Rheinland-Pfalz, Germany

    Petra Ahrweiler

  2. Johannes Gutenberg University of Mainz, Mainz, Rheinland-Pfalz, Germany

    Martin Neumann

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Feliciani, T., Lucas, P., Luo, J., Shankar, K. (2021). Building a Data-Driven Model of Peer Review: The Case of Science Foundation Ireland. In: Ahrweiler, P., Neumann, M. (eds) Advances in Social Simulation. ESSA 2019. Springer Proceedings in Complexity. Springer, Cham. https://doi.org/10.1007/978-3-030-61503-1_21

Download citation

Publish with us

Policies and ethics

Search

Navigation