Abstract
Peer evaluation of research grant applications is a crucial step in the funding decisions of many science funding agencies. Funding bodies take various measures to increase the independence and quality of this process, sometimes leading to difficult combinatorial problems. We propose a novel method based on network flow theory to find assignments of evaluators to grant applications that obey the rules formulated by the Slovak Research and Development Agency.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
References
Abramo, G., & D’Angelo, C. A. (2011). Evaluating research: From informed peer review to bibliometrics. Scientometrics, 87, 499–514.
Berezin, A. (1998). The perils of centralized research funding systems. Knowledge, Technology & Policy, 11, 5–26.
Cook, W., Golany, B., Kress, M., Penn, M., & Raviv, T. (2005). Optimal allocation of proposals to reviewers to facilitate effective ranking. Management Science, 51(4), 655–661.
Fang, H. (2011). Peer review and over-competitive research funding fostering mainstream opinion to monopoly. Scientometrics, 87, 293–301.
Fedderke, J. W. (2013). The objectivity of national research foundation peer review in South Africa assessed against bibliometric indexes. Scientometrics, doi:10.1007/s11192-013-0981-0.
Gallo, G., Grigoriadis, M. D., & Tarjan, R. E. (1989). A fast parametric maximum flow algorithm and applications. SIAM Journal on Computing, 18, 30–55.
Garg, N., Kavitha, T., Kumar, A., Mehlhorn, K., & Mestre, J. (2010). Assigning papers to referees. Algorithmica, 58, 119–136.
Goldberg, A. V., & Tarjan, R. E. (1988). A new approach to the maximum flow problem. Journal of the Association for Computing Machinery, 35(4), 921–940.
Hartvigsen, D., Wei, J. C., & Czuchlewski, R. (1999). The conference paper-reviewer assignment problem. Decision Sciences, 30(3), 865–876.
Huang, M., Liu, B., & Hong, L. (2010). On assigning papers to reviewers. In 2nd international workshop on database technology and applications (DBTA), (pp. 1–4).
Kalmukov, Y., & Rachev, B. (2010). Comparative analysis of existing methods and algorithms for automatic assignment of reviewers to papers. Journal of Information Technologies and Control, 2, 20–31.
Korte, B., & Vygen, J. (2008). Combinatorial optimization, theory and algorithms. Berlin: Springer.
Langfeldt, L., & Kyvik, S. (2011). Researchers as evaluators: Tasks, tensions and politics. Higher Education, 62, 199–212.
Lee, M., Om, K., & Koh, J. (1999). Blind review of research proposals in Korea: Its effectivenes and factors affecting applicant detection. Scientometrics, 45(1), 17–31.
Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Improving the peer-review process for grant applications. American Psychologist, 63(3), 160–168.
Reinhart, M. (2009). Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics, 81(3), 789–809.
Schrijver, A. (2003). combinatorial optimization: polyhedra and efficiency. Springer, Algorithms and Combinatorics 24, ISBN 9783540443896.
Sandström, U., & Hällsten, M. (2008). Persistent nepotism in peer-review. Scientometrics, 74(2), 175–189.
Wang, F., Chen, B., & Miao, Z. (2008). A survey on reviewer assignment problem. In New Frontiers in applied artificial intelligence, Lecture Notes in Computer Science, 5027, (pp. 718–727).
Jayasinghe, U. W., Marsch, H. W., & Bond, N. (2006). A new reader trial approach to peer review in funding research grants: An Australian experiment. Scientometrics, 69(3), 591–606.
Acknowledgement
This work was supported by Grants 1/410/11 and 1/0479/12 of the Slovak grant agency VEGA (Cechlárová, Potpinková), by OTKA K108383 and the ELTE-MTA Egerváry Research Group (Fleiner) and by the Operational Program Education and Research funded by the European Social Fund, grant Education at UPJŠ Heading towards Excellent European Universities, ITMS project code: 26110230056. The authors are also grateful to the anonymous referee for careful reading and constructive comments that helped to improve the presentation of the paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cechlárová, K., Fleiner, T. & Potpinková, E. Assigning evaluators to research grant applications: the case of Slovak Research and Development Agency. Scientometrics 99, 495–506 (2014). https://doi.org/10.1007/s11192-013-1187-1
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-013-1187-1