Skip to main content
SpringerLink
Log in

Reviewer recommendation method for scientific research proposals: a case for NSFC

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Peer review is one of the important procedures to determine which research proposals are to be funded and to evaluate the quality of scientific research. How to find suitable reviewers for scientific research proposals is an important task for funding agencies. Traditional methods for reviewer recommendation focus on the relevance of the proposal and knowledge of candidate reviewers by mainly matching the keywords or disciplines. However, the sparsity of keyword space and the broadness of disciplines lead to inaccurate reviewer recommendations. To overcome these limitations, this paper introduces a reviewer recommendation method (RRM) for scientific research proposals. This research applies word embedding to construct vector representation for terms, which provides a semantic and syntactic measurement. Further, we develop representation models for reviewers’ knowledge and proposals, and recommend reviewers by matching two representation models incorporating ranking fusions. The proposed method is implemented and tested by recommending reviewers for scientific research proposals of the National Natural Science Foundation of China. This research invites reviewers to provide feedback, which works as the benchmark for evaluation. We construct three evaluation metrics, Precision, Strict-precision, and Recall. The results show that the proposed reviewer recommendation method highly improves the accuracy. Research results can provide feasible options for the decision-making of the committee, and improve the efficiency of funding agencies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Abdoul, H., Perrey, C., Amiel, P., Tubach, F., Gottot, S., Durand-Zaleski, I., & Alberti, C. (2012). Peer review of grant applications: Criteria used and qualitative study of reviewer practices. PLoS ONE, 7, e46054.

    Article  Google Scholar 

  • Alhosan, N., Fayyoumi, A., & Faris, H. (2014). Shaping an experts locator system: Recommending the right expert. Journal of Theoretical & Applied Information Technology, 66, 645–651.

    Google Scholar 

  • Arora, S., Liang, Y., & Ma, T. (2016). A simple but tough-to-beat baseline for sentence embeddings. International conference on learning representations.

  • Balog, K., Azzopardi, L., & De Rijke, M. (2006). Formal models for expert finding in enterprise corpora. Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 43–50). ACM.

  • Balog, K., & de Rijke, M. (2008). Associating people and documents. In C. Macdonald, I. Ounis, V. Plachouras, I. Ruthven, & R. W. White (Eds.), Advances in information retrieval (pp. 296–308). Springer.

    Chapter  Google Scholar 

  • Balog, K., Fang, Y., de Rijke, M., Serdyukov, P., & Si, L. (2012). Expertise retrieval. Foundations and Trends® in Information Retrieval, 6, 127–256.

    Article  Google Scholar 

  • Biswas, H. K., & Hasan, M. M. (2007). Using publications and domain knowledge to build research profiles: An application in automatic reviewer assignment. 2007 International Conference on Information and Communication Technology (pp. 82–86). IEEE.

  • Blacoe, W., & Lapata, M. (2012). A comparison of vector-based representations for semantic composition. Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning (pp. 546–556). Association for Computational Linguistics.

  • Campbell, C. S., Maglio, P. P., Cozzi, A., & Dom, B. (2003). Expertise identification using email communications. Proceedings of the twelfth international conference on Information and knowledge management (pp. 528–531).

  • Chen, B., Mueller, C., & Willett, P. (2010). Combination rules for group fusion in similarity-based virtual screening. Molecular Informatics, 29, 533–541.

    Article  Google Scholar 

  • Cook, W. D., Golany, B., Kress, M., Penn, M., & Raviv, T. (2005). Optimal allocation of proposals to reviewers to facilitate effective ranking. Management Science, 51, 655–661.

    Article  Google Scholar 

  • Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In International ACM SIGIR conference on research and development in information retrieval (pp. 758–759).

  • Daud, A. (2012). Using time topic modeling for semantics-based dynamic research interest finding. Knowledge-Based Systems, 26, 154–163.

    Article  Google Scholar 

  • Daud, A., Li, J. Z., Zhou, L. Z., & Muhammad, F. (2010). Temporal expert finding through generalized time topic modeling. Knowledge-Based Systems, 23, 615–625.

    Article  Google Scholar 

  • Davoodi, E., Kianmehr, K., & Afsharchi, M. (2013). A semantic social network-based expert recommender system. Applied Intelligence, 39, 1–13.

    Article  Google Scholar 

  • Erjavec, T., & Fišer, D. (2006). Building the Slovene Wordnet: First steps, first problems. Proceedings of the third international WordNet Conference—GWC.

  • Fellbaum, C. (1998). WordNet: An electronic lexical database. The encyclopedia of applied linguistics. MIT Press.

    Google Scholar 

  • Galke, L., Saleh, A., & Scherp, A. (2017). Word embeddings for practical information retrieval. INFORMATIK 2017.

  • Guy, I., Avraham, U., Carmel, D., Ur, S., Jacovi, M., & Ronen, I. (2013). Mining expertise and interests from social media. Proceedings of the 22nd international conference on World Wide Web (pp. 515–526). ACM.

  • Han, S., Jiang, J., Yue, Z., & He, D. (2013). Recommending program committee candidates for academic conferences. Proceedings of the 2013 workshop on Computational scientometrics: theory & applications (pp. 1–6).

  • Henriksen, A. D., & Traynor, A. J. (1999). A practical R&D project-selection scoring tool. IEEE Transactions on Engineering Management, 46, 158–170.

    Article  Google Scholar 

  • Hoang, D. T., Nguyen, N. T., Collins, B., & Hwang, D. (2021). Decision support system for solving reviewer assignment problem. Cybernetics and Systems, 52, 379–397.

    Article  Google Scholar 

  • Hu, K., Wu, H. Y., Qi, K. L., Yu, J. M., Yang, S. L., Yu, T. X., Zheng, J., & Liu, B. (2018). A domain keyword analysis approach extending Term Frequency-Keyword Active Index with Google Word2Vec model. Scientometrics, 114, 1031–1068.

    Article  Google Scholar 

  • Huang, Y., Porter, A., Zhang, Y., & Barrangou, R. (2019). Collaborative networks in gene editing. Nature Biotechnology, 37, 1107–1109.

    Article  Google Scholar 

  • Karimzadehgan, M., Zhai, C., & Belford, G. (2008). Multi-aspect expertise matching for review assignment. Proceedings of the 17th ACM conference on Information and knowledge management (pp. 1113–1122).

  • Leicht, E. A., Holme, P., & Newman, M. E. (2006). Vertex similarity in networks. Physical Review E, 73, 026120.

    Article  Google Scholar 

  • Liben-Nowell, D., & Kleinberg, J. (2007). The link-prediction problem for social networks. Journal of the American Society for Information Science and Technology, 58, 1019–1031.

    Article  Google Scholar 

  • Li, D., Hao, Y., & He, X. (2014). The review and reconstruction for the peer review expert information database of science foundation. Bulletin of National Natural Science Foundation of China, 2014, 209–213.

    Google Scholar 

  • Lin, S., Hong, W., Wang, D., & Li, T. (2017). A survey on expert finding techniques. Journal of Intelligent Information Systems, 49, 255–279.

    Article  Google Scholar 

  • Liu, O., Wang, J., Ma, J., & Sun, Y. (2016). An intelligent decision support approach for reviewer assignment in R&D project selection. Computers in Industry, 76, 1–10.

    Article  Google Scholar 

  • Liu, X., & Porter, A. L. (2020). A 3-dimensional analysis for evaluating technology emergence indicators. Scientometrics, 124, 27–55.

    Article  Google Scholar 

  • Ma, J., Xu, W., Sun, Y. H., Turban, E., Wang, S., & Liu, O. (2012). An ontology-based text-mining method to cluster proposals for research project selection. IEEE Transactions on Systems, Man, and Cybernetics Part a: Systems and Humans, 42, 784–790.

    Article  Google Scholar 

  • Macdonald, C., & Ounis, I. (2006). Voting for candidates: adapting data fusion techniques for an expert search task. Proceedings of the 15th ACM international conference on Information and knowledge management (pp. 387–396). ACM.

  • Macdonald, C., & Ounis, I. (2008). Voting techniques for expert search. Knowledge and Information Systems, 16, 259–280.

    Article  Google Scholar 

  • Magooda, A. E., Zahran, M., Rashwan, M., Raafat, H., & Fayek, M. (2016). Vector based techniques for short answer grading. The Twenty-Ninth International Flairs Conference.

  • Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge University Press.

    Book  MATH  Google Scholar 

  • Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781.

  • Mirzaei, M., Sander, J., & Stroulia, E. (2019). Multi-aspect review-team assignment using latent research areas. Information Processing & Management, 56, 858–878.

    Article  Google Scholar 

  • Mitchell, J., & Lapata, M. (2008). Vector-based models of semantic composition. Proceedings of ACL-08: HLT (pp. 236–244).

  • Mitchell, J., & Lapata, M. (2010). Composition in distributional models of semantics. Cognitive Science, 34, 1388–1429.

    Article  Google Scholar 

  • Mockus, A., & Herbsleb, J. D. (2002). Expertise browser: A quantitative approach to identifying expertise. Proceedings of the 24th international conference on software engineering. ICSE 2002 (pp. 503–512). IEEE.

  • Nuray, R., & Can, F. (2006). Automatic ranking of information retrieval systems using data fusion. Information Processing & Management, 42, 595–614.

    Article  MATH  Google Scholar 

  • Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543).

  • Pradhan, T., Sahoo, S., Singh, U., & Pal, S. (2021). A proactive decision support system for reviewer recommendation in academia. Expert Systems with Applications, 169, 114331.

    Article  Google Scholar 

  • Protasiewicz, J., Pedrycz, W., Kozłowski, M., Dadas, S., Stanisławek, T., Kopacz, A., & Gałężewska, M. (2016). A recommender system of reviewers and experts in reviewing problems. Knowledge-Based Systems, 106, 164–178.

    Article  Google Scholar 

  • Sachdeva, P., Verma, S., & Singh, S. K. (2014). An improved approach to word sense disambiguation. 2014 IEEE international symposium on signal processing and information technology (ISSPIT) (pp. 235–240). IEEE.

  • Serdyukov, P., & Hiemstra, D. (2008a). Being omnipresent to be almighty: The importance of the global web evidence for organizational expert finding. Proceedings of the SIGIR 2008a Workshop on Future Challenges in Expertise Retrieval (fCHER) (pp. 17–24).

  • Serdyukov, P., & Hiemstra, D. (2008). Modeling documents as mixtures of persons for expert finding. European conference on information retrieval (pp. 309–320). Springer.

    Google Scholar 

  • Serdyukov, P., Rode, H., & Hiemstra, D. (2008). Modeling multi-step relevance propagation for expert finding. Proceedings of the 17th ACM conference on Information and knowledge management (pp. 1133–1142). ACM.

  • Sharma, Y., Agrawal, G., Jain, P., & Kumar, T. (2017). Vector representation of words for sentiment analysis using GloVe. 2017 International conference on intelligent communication and computational techniques (ICCT) (pp. 279–284). IEEE.

  • Shen, X. X., Yi, B. L., Liu, H., Zhang, W., Zhang, Z. L., Liu, S. Y. Y., & Xiong, N. X. (2021). Deep variational matrix factorization with knowledge embedding for recommendation system. IEEE Transactions on Knowledge and Data Engineering, 33, 1906–1918.

    Google Scholar 

  • Shon, H. S., Han, S. H., Kim, K. A., Cha, E. J., & Ryu, K. H. (2017). Proposal reviewer recommendation system based on big data for a national research management institute. Journal of Information Science, 43, 147–158.

    Article  Google Scholar 

  • Silva, T., Guo, Z., Ma, J., Jiang, H., & Chen, H. (2013). A social network-empowered research analytics framework for project selection. Decision Support Systems, 55, 957–968.

    Article  Google Scholar 

  • Speer, R., & Havasi, C. (2013). ConceptNet 5: A large semantic network for relational knowledge. The people’s web meets NLP (pp. 161–176). Springer.

    Chapter  Google Scholar 

  • Sriramoju, S. B. (2015). A framework for keyword based query and response system for web based expert search. International Journal of Science and Research, 6, 391.

    Google Scholar 

  • Stankovic, M., Wagner, C., Jovanovic, J., & Laublet, P. (2010). Looking for Experts? What can Linked Data do for you? LDOW.

    Google Scholar 

  • Tan, S., Duan, Z., Zhao, S., Chen, J., & Zhang, Y. (2021). Improved reviewer assignment based on both word and semantic features. Information Retrieval Journal, 24, 175–204.

    Article  Google Scholar 

  • Turney, P. D., & Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37, 141–188.

    Article  MathSciNet  MATH  Google Scholar 

  • Vivacqua, A. S., Oliveira, J., & De Souza, J. M. (2009). i-ProSE: Inferring user profiles in a scientific context. The Computer Journal, 52, 789–798.

    Article  Google Scholar 

  • Vo, D.-T., Hai, V. T., & Ock, C.-Y. (2015). Exploiting language models to classify events from Twitter. Computational Intelligence and Neuroscience, 2015, 4.

    Article  Google Scholar 

  • Wang, G. A., Jiao, J., Abrahams, A. S., Fan, W. G., & Zhang, Z. J. (2013). ExpertRank: A topic-aware expert finding algorithm for online knowledge communities. Decision Support Systems, 54, 1442–1451.

    Article  Google Scholar 

  • Wang, K., Wang, C. K., & Hu, C. (2005). Analytic hierarchy process with fuzzy scoring in evaluating multidisciplinary R&D projects in China. IEEE Transactions on Engineering Management, 52, 119–129.

    Article  Google Scholar 

  • Wang, Q., Ma, J., Liao, X., & Du, W. (2017). A context-aware researcher recommendation system for university-industry collaboration on R&D projects. Decision Support Systems, 103, 46–57.

    Article  Google Scholar 

  • Wei, T., Lu, Y., Chang, H., Zhou, Q., & Bao, X. (2015). A semantic approach for text clustering using WordNet and lexical chains. Expert Systems with Applications, 42, 2264–2275.

    Article  Google Scholar 

  • Xu, W., Sun, J., Ma, J., & Du, W. (2016). A personalized information recommendation system for R&D project opportunity finding in big data contexts. Journal of Network and Computer Applications, 59, 362–369.

    Article  Google Scholar 

  • Xu, Y., Guo, X., Hao, J., Ma, J., Lau, R. Y. K., & Xu, W. (2012). Combining social network and semantic concept analysis for personalized academic researcher recommendation. Decision Support Systems, 54, 564–573.

    Article  Google Scholar 

  • Yancheva, M., & Rudzicz, F. (2016) Vector-space topic models for detecting Alzheimer’s disease. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol. 1, pp. 2337–2346) Long Papers.

  • Yang, C., Ma, J., Silva, T., Liu, X. Y., & Hua, Z. S. (2015). A multilevel information mining approach for expert recommendation in online scientific communities. Computer Journal, 58, 1921–1936.

    Article  Google Scholar 

  • Yong, Y., Yao, Z., & Zhao, Y. (2021) A framework for reviewer recommendation based on knowledge graph and rules matching. IEEE International Conference on Information Communication and Software Engineering (ICICSE) (pp. 199–203). Sichuan Inst Elect.

  • Yukawa, T., Kasahara, K., Kato, T., & Kita, T. (2001). An expert recommendation system using concept-based relevance discernment. Proceedings 13th IEEE International Conference on Tools with Artificial Intelligence. ICTAI 2001 (pp. 257–264). IEEE.

  • Zainal, A. A., Yusri, N., Malim, N., & Arif, S. M. (2013). The Influence of similarity measures and fusion rules toward turbo similarity searching. Procedia Technology, 11, 823–833.

    Article  Google Scholar 

  • Zhang, M., Ma, J., Liu, Z., Sun, J., & Silva, T. (2016). A research analytics framework-supported recommendation approach for supervisor selection. British Journal of Educational Technology, 47, 403–420.

    Article  Google Scholar 

  • Zhang, Y., Lu, J., Liu, F., Liu, Q., Porter, A., Chen, H. S., & Zhang, G. Q. (2018). Does deep learning help topic extraction? A kernel k-means clustering method with word embedding. Journal of Informetrics, 12, 1099–1117.

    Article  Google Scholar 

  • Zhao, S., Zhang, D., Duan, Z., Chen, J., Zhang, Y. P., & Tang, J. (2018). A novel classification method for paper-reviewer recommendation. Scientometrics, 115, 1293–1313.

    Article  Google Scholar 

Download references

Acknowledgements

This work was undertaken with support from the National Natural Science Foundation of China (Award #72104013, #71673024). The findings and observations contained in this work are those of the authors and do not necessarily reflect the views of the National Natural Science Foundation of China.

Author information

Authors and Affiliations

  1. Department of Management, Beijing Electronic Science & Technology Institute, Beijing, 100070, China

    Xiaoyu Liu

  2. School of Management and Economics, Beijing Institute of Technology, Beijing, 100081, China

    Xuefeng Wang & Donghua Zhu

Authors
  1. Xiaoyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuefeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Donghua Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoyu Liu.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 38 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, X., Wang, X. & Zhu, D. Reviewer recommendation method for scientific research proposals: a case for NSFC. Scientometrics 127, 3343–3366 (2022). https://doi.org/10.1007/s11192-022-04389-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-022-04389-4

Keywords

JEL Classification

Mathematics Subject Classification

Search

Navigation