, Volume 119, Issue 2, pp 959–972 | Cite as

Research fund evaluation based on academic publication output analysis: the case of Chinese research fund evaluation

  • Gao Ji-ping Email author
  • Su Cheng 
  • Wang Hai-yan 
  • Zhai Li-hua 
  • Pan Yun-tao 


Government funding is a key scientific research resource, and it has made a concrete contribution to the world’s science and technology development. But these funds come from common taxpayers, so we need to evaluate the effectiveness of these funds. Generally speaking, policymakers adopt the method of peer review to make assessments. Compared to kinds of shortcomings of peer review, the paper here proposed the benchmarking evaluation method based on the academic publication outputs of supporting funds, mainly guiding indicators from scientometrics. At first, with the academic publication output extracted from the concluding report project manager submitted to the government after the fund finished, we designed the analysis framework to search and define the research field the fund belonged to. And then from the following three perspectives, including quantity, quality and relative influence, we compared the research fund output to the field output. Later, we took one fund program from national program on key basic research project of China (973 Program) in the field of quantum physics as an example to make an empirical analysis to demonstrate its effectiveness. At last, we found that the funded program performance was superior to the field, and even about 11.65% of the research achievement reaches the top 1/1000 of the world, but the research was lacking in remarkable papers, so it needs further improvement.


Research fund evaluation Academic publication output analysis Benchmarking bibliometrics Research achievement Scientometrics 



This work is supported by National Social Science Foundation of China (No. 17CTQ029). Also we would like to thank anonymous reviewers for their helpful comments.


  1. Bo, Y., & Zeng, J. (2011). Metrological analysis of national social sciences funding annual projects (2000–2009). Science and Technology Management Research, 31, 45–48.Google Scholar
  2. Bonzi, S., & Snyder, H. W. (1991). Motivations for citation: A comparison of self citation and citation to others. Scientometrics, 21, 245–254.CrossRefGoogle Scholar
  3. Boyack, K. W., & Jordan, P. (2011). Metrics associated with NIH funding: A high-level view. Journal of the American Medical Informatics Association, 18, 423–431.CrossRefGoogle Scholar
  4. Cao, H., Chen, G., & Dong, E. (2013). Progress of basic research in Parkinson’s disease in China: Data mini-review from the National Natural Science Foundation. Translational neurodegeneration, 2, 18.CrossRefGoogle Scholar
  5. Chen, H., Roco, M. C., Son, J., Jiang, S., Larson, C. A., & Gao, Q. (2013). Global nanotechnology development from 1991 to 2012: Patents, scientific publications, and effect of NSF funding. Journal of Nanoparticle Research, 15, 1–21.Google Scholar
  6. Costas, R., & Leeuwen, T. N. (2012). Approaching the “reward triangle”: General analysis of the presence of funding acknowledgments and “peer interactive communication” in scientific publications. Journal of the American Society for Information Science and Technology, 63, 1647–1661.CrossRefGoogle Scholar
  7. Feng, M., & Yishan, W. (2009). A survey study on motivations for citation: A case study on periodicals research and library and information science community in China. Journal of Data and Information, 2, 28–43.Google Scholar
  8. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178, 471–479.CrossRefGoogle Scholar
  9. Gök, A., Rigby, J., & Shapira, P. (2016). The impact of research funding on scientific outputs: Evidence from six smaller European countries. Journal of the Association for Information Science and Technology, 67, 715–730.CrossRefGoogle Scholar
  10. Grassano, N., Rotolo, D., Hutton, J., Lang, F., & Hopkins, M. M. (2017). Funding data from publication acknowledgments: Coverage, uses, and limitations. Journal of the Association for Information Science and Technology, 68, 999–1017.CrossRefGoogle Scholar
  11. Guo, H., Pan, Y., Ma, Z., Su, C., Yu, Z., & Xu, B. (2011). Quantitative analysit of papers sponsored by Natural Science Foundation of China. Science & Technology Review, 29, 61–66.Google Scholar
  12. Handley, I. M., Brown, E. R., Moss-Racusin, C. A., & Smith, J. L. (2015). Quality of evidence revealing subtle gender biases in science is in the eye of the beholder. Proceedings of the National Academy of Sciences, 112, 13201.CrossRefGoogle Scholar
  13. Hu, Z., & Wu, Y. (2014). Regularity in the time-dependent distribution of the percentage of never-cited papers: An empirical pilot study based on the six journals. Journal of Informetrics, 8, 136–146.CrossRefGoogle Scholar
  14. Hu, Z., & Wu, Y. (2017). A probe into causes of non-citation based on survey data. Social Science Information, 56, 1–13.Google Scholar
  15. Huang, Z., Chen, H., Li, X., & Roco, M. C. (2006). Connecting NSF funding to patent innovation in nanotechnology (2001–2004). Journal of Nanoparticle Research, 8, 859–879.CrossRefGoogle Scholar
  16. Huang, Y., Zhang, Y., Youtie, J., Porter, A. L., & Wang, X. (2016). How does national scientific funding support emerging interdisciplinary research: A comparison study of big data research in the US and China. PLoS ONE, 11, e154509.Google Scholar
  17. Khan, S., & Hanjra, M. A. (2009). Footprints of water and energy inputs in food production—global perspectives. Food Policy, 34, 130–140.CrossRefGoogle Scholar
  18. Leeuwen, T. N. V., & Moed, H. F. (2005). Characteristics of journal impact factors: The effects of uncitedness and citation distribution on the understanding of journal impact factors. Scientometrics, 63, 357–371.CrossRefGoogle Scholar
  19. Lewison, G. (1998). Gastroenterology research in the United Kingdom: funding sources and impact. Gut, 43, 288–293.CrossRefGoogle Scholar
  20. LI, G. (2004) Improving the efficiency of scientific research depends on the ouput impact. Invention and Innovation, 1–5.Google Scholar
  21. Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Archivum immunologiae et therapiae experimentalis, 57, 13.CrossRefGoogle Scholar
  22. Rigby, J. (2013). Looking for the impact of peer review: Does count of funding acknowledgements really predict research impact? Scientometrics, 94, 57–73.CrossRefGoogle Scholar
  23. Siyao, L., & Yongxin, Z. (2016). Basic research is the supply side of innovation—an interview with Yang Wei, director of the National Natural Science Foundation of China. Beijing: People’s Daily.Google Scholar
  24. Tang, L., Hu, G., & Liu, W. (2017). Funding acknowledgment analysis: Queries and caveats. Journal of the Association for Information Science and Technology, 68, 790–794.CrossRefGoogle Scholar
  25. Wang, X., Liu, D., Ding, K., & Wang, X. (2011). Science funding and research output: a study on 10 countries. Scientometrics, 91, 591–599.CrossRefGoogle Scholar
  26. Wang, J., & Shapira, P. (2011). Funding acknowledgement analysis: An enhanced tool to investigate research sponsorship impacts: The case of nanotechnology. Scientometrics, 87, 563–586.CrossRefGoogle Scholar
  27. Wang, J., & Shapira, P. (2015). Is there a relationship between research sponsorship and publication impact? An analysis of funding acknowledgments in nanotechnology papers. PLoS ONE, 10, 555–582.Google Scholar
  28. Wikipedia (2018) Benchmarking. Accessed 24 November 2018.
  29. Xu, X., Tan, A. M., & Zhao, S. X. (2015). Funding ratios in social science: the perspective of countries/territories level and comparison with natural sciences. Scientometrics, 104, 673–684.CrossRefGoogle Scholar
  30. Yu, Z., Ma, Z., Guo, H., Wang, N., & Jia, J. (2013). A quantitative analysis of papers sponsored by National Science and Technology programs. Journal of Intelligence, 32, 1–5.Google Scholar
  31. Zhang, A., Gao, P., & Liu, S. (2010). Analysis on the output of social science paper subsidized by science foundations of countries. Information Science, 28, 705–708.Google Scholar
  32. Zhao, D. (2010). Characteristics and impact of grant-funded research: A case study of the library and information science field. Scientometrics, 84, 293–306.CrossRefGoogle Scholar
  33. Zou, G., & Tang, Y. (2011). Survey and analysis on input and output of major Projects of Natural Science Fund in China. Science and Technology Management Research, 31, 32–35.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2019

Authors and Affiliations

  1. 1.Center of Scientometrics and Research EvaluationInstitute of Scientific and Technical Information of China (ISTIC)HaidianChina

Personalised recommendations