Abstract
This paper examines the impact of NIH funding on research outcomes using data from 108,803 projects funded by NIH between January 2009 and March 2017. We extend the prior knowledge on this topic by incorporating the correlation structure of multiple research outcomes, as well as a comprehensive list of grant-level features capturing information on funding size, gender composition and funding type. Specifically, we utilize partial least squares regression (PLS) to jointly model all three primary outcomes (publications, patents and citation impact) and identify the effects of grant-level features on research outputs. Our results show that joint modeling of research outcomes via PLS yields a more accurate prediction than analyzing each outcome separately. Additionally, we find that when other grant-level features are held constant, a 2-year-longer project duration would produce a similar improvement in research outputs to that achieved by $1 million in additional funding. Based on this finding, we recommend no-cost extension of funded projects instead of increased funding support to achieve a comparable increase in research outputs. Promoting multi-organizational grants is found to be more effective for increasing patents, whereas encouraging multiple-PI grants is more productive in terms of publications and citation impact. Of the various NIH grant types, program project/center grants (P series) and research training grants (T series) are the two most productive and impactful. Results also suggest that projects with a higher proportion of male PIs tend to produce more research outputs. This finding, however, needs to be interpreted with caution due to the limitation of our data set.
Similar content being viewed by others
References
Agrawal, A., & Henderson, R. (2002). Putting patents in context: Exploring knowledge transfer from MIT. Management Science, 48(1), 44–60.
Arrow, K. (1962). Economic welfare and the allocation of resources for invention. In Economic Growth of the Social Science Research Council (Ed.), The rate and direction of inventive activity: Economic and social factors (pp. 609–626). Princeton, NJ: Princeton University Press.
Azoulay, P., Ding, W., & Stuart, T. (2009). The impact of academic patenting on the rate, quality and direction of (public) research output. The Journal of Industrial Economics, 57(4), 637–676.
Azoulay, P., Zivin, J. S. G., Li, D., & Sampat, B. N. (2015). Public R&D investments and private-sector patenting: Evidence from NIH funding rules. (National Bureau of Economic Research No. 20889) Cambridge, MA: National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w20889.
Bacchiocchi, E., & Montobbio, F. (2009). Knowledge diffusion from university and public research. A comparison between US, Japan and Europe using patent citations. The Journal of Technology Transfer, 34(2), 169–181.
Blumenthal, D., Campbell, E. G., Anderson, M. S., Causino, N., & Louis, K. S. (1997). Withholding research results in academic life science: Evidence from a national survey of faculty. JAMA, 277(15), 1224–1228.
Boyack, K. W., & Börner, K. (2003). Indicator-assisted evaluation and funding of research: Visualizing the influence of grants on the number and citation counts of research papers. Journal of the American Society for Information Science and Technology, 54(5), 447–461.
Boyack, K. W., & Jordan, P. (2011). Metrics associated with NIH funding: A high-level view. Journal of the American Medical Informatics Association, 18(4), 423–431.
Campbell, D., Picard-Aitken, M., Côté, G., Caruso, J., Valentim, R., Edmonds, S., et al. (2010). Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation, 31(1), 66–83.
Cohen, W. M., Nelson, R. R., & Walsh, J. P. (2002). Links and impacts: The influence of public research on industrial R&D. Management Science, 48(1), 1–23.
Cronin, B., & Shaw, D. (1999). Citation, funding acknowledgement and author nationality relationships in four information science journals. Journal of Documentation, 55(4), 402–408.
Eloy, J. A., Svider, P. F., Kovalerchik, O., Baredes, S., Kalyoussef, E., & Chandrasekhar, S. S. (2013). Gender differences in successful NIH grant funding in otolaryngology. Otolaryngology-Head and Neck Surgery, 149(1), 77–83.
Fabrizio, K. R., & Di Minin, A. (2008). Commercializing the laboratory: Faculty patenting and the open science environment. Research Policy, 37(5), 914–931.
Fortin, J.-M., & Currie, D. J. (2013). Big science vs. little science: How scientific impact scales with funding. PLoS ONE, 8(6), e65263.
Ginther, D. K., Haak, L. L., Schaffer, W. T., & Kington, R. (2012). Are race, ethnicity, and medical school affiliation associated with NIH R01 type award probability for physician investigators? Academic Medicine: Journal of the Association of American Medical Colleges, 87(11), 1516.
Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., et al. (2011). Race, ethnicity, and NIH research awards. Science, 333(6045), 1015–1019.
Henderson, R., Jaffe, A. B., & Trajtenberg, M. (1998). Universities as a source of commercial technology: A detailed analysis of university patenting, 1965–1988. Review of Economics and Statistics, 80(1), 119–127.
Jacob, B. A., & Lefgren, L. (2011). The impact of research grant funding on scientific productivity. Journal of Public Economics, 95(9–10), 1168–1177.
Jaffe, A. B., & Trajtenberg, M. (1999). International knowledge flows: Evidence from patent citations. Economics of Innovation and New Technology, 8(1–2), 105–136.
Jones, B. F. (2009). The burden of knowledge and the “death of the renaissance man”: Is innovation getting harder? The Review of Economic Studies, 76(1), 283–317.
Lane, J. (2009). Assessing the impact of science funding. Science, 324(5932), 1273–1275.
Lane, J., & Bertuzzi, S. (2011). Measuring the results of science investments. Science, 331(6018), 678–680.
Ley, T. J., & Hamilton, B. H. (2008). The gender gap in NIH grant applications. Science, 322(5907), 1472–1474.
Li, D., Azoulay, P., & Sampat, B. N. (2017). The applied value of public investments in biomedical research. Science, 356(6333), 78–81.
Liu, W., & Ruths, D. (2013). What’s in a name? Using first names as features for gender inference in Twitter. Paper presented at the AAAI Spring Symposium: Analyzing Microtext, Stanford, CA.
Louis, K. S., Blumenthal, D., Gluck, M. E., & Stoto, M. A. (1989). Entrepreneurs in academe: An exploration of behaviors among life scientists. Administrative Science Quarterly, 34(1), 110–131.
Ma, A., Mondragón, R. J., & Latora, V. (2015). Anatomy of funded research in science. Proceedings of the National Academy of Sciences, 112(48), 14760–14765.
MacGarvie, M. (2005). The determinants of international knowledge diffusion as measured by patent citations. Economics Letters, 87(1), 121–126.
Mowery, D. C., Sampat, B. N., & Ziedonis, A. A. (2002). Learning to patent: Institutional experience, learning, and the characteristics of US university patents after the Bayh-Dole Act, 1981–1992. Management Science, 48(1), 73–89.
Murray, F., & O’Mahony, S. (2007). Exploring the foundations of cumulative innovation: Implications for organization science. Organization Science, 18(6), 1006–1021.
Murray, F., & Stern, S. (2007). Do formal intellectual property rights hinder the free flow of scientific knowledge?: An empirical test of the anti-commons hypothesis. Journal of Economic Behavior and Organization, 63(4), 648–687.
National Institutes of Health (2011). Multiple principal investigators—general information. Retrieved April 11, 2018 from https://grants.nih.gov/grants/multi_pi/overview.htm.
National Science Board. (2016). Science and engineering indicators 2016. Retrieved April 17, 2018 from https://www.nsf.gov/statistics/2016/nsb20161/uploads/1/nsb20161.pdf.
Nicholson, J. M., & Ioannidis, J. P. (2012). Research grants: Conform and be funded. Nature, 492(7427), 34.
Rigby, J. (2013). Looking for the impact of peer review: Does count of funding acknowledgements really predict research impact? Scientometrics, 94(1), 57–73.
Svider, P. F., Mauro, K. M., Sanghvi, S., Setzen, M., Baredes, S., & Eloy, J. A. (2013). Is NIH funding predictive of greater research productivity and impact among academic otolaryngologists? The Laryngoscope, 123(1), 118–122.
Thursby, J. G., & Thursby, M. C. (2002). Who is selling the ivory tower? Sources of growth in university licensing. Management Science, 48(1), 90–104.
Thursby, M., Thursby, J., & Gupta-Mukherjee, S. (2007). Are there real effects of licensing on academic research? A life cycle view. Journal of Economic Behavior and Organization, 63(4), 577–598.
Trochim, W. M., Marcus, S. E., Mâsse, L. C., Moser, R. P., & Weld, P. C. (2008). The evaluation of large research initiatives: A participatory integrative mixed-methods approach. American Journal of Evaluation, 29(1), 8–28.
Walsh, J. P., Cohen, W. M., & Cho, C. (2007). Where excludability matters: Material versus intellectual property in academic biomedical research. Research Policy, 36(8), 1184–1203.
Waltman, L., van Eck, N. J., van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272–285.
Wang, J., & Shapira, P. (2011). Funding acknowledgement analysis: An enhanced tool to investigate research sponsorship impacts: The case of nanotechnology. Scientometrics, 87(3), 563–586.
Wang, X., Liu, D., Ding, K., & Wang, X. (2011). Science funding and research output: A study on 10 countries. Scientometrics, 91(2), 591–599.
Zhang, Y. (2013). Likelihood-based and Bayesian methods for Tweedie compound Poisson linear mixed models. Statistics and Computing, 23, 743–757.
Zhao, D. (2010). Characteristics and impact of grant-funded research: a case study of the library and information science field. Scientometrics, 84(2), 293–306.
Acknowledgements
This project was made possible in part by the Institute of Museum and Library Services (Grant Award Number: RE-07-15-0060-15), for the project titled “Building an entity-based research framework to enhance digital services on knowledge discovery and delivery”.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhang, F., Yan, E., Niu, X. et al. Joint modeling of the association between NIH funding and its three primary outcomes: patents, publications, and citation impact. Scientometrics 117, 591–602 (2018). https://doi.org/10.1007/s11192-018-2846-z
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-018-2846-z