Skip to main content
SpringerLink
Log in

Joint modeling of the association between NIH funding and its three primary outcomes: patents, publications, and citation impact

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper examines the impact of NIH funding on research outcomes using data from 108,803 projects funded by NIH between January 2009 and March 2017. We extend the prior knowledge on this topic by incorporating the correlation structure of multiple research outcomes, as well as a comprehensive list of grant-level features capturing information on funding size, gender composition and funding type. Specifically, we utilize partial least squares regression (PLS) to jointly model all three primary outcomes (publications, patents and citation impact) and identify the effects of grant-level features on research outputs. Our results show that joint modeling of research outcomes via PLS yields a more accurate prediction than analyzing each outcome separately. Additionally, we find that when other grant-level features are held constant, a 2-year-longer project duration would produce a similar improvement in research outputs to that achieved by $1 million in additional funding. Based on this finding, we recommend no-cost extension of funded projects instead of increased funding support to achieve a comparable increase in research outputs. Promoting multi-organizational grants is found to be more effective for increasing patents, whereas encouraging multiple-PI grants is more productive in terms of publications and citation impact. Of the various NIH grant types, program project/center grants (P series) and research training grants (T series) are the two most productive and impactful. Results also suggest that projects with a higher proportion of male PIs tend to produce more research outputs. This finding, however, needs to be interpreted with caution due to the limitation of our data set.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Agrawal, A., & Henderson, R. (2002). Putting patents in context: Exploring knowledge transfer from MIT. Management Science, 48(1), 44–60.

    Article  Google Scholar 

  • Arrow, K. (1962). Economic welfare and the allocation of resources for invention. In Economic Growth of the Social Science Research Council (Ed.), The rate and direction of inventive activity: Economic and social factors (pp. 609–626). Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Azoulay, P., Ding, W., & Stuart, T. (2009). The impact of academic patenting on the rate, quality and direction of (public) research output. The Journal of Industrial Economics, 57(4), 637–676.

    Article  Google Scholar 

  • Azoulay, P., Zivin, J. S. G., Li, D., & Sampat, B. N. (2015). Public R&D investments and private-sector patenting: Evidence from NIH funding rules. (National Bureau of Economic Research No. 20889) Cambridge, MA: National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w20889.

  • Bacchiocchi, E., & Montobbio, F. (2009). Knowledge diffusion from university and public research. A comparison between US, Japan and Europe using patent citations. The Journal of Technology Transfer, 34(2), 169–181.

    Article  Google Scholar 

  • Blumenthal, D., Campbell, E. G., Anderson, M. S., Causino, N., & Louis, K. S. (1997). Withholding research results in academic life science: Evidence from a national survey of faculty. JAMA, 277(15), 1224–1228.

    Article  Google Scholar 

  • Boyack, K. W., & Börner, K. (2003). Indicator-assisted evaluation and funding of research: Visualizing the influence of grants on the number and citation counts of research papers. Journal of the American Society for Information Science and Technology, 54(5), 447–461.

    Article  Google Scholar 

  • Boyack, K. W., & Jordan, P. (2011). Metrics associated with NIH funding: A high-level view. Journal of the American Medical Informatics Association, 18(4), 423–431.

    Article  Google Scholar 

  • Campbell, D., Picard-Aitken, M., Côté, G., Caruso, J., Valentim, R., Edmonds, S., et al. (2010). Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation, 31(1), 66–83.

    Article  Google Scholar 

  • Cohen, W. M., Nelson, R. R., & Walsh, J. P. (2002). Links and impacts: The influence of public research on industrial R&D. Management Science, 48(1), 1–23.

    Article  Google Scholar 

  • Cronin, B., & Shaw, D. (1999). Citation, funding acknowledgement and author nationality relationships in four information science journals. Journal of Documentation, 55(4), 402–408.

    Article  Google Scholar 

  • Eloy, J. A., Svider, P. F., Kovalerchik, O., Baredes, S., Kalyoussef, E., & Chandrasekhar, S. S. (2013). Gender differences in successful NIH grant funding in otolaryngology. Otolaryngology-Head and Neck Surgery, 149(1), 77–83.

    Article  Google Scholar 

  • Fabrizio, K. R., & Di Minin, A. (2008). Commercializing the laboratory: Faculty patenting and the open science environment. Research Policy, 37(5), 914–931.

    Article  Google Scholar 

  • Fortin, J.-M., & Currie, D. J. (2013). Big science vs. little science: How scientific impact scales with funding. PLoS ONE, 8(6), e65263.

    Article  Google Scholar 

  • Ginther, D. K., Haak, L. L., Schaffer, W. T., & Kington, R. (2012). Are race, ethnicity, and medical school affiliation associated with NIH R01 type award probability for physician investigators? Academic Medicine: Journal of the Association of American Medical Colleges, 87(11), 1516.

    Article  Google Scholar 

  • Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., et al. (2011). Race, ethnicity, and NIH research awards. Science, 333(6045), 1015–1019.

    Article  Google Scholar 

  • Henderson, R., Jaffe, A. B., & Trajtenberg, M. (1998). Universities as a source of commercial technology: A detailed analysis of university patenting, 1965–1988. Review of Economics and Statistics, 80(1), 119–127.

    Article  Google Scholar 

  • Jacob, B. A., & Lefgren, L. (2011). The impact of research grant funding on scientific productivity. Journal of Public Economics, 95(9–10), 1168–1177.

    Article  Google Scholar 

  • Jaffe, A. B., & Trajtenberg, M. (1999). International knowledge flows: Evidence from patent citations. Economics of Innovation and New Technology, 8(1–2), 105–136.

    Article  Google Scholar 

  • Jones, B. F. (2009). The burden of knowledge and the “death of the renaissance man”: Is innovation getting harder? The Review of Economic Studies, 76(1), 283–317.

    Article  MATH  Google Scholar 

  • Lane, J. (2009). Assessing the impact of science funding. Science, 324(5932), 1273–1275.

    Google Scholar 

  • Lane, J., & Bertuzzi, S. (2011). Measuring the results of science investments. Science, 331(6018), 678–680.

    Article  Google Scholar 

  • Ley, T. J., & Hamilton, B. H. (2008). The gender gap in NIH grant applications. Science, 322(5907), 1472–1474.

    Article  Google Scholar 

  • Li, D., Azoulay, P., & Sampat, B. N. (2017). The applied value of public investments in biomedical research. Science, 356(6333), 78–81.

    Article  Google Scholar 

  • Liu, W., & Ruths, D. (2013). What’s in a name? Using first names as features for gender inference in Twitter. Paper presented at the AAAI Spring Symposium: Analyzing Microtext, Stanford, CA.

  • Louis, K. S., Blumenthal, D., Gluck, M. E., & Stoto, M. A. (1989). Entrepreneurs in academe: An exploration of behaviors among life scientists. Administrative Science Quarterly, 34(1), 110–131.

    Article  Google Scholar 

  • Ma, A., Mondragón, R. J., & Latora, V. (2015). Anatomy of funded research in science. Proceedings of the National Academy of Sciences, 112(48), 14760–14765.

    Article  Google Scholar 

  • MacGarvie, M. (2005). The determinants of international knowledge diffusion as measured by patent citations. Economics Letters, 87(1), 121–126.

    Article  Google Scholar 

  • Mowery, D. C., Sampat, B. N., & Ziedonis, A. A. (2002). Learning to patent: Institutional experience, learning, and the characteristics of US university patents after the Bayh-Dole Act, 1981–1992. Management Science, 48(1), 73–89.

    Article  Google Scholar 

  • Murray, F., & O’Mahony, S. (2007). Exploring the foundations of cumulative innovation: Implications for organization science. Organization Science, 18(6), 1006–1021.

    Article  Google Scholar 

  • Murray, F., & Stern, S. (2007). Do formal intellectual property rights hinder the free flow of scientific knowledge?: An empirical test of the anti-commons hypothesis. Journal of Economic Behavior and Organization, 63(4), 648–687.

    Article  Google Scholar 

  • National Institutes of Health (2011). Multiple principal investigators—general information. Retrieved April 11, 2018 from https://grants.nih.gov/grants/multi_pi/overview.htm.

  • National Science Board. (2016). Science and engineering indicators 2016. Retrieved April 17, 2018 from https://www.nsf.gov/statistics/2016/nsb20161/uploads/1/nsb20161.pdf.

  • Nicholson, J. M., & Ioannidis, J. P. (2012). Research grants: Conform and be funded. Nature, 492(7427), 34.

    Article  Google Scholar 

  • Rigby, J. (2013). Looking for the impact of peer review: Does count of funding acknowledgements really predict research impact? Scientometrics, 94(1), 57–73.

    Article  Google Scholar 

  • Svider, P. F., Mauro, K. M., Sanghvi, S., Setzen, M., Baredes, S., & Eloy, J. A. (2013). Is NIH funding predictive of greater research productivity and impact among academic otolaryngologists? The Laryngoscope, 123(1), 118–122.

    Article  Google Scholar 

  • Thursby, J. G., & Thursby, M. C. (2002). Who is selling the ivory tower? Sources of growth in university licensing. Management Science, 48(1), 90–104.

    Article  Google Scholar 

  • Thursby, M., Thursby, J., & Gupta-Mukherjee, S. (2007). Are there real effects of licensing on academic research? A life cycle view. Journal of Economic Behavior and Organization, 63(4), 577–598.

    Article  Google Scholar 

  • Trochim, W. M., Marcus, S. E., Mâsse, L. C., Moser, R. P., & Weld, P. C. (2008). The evaluation of large research initiatives: A participatory integrative mixed-methods approach. American Journal of Evaluation, 29(1), 8–28.

    Article  Google Scholar 

  • Walsh, J. P., Cohen, W. M., & Cho, C. (2007). Where excludability matters: Material versus intellectual property in academic biomedical research. Research Policy, 36(8), 1184–1203.

    Article  Google Scholar 

  • Waltman, L., van Eck, N. J., van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272–285.

    Article  Google Scholar 

  • Wang, J., & Shapira, P. (2011). Funding acknowledgement analysis: An enhanced tool to investigate research sponsorship impacts: The case of nanotechnology. Scientometrics, 87(3), 563–586.

    Article  Google Scholar 

  • Wang, X., Liu, D., Ding, K., & Wang, X. (2011). Science funding and research output: A study on 10 countries. Scientometrics, 91(2), 591–599.

    Article  Google Scholar 

  • Zhang, Y. (2013). Likelihood-based and Bayesian methods for Tweedie compound Poisson linear mixed models. Statistics and Computing, 23, 743–757.

    Article  MathSciNet  MATH  Google Scholar 

  • Zhao, D. (2010). Characteristics and impact of grant-funded research: a case study of the library and information science field. Scientometrics, 84(2), 293–306.

    Article  Google Scholar 

Download references

Acknowledgements

This project was made possible in part by the Institute of Museum and Library Services (Grant Award Number: RE-07-15-0060-15), for the project titled “Building an entity-based research framework to enhance digital services on knowledge discovery and delivery”.

Author information

Authors and Affiliations

  1. Department of Psychology, Drexel University, Philadelphia, PA, 19104, USA

    Fengqing Zhang & Xin Niu

  2. College of Computing and Informatics, Drexel University, Philadelphia, PA, 19104, USA

    Erjia Yan

  3. Department of Library and Information Science, Sungkyunkwan University, Seoul, South Korea

    Yongjun Zhu

Authors
  1. Fengqing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erjia Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Niu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yongjun Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fengqing Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, F., Yan, E., Niu, X. et al. Joint modeling of the association between NIH funding and its three primary outcomes: patents, publications, and citation impact. Scientometrics 117, 591–602 (2018). https://doi.org/10.1007/s11192-018-2846-z

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-018-2846-z

Keywords

Search

Navigation