1 Introduction

It should be no surprise that the National Institutes of Health (NIH) is the largest funder of basic health care research in the country. Pertinent to our study, however, is that it is also a leading funder of heathcare ventures, contributing just over $1 billion in 2019 through the Small Business Innovation Research (SBIR) grant program. Whether this money is well-spent is the focus of this paper.

Some scholars have argued that government investment in technology ventures is pivotal to enhance the economic strength and country’s future prosperity (Block & Keller, 2015; Mazzucato, 2013), which indirectly benefits in catalyzing the creation of new technology ventures (Qian & Haynes, 2014). However, others believe that perhaps governments are incapable of efficiently allocating money to the deserving ventures because of the potential for political influence, biased selection processes, or a lack of expertise required to conduct sufficient due diligence (Hegde & Mowery, 2008; Lerner, 1999). In the case of the NIH, these concerns may be exacerbated because their SBIR evaluation processes are similar to the evaluation processes for R01 research grants, prioritizing scientific merit over commercialization potential (National Academies of Sciences, Engineering, and Medicine Report, 2022). Compounding these concerns is that NIH’s selection processes have been criticized in the past (Azoulay et al., 2012; Bourne & Lively, 2012).

Any inefficient allocation flowing through the NIH to support healthcare innovation will certainly translate into loss of social or private value, so understanding the selection capabilities of federal agencies like the NIH is warranted for any policy discussion, especially with regards to healthcare policy, on the optimality of resource allocations for public research funding. In this paper, we evaluate whether the NIH prioritizes the right SBIR applicants for funding—those offering the highest technical and commercial merit, while attending to the goal of investing in high-risk technologies.

Little is known about project selection capabilities because federal agencies do not disclose applications denied funding. Even if agencies made such data available, the best researchers could do is control for selection effects (Howell, 2017; Wallsten, 2000) rather than ascertain whether the grants are awarded to appropriate ventures. An exception is a recent study that evaluates the SBIR selection capabilities of the National Science Foundation (NSF) by exploiting a quasi-natural experiment around an infusion of funding through the American Recovery and Reinvestment Act (ARRA) of 2009, concluding that NSF selection processes were appropriately prioritizing ventures to fund (Dutta et al., 2022). Since NIH’s SBIR program is unique with an exclusive focus on healthcare, and the selection processes and funding objectives differ considerably from NSF, it is important to study whether the NIH has a record of prioritizing the right ventures to fund.Footnote 1

We emulate the empirical design of Dutta et al. (2022) in using the ARRA quasi-natural experiment to study NIH selection capabilities because ARRA funding was also distributed to the NIH. In disbursing the ARRA money, the NIH allocated a portion to fund some SBIR applicants that did not originally receive the award due to the fiscal limitations of the regular NIH budget in FY 2008 or 2009.Footnote 2 Because this “extra” funding was awarded after the regular budget funding determinations were made, the firms funded through the regular budget were clearly prioritized over the firms funded through the extra ARRA money. Since the ARRA-funded and regular budget-funded firms both received the grant, any differences in performance between these two sets of firms can be attributable to NIH’s capabilities in prioritizing the selected firms. A belief that the NIH prioritized appropriately would suggest that regular-budget funded firms should outperform ARRA-funded firms.Footnote 3 Several firm-level performance metrics are used to analyze the performance of ventures receiving SBIR funding, including patents granted, citations to patents, venture-capital infusion, and incidence of initial public offering or acquisition. We also consider whether the NIH prioritizes potential breakthrough projects, as is mandated in their program objectives.

Exploiting this unique setting enables us to reflect on the benefits the grant money can provide to the marginal firms at the cusp of not receiving the money, and perhaps would not have the resources to catalyze the innovation. More broadly, it sheds light on the concerns of federal R&D budget contraction over the last decade, including the U.S. budget sequestration that called for a spending cut of over $85 billion in 2013, which takes a greater toll on young firms developing novel healthcare innovations to survive the Valley of Death. The impending budget cuts imply more competition for fewer grants and, consequently, heightens risk-averse behavior by the NIH—perhaps forgoing pioneering approaches and innovative solutions likely to disrupt industry norms. Given the threat of continued decline in federal R&D budgets for healthcare, answers to the questions we target are timely.

2 NIH SBIR grant program

The SBIR program is mandated by Congress to foster research and development investment in small businesses within the United States that have strong commercialization potential. All federal agencies with extramural research budgets over $100 million are required to participate in the program; yet all agencies establish their own selection processes to meet agency-specific program objectives. Across all federal agencies participating in the program, the NIH is the largest funder of non-defense innovation projects, accounting for 31% of the total dollar obligations to the SBIR program in 2019.

This study focuses explicitly on the Phase I SBIR grants. NIH’s SBIR grants support health and life science businesses that are creating innovative technologies in alignment with their mission to improve health and save lives.Footnote 4 In evaluating the applications, the NIH follows a two-stage peer review process, assigning priority scores for each application based on the following five evaluation criteria: significance, investigator(s), innovation, approach, and environment.Footnote 5 Notably, these are the same criteria used for evaluating the scientific merit of R01 research grants, with very little attention paid to a project’s commercial impact. The review process is further detailed in Online Resource 1.

3 Effects of government initiatives on firm outcomes

A central theoretical rationale for government intervention in entrepreneurial markets is to overcome market failures inhibiting sufficient access to capital for new ventures (Arrow, 1962; Nelson, 1959). There is good reason to believe that capital infused through grant programs is particularly important for new technology ventures because of the potential reluctance of market resource holders to invest in early stages of innovation due to high levels of uncertainty or a lack of interest in increasing social welfare at the expense of financial returns. The weight of evidence from prior research suggests a strong correlation between government initiatives, including the SBIR program, and venture success on a number of fronts, including employment and sales growth (Lerner, 1999), advancing technological innovation (Audretsch et al., 2002; Link & Scott, 2009) with the option to experiment with risky projects (Belz & Giga, 2018), attracting research partnerships (Feldman & Kelley, 2006), commercialization of innovation (Audretsch et al., 2002), investment in research and development (Almus & Czarnitzki, 2003; Audretsch et al., 2002; Bronzini & Iachini, 2014; González et al., 2005), and access to subsequent external funding (Feldman & Kelley, 2006; Toole & Turvey, 2009; Zhao & Ziedonis, 2020), which in turn enhances firm survival rates (Smith et al., 2018). In contrast, some research contradicts these findings, suggesting that firms receiving SBIR grants do not have higher employment growth compared to unfunded firms (Wallsten, 2000) and may not receive follow-on funding from venture capital investors (Berger & Hottenrott, 2021).

However, the prior literature is relatively silent on whether the effectiveness of the grant program is partially (or completely) driven by whether the government is capable of identifying, prioritizing, and funding high quality firms. Such evidence might provide a rebuttal to criticisms that governments should be relatively uninvolved in entrepreneurship and innovation policy because they are incapable of selecting quality ventures. Dutta et al. (2022) provide evidence that National Science Foundation prioritized the right ventures in their SBIR program. In this line of inquiry, some research has examined attributes of the selected firms suggesting that firms smaller in size are more likely to obtain SBIR grants than large firms (Busom, 2000), and as such the smaller firms tend to benefit more from the grant money (Giga et al., 2022). In the challenging entrepreneurial landscape, selecting the right innovation projects to fund is crucial because it has ramifications on the effectiveness of the program in meeting the overall objectives. Therefore, understanding whether the government selects to fund the best projects, with the strong technical and commercial potential, is important. Suboptimal allocations could translate into millions of dollars in lost social or private value, and a greater understanding of selection effects could inspire new processes to remedy this problem.

4 Challenges to optimal grant selection

While the NIH leadership strives to put the right processes in place to help select the most qualified ventures for funding, there are several challenges that may impede optimal selection. For example, some companies have been identified as SBIR mills because of their ability to effectively write grant applications to continually secure grants, which may supersede the underlying quality of the innovation (Lerner, 1999; Link & Scott, 2009; Wessner, 2004). Political influence may also come into play throughout the selection process (Hegde & Mowery, 2008), with agency leaders feeling pressured to unequally distribute funding based on geography (Lerner, 1999), ethnicity of applicants (Ginther et al., 2011), or topic (Bisias et al., 2012). Reviewer biases may also make it difficult to select optimally, such that reviewers may either favor projects associated with their personal areas of expertise (Li, 2012), or be overly critical of projects aligned with their personal interests (Boudreau et al., 2016). Biases aside, given the high uncertainty around early-stage innovations, it is possible that the agency lacks the right expertise and/or processes to ascertain the technological feasibility and conduct thorough due diligence to select the most capable ventures for funding (Lerner, 1999).

5 Scope of paper

This paper provides a first assessment of the selection capabilities at the NIH within the SBIR program. In so doing, the findings of this study shed light on the value the grant may provide to firms at the margin of not receiving the award—a topic of interest to policy makers allocating funding to such grant programs. While this study is focused exclusively on the NIH, the findings provide an important step in helping federal agencies to optimize their selection processes. Our analyses should also help inform broader thinking around government’s role in the entrepreneurial ecosystem allocating taxpayers’ money to fund the types of ventures they set out to select through their existing selection procedures. Does the NIH selection prioritize projects with the highest innovation impact and commercialization potential? Does the NIH prioritize selecting risky innovation in alignment with the government’s objective? We examine these questions below through linking data on SBIR awardees funded through either ARRA or the regular budget cycle to patent and commercialization outcome data.

6 Study data and methods

6.1 ARRA quasi-experiment

The ARRA, enacted by the 111th U.S. Congress to spur technology advancements in science and health, is the single largest stimulus flowing into the U.S. economy through many federal agencies, including the NIH.Footnote 6 The NIH awarded $1.73 billion of the extra funds to different grant programs including the SBIR grant, mainly to applicants that had been previously reviewed and deemed meritorious but were denied funding due to limitations imposed by the regular budget.Footnote 7 This enables us to establish priority candidates in NIH’s selection process.

Our analysis includes ARRA disbursements given to firms originally denied funds in the regular budget (ARRA-funded) compared to firms in the same grant solicitation pool that were prioritized to receive the award during the regular budget cycle (regular-funded). Thus, we have a set of firms applying to the same grant solicitation risk set, with some ventures receiving the grant through the regular-budget cycle and a small number of ventures who were below the budget threshold rejected in the regular-budget cycle, but ultimately receiving the grant through the influx of ARRA money. The ARRA experimental design enables us to examine whether the NIH prioritizes to award the grants to:

  • firms with greater technological merit, innovation impact, and commercialization potential;

  • the firms pursuing risky innovation; and

  • whether it makes a substantive difference between the prioritized firms funded through the regular budget and those funded through the ARRA stimulus.

6.2 Grant recipient data

The sample was constructed by identifying relevant grant solicitations where both ARRA and regular-budget awards were made (see Online Resource 2). We started with the identification of 19 phase I awardees funded through ARRA distributions (ARRA-funded) with information sourced from the ARRA website, and a comparable sample of 479 awardees funded through the regular budget (regular-funded) for the same grant solicitations were identified from NIH Reporter. Performance data (i.e., patents, patent citations, venture capital investment, IPOs, acquisitions) were obtained from a range of publicly available data including the Patent and Trademark Office (USPTO), Securities Data Company (SDC) Platinum, Dun & Bradstreet, Thomson One VentureXpert, and Factiva news. The sample composition efforts yielded a set of 498 phase-I SBIR awarded firms.Footnote 8

6.3 Indicators of innovation and commercialization potential

Innovation performance of ventures is measured in two ways: patent count (a firm’s annual number of patents filed, and eventually granted) approximates innovation rate; and patent citations (“forward” citations to a firm’s patents in the year of the patent application and three subsequent years), which is a proxy for the innovation impact (Trajtenberg, 1990).

Commercialization potential is also measured in two ways: whether a firm receives venture capital investment, which signals a firm’s growth prospects (Megginson & Weiss, 1991), and whether a venture realizes an IPO or acquisition event in the years subsequent to receiving the grant. Innovation riskiness was measured using patent originality score—a Herfindahl index of the patent classes associated with the patents that a focal patent cites (Hall et al., 2005) to assess whether the NIH prioritizes risky innovation projects, a key mandate of the SBIR program. Online Resource 3 tabulates the data source and defines the variable measures.

6.4 Small sample size limitation

The NIH used the extra money from ARRA to go beyond their budget pay line. However, they awarded Phase-I SBIR grants to only 19 firms that were rejected in the regular-budget cycle (even if many more ventures were awarded funding through new grant solicitations). Because this is a relatively small sample size compared to the 479 firms funded through the regular budget, we are reluctant to make definitive claims about the findings. It is important, however, to clarify that these 19 ARRA-funded firms fell below the selection purview of the NIH, and perhaps would not have been able to pursue the innovation without the grant money. Therefore, any difference we observe in the outcomes, before and after the grant, is indicative of the selection preferences at the NIH—shedding light on the objective attributes the NIH evaluates at the time of selecting the awardees and whether the selection processes at work have the capability to look beyond the objective parameters to identify meritorious firms that have the potential to generate impactful innovation.

7 Study results

To analyze selection capabilities, we compare ARRA-funded and regular-funded firms both (a) four years prior to selection, and (b) four years subsequent to receiving the grant.Footnote 9 We believe that observing (a) helps assess the selection ability of the NIH along readily observable criteria prior to awarding the grant, while (b) compares the subsequent performance of regular-funded awardees versus ARRA-funded awardees. As such, (b) has the advantage of also assessing the unobservable selection capabilities of the NIH in discerning the potential of the innovation, because performance should be a function of both observable and unobservable factors.

Figure 1 considers the innovation measures pertaining to patent count, patent citations, and patent originality.Footnote 10 The left side of the figure compares selection capabilities along observable factors at the time the grant was awarded. Compared to ARRA-funded firms, regular-funded firms had more patents, higher citations to those patents, and patents with higher levels of originality. Particularly noteworthy is the striking difference in patent citations, where regular-funded firms received on average 2.7 citations per patent compared to 0.36 citations per patent for ARRA-funded firms. These findings indicate that in selecting grant awardees, the NIH prioritized the firms showcasing a strong innovation portfolio both in terms of quantity and quality, but seemingly did not emphasize prioritizing riskier innovation, as evidenced in similar patent originality scores.

Fig. 1
figure 1

Patent Count, Patent Citations, and Patent Originality (Pre & Post Grant) for ARRA-Funded versus Regular Budget-Funded Firms. This figure shows the average patent count, average patent citation, and the maximum patent originality at the firm level. “Patent count” is the number of patents filed (and eventually granted) by a firm in a year. “Patent citation” is the number of times a patent is cited by other patents in the calendar year of the patent application and three subsequent years. “Patent originality” is one minus Herfindahl concentration of patent class assignments associated with patents cited by the focal patent. Analysis includes 19 ARRA-funded and 479 regular-funded firms. The patent citation and patent originality measures are calculated only for firms that have filed at least one patent in the given time window (pre-grant period includes 5 ARRA-funded and 159 regular-funded firms; post-grant period includes 7 ARRA-funded and 233 regular-funded firms)

Full size image

The right side of Fig. 1 compares innovation performance subsequent to receiving the grant. The figure shows that after receiving grant money, regular-funded firms continued to outpace ARRA-funded firms on patent count. However, the impact and riskiness of the patents filed by ARRA-funded firms, approximated through patent citations and originality, superseded the regular-funded firms. In fact, the difference in patent citations reversed from the pre-grant period, where ARRA-funded firms’ patents received citations twice more than regular-funded firms. Overall, these findings reflect that the NIH effectively identifies and prioritizes firms based on strong observable innovation parameters but perhaps needs to enhance the capability to ascertain the underlying tacit value of the innovation pursued by firms that have the potential to generate risky and impactful outcomes in the future.

Figure 2 shows differences across regular-funded versus ARRA-funded firms in the likelihood of securing venture capital in the four years subsequent to receiving the grant. It clearly shows that regular-funded firms have a higher likelihood of securing venture capital investments (17% of regular-funded vs. 11% of ARRA-funded firms secured venture capital by the fourth year after receiving the grant). Likewise, Fig. 3 examines the differences in achieving a successful exit (IPO or acquisition). It shows that regular-funded firms have a higher likelihood of realizing a commercialization event. In fact, none of the ARRA-funded firms in our sample realized an IPO or acquisition in the four years subsequent to the grant. From a commercialization perspective, the NIH prioritizes selection appropriately, awarding grants to innovations that are commercially viable.

Fig. 2
figure 2

Venture Capital Financing for ARRA-Funded versus Regular-Funded Firms Post Grant. This figure shows the cumulative percentage likelihood of receiving venture capital investment in the four years period post-grant. Analysis includes 19 ARRA-funded and 479 regular-funded firms

Full size image
Fig. 3
figure 3

Exit Event for ARRA-Funded versus Regular-Funded Firms Post Grant. This figure shows the cumulative percentage likelihood of realizing an IPO or acquisition in the four years period post-grant. Analysis includes 19 ARRA-funded and 479 regular-funded firms

Full size image

7.1 Supplemental analyses

Given the sample size of ARRA-funded ventures is small, to supplement our analysis, we considered additional measures of performance outcomes after receiving the grant through firm sales, firm employees, and whether the firm received a phase-II grant—parameters that reflect growth and progress towards commercialization.Footnote 11 To do so, we created a matched sample of ARRA-funded and regular budget-funded firms using coarsened exact matching (CEM) approach. Following the literature on innovation (e.g., Pahnke et al., 2015; Wallsten, 2000), firms are matched on attributes at the time of grant selection: geographical location, industry, founding year, and patenting activity. The matching process yields a sample of 78 firms (16 ARRA-funded and 62 regular budget-funded) that allows to create a balanced sample size across ARRA-funded and regular budget-funded firms. This matched sample is used to assess differences in firm sales, employees, and phase-II grant. We also observe the matched firms until 2021 to ascertain any differences in securing venture capital money and whether the firm realized an IPO or acquisition event. Information on firm sales and employees (sourced from Dun & Bradstreet, Privco, and Reference USA) is only publicly available for current period (i.e., year 2021). Moreover, it is unclear whether the firms absent from the databases have not survived, or merely have zero revenues. Therefore, we present the differences in sales, assuming they did not survive and assuming they survived, but have zero revenues.

Figure 4a shows differences across the matched sample of regular-funded versus ARRA-funded firms in the number of employees and sales observed as of 2021. While there is not much difference in employee count, regular-funded firms report higher sales than ARRA-funded firms (8.83 million regular-funded vs. 5.01 million ARRA-funded). Likewise, Fig. 4b shows differences across the matched sample of regular-funded versus ARRA-funded firms in the likelihood of getting Phase-II grant money, securing venture capital, and in realizing a successful exit event (IPO or acquisition). The findings reflect a pattern similar to the main results suggesting that regular-funded firms have a higher likelihood of securing venture capital and realizing a commercialization event. Moreover, the regular funded firms show a significantly higher likelihood of securing a Phase-II grants (55% of regular-funded vs. 12% of ARRA-funded), which corroborates with the view that the NIH prioritizes ventures that are commercially viable.

Fig. 4
figure 4

a Employees and Sales for Matched ARRA-Funded versus Regular-Funded Firms Post Grant. Figure a shows the average employees and sales at the firm level post-grant in 2021. Analysis includes CEM matched sample of 16 ARRA-funded and 62 regular-funded firms. b Exit Event for Matched ARRA-Funded versus Regular-Funded Firms Post Grant. Figure b shows the percentage likelihood of obtaining phase-II grant, receiving venture capital, and realizing an IPO or acquisition post-grant (observed until 2021). Analysis includes CEM matched sample of 16 ARRA-funded and 62 regular-funded firms

Full size image

8 Discussion

Policy research examining the influence of the government in spurring entrepreneurship has acknowledged the value of the SBIR federal grant program, albeit with mixed empirical results (Howell, 2017; Wallsten, 2000). In this stream, recent work assessing government programs highlighted the benefits associated with such initiatives in enhancing survival rates (Smith et al. 2018), attracting market resource holders (Meuleman & De Maeseneire, 2012; Zhao & Ziedonis 2020), and smaller firms tend to benefit more from such grants (Giga et al., 2022). Alternatively, prior work has also acknowledged the importance of other stakeholders like partnerships with universities (Siegel & Wessner, 2012) and large corporations (Feldman & Kelley, 2006) complement the government support in enhancing the performance of new technology ventures. However, with one exception (Dutta et al., 2022), this scholarship has not drawn much attention to the selection capabilities inside the government. Our study evaluates whether the SBIR grant program administered by the NIH prioritizes the selection of the best and riskiest healthcare innovations. Two rationales compel our examination of NIH’s SBIR program. First, our work has important policy implications regarding the effectiveness of implementing one of the most important entrepreneurial programs in the United States focusing exclusively on healthcare. With over $1.2 billion in grant money awarded by the NIH annually, it is the largest non-defense SBIR program and certainly ranks as one of the most expensive. Second, NIH employs the same selection process in their SBIR program as they do in their R01 research grant program, which has been criticized as being overly conservative (Azoulay et al., 2012; Bourne & Lively, 2012).

Using a quasi-natural experiment made possible through the American Reinvestment and Recovery Act of 2009 and augmenting it with patent and firm data, we find that the NIH seems to effectively identify healthcare ventures accounting for observable metrics suggestive of success. However, since the ventures they prioritize do not clearly outperform others after receiving the grant, forces us to question whether the NIH is able to effectively discern the unobservable facets of a venture, its affiliations with other resource holders, and the attributes of its founding team that might distinguish it to have the potential to subsequently develop impactful innovation. The findings suggest several important policy implications.

In complementing the research stream analyzing the impact of government programs, this paper underscores the importance to orient the focus on selection capabilities when assessing the impact of government programs aimed at spurring entrepreneurial innovations. Given that the broader policy objective of the SBIR initiative is to support the marginal young firm striving to develop novel innovations and survive the Valley of Death, it is critical for federal agencies to enhance their selection capabilities in ascertaining the underlying potential of the innovation being pursued, looking beyond the quantifiable metrics to recognize the ventures with the prospects to generate breakthrough innovation.

Our findings, albeit with limited sample, suggest the NIH’s SBIR program is effective in prioritizing commercially viable innovation but does not place much emphasis on prioritizing the selection of “cutting-edge, high-risk” projects. Unlike Dutta et al. (2022) who found NSF selection prioritizes riskier innovation projects, we find the NIH selection processes tend to remain conservative. Supporting risky innovation is paramount for any policy program because the private market tends to under invest in such projects—the type that might have the most positive externalities for society. Indeed, initiatives like the ARPA-H, recently adopted by the Biden administration to fund high risk-high reward healthcare innovation, is a step in the right direction.

More broadly, our empirical findings appeal to organizational scholars in advancing theories to better understand the role of government in spurring innovation and entrepreneurship: what objectives should the government fulfil, what factors take priority in selecting the projects, what biases may hinder them, and how the mechanisms for the government and private markets can work in complement rather than substitute the efforts. Theoretical efforts in this direction have a direct application for policymakers to design and structure new initiatives tuned to maximize the objectives of the government and efficiently utilize tax payer dollars in facilitating novel innovations. In so doing, our results motivate future scholars to place more attention on the government as an organization, as the “dynamic capabilities not just of firms but of nation states must now be part of the strategic management scholar’s orbit” (Kattel & Mazzucato, 2018).

Finally, lack of access to data hinders the efforts to undertake large data analysis to effectively determine agencies’ selection capabilities. The exogenous inflow of capital through the American Recovery Act accidently revealed a set of firms rejected in the regular application process, which allowed us to exploit the unique experimental setting to parse the selection preferences at the NIH. If information was accessible for the full spectrum of firms in the application pool and the scorings given in the peer review process, empirical analyses would enable us to ascertain the factors that drive the rank order of the selection preferences and how the firms in the priority ranking may benefit differently from the grant. We strongly encourage the NIH and other federal agencies to make such data available for policy researchers to enable a broader and more fluid analysis of selection capabilities over time. Given that government agencies frequently alter their processes, regular assessments are warranted to ascertain changes in effectiveness and the need for further process optimizations. Moreover, it might enable a larger effort to understand how the heterogeneity in selection processes across agencies differentially impact selection capabilities. Such efforts may also help agencies to learn and adopt the best practices followed by their counterparts. The need to link processes to outcomes is particularly pertinent in crisis scenarios like COVID where agencies may decide to deviate from business as usual to reprioritize certain types of firms over others, or to adjust selection processes in alignment with changes in their risk profile.

9 Conclusion

Our work provides new evidence for policy discussions around the capacity of agencies such as the NIH to select qualified healthcare ventures for funding through the SBIR grant program. Future research should extend our analyses, adopting both qualitative and quantitative efforts, to look deeper into the selection capabilities, including the human capital inside the agencies, the structure of the review processes, and the selection criteria emphasized by agencies’ leadership. Such efforts will advance the discussion around selection capabilities and help agencies identify their strengths and weaknesses in supporting the right innovation.