Multilingual abstract

Please see Additional file 1 for translation of the abstract into Arabic.

Introduction

The role of public-funded research in the creation of new knowledge in the national innovation system is fast growing in importance (Etzkowitz et al. 2005). Research and research-related commercialisation activities from universities, public research institutes (PRIs) and health institutions contribute to economic growth in significant ways. The development of human capital in the academic and PRIs and their subsequent knowledge transfer through publications, consulting or migration to the industry could have profound social and economic impact (Cohen et al. 2002).

Although it is widely acknowledged that public-funded research has contributed to the intellectual capital, human capital and industrial capital to fuel economic development and advance the society, the assessment of such contributions has presented challenges to the government funding agencies that pursue national innovation system as part of their economic growth strategy. These issues are pertinent to the policy makers not only in North America (Vincett 2010; Roessner et al. 2013) but also in Europe and Asia Pacific. In the USA, for example, over USD37 billion had been spent annually on R&D by the National Science Foundation and the National Institutes of Health (NSF 2015; NIH 2015). Based on a recent survey of public research institutes in OECD countries, the absolute real R&D expenditure in the public sector increased from USD69.1 billion in 1998 to USD 84.3 billion in 2008, with the strongest being in Spain and Turkey at annual growth of over 9 and 16 %, respectively (OECD 2011). For a few OECD countries that reduced their government’s R&D intensity on PRIs during the period, a corresponding increase was found in the university’s R&D expenditure, thereby indicating the significance of PRIs and universities as two main actors of the public research systems.

Given the significant expenditures on the public research centres, it is essential for scholars and public policy makers to have meaningful, relevant and practical metrics for measuring the impact of public-funded research and innovation. These indicators serve as important tools to inform policy and decision-making, shape research agenda and optimise resource allocation. The public research commercialisation metrics currently used by many research commercialisation offices include the number of intellectual property (IP) licenses, IP licensing revenues and spin-out formation rate. While these primary metrics are important, relevant and easy to capture (AUTM 1991–2012), variation in methodological approaches across the studies does not facilitate benchmarking with similar organisations.

According to Metrics for Knowledge Transfer from Public Research Organisations in Europe published by European Commission (2009), differences in target populations and expected response rates, as well as the normalisation approach for variation in research expenditures and research resource input pose challenges for policy makers in comparing inter-university/PRI performance in their contribution to their regional economy. In addition, there is concern that the current indicators do not adequately reflect the value of the technological innovation that public research organisations have brought to the firm and industry (Roessner et al. 2013). Large-scale investments in academic R&D are generally rationalised by a broad set of public principles or priorities. However, the evaluations are typically made in strictly commercial terms. Although some R&D projects that might have been shelved or aborted did not yield any economic return, prior studies have shown that failed R&D projects could still contribute to the overall innovation process (Elmquist and Le Masson 2009). The first global attempt to measure the socio-economic impact of public-funded research comprehensively was made for the Australian Research Quality Framework. Case study approach was used and indicators verified and proposed. However, the framework was never implemented due to a change in government in 2007 (Penfield et al. 2014).

It is evident that the current public research impact literature has several issues that require attention. First, most studies assumed linear knowledge flow from universities/PRIs to the industry to achieve commercial spill-overs and deliver economic impact (Smith et al. 2011). Second, the lack of clear and consistent definition about the term “impact” and its host of related indicators may be seen in the diverse metrics used in the studies: R&D intensity (Nelson 1986), invention disclosures (Carlsson and Fridh 2002), publications (Nelson 2009), patents (Fischer and Varga 2003), technology licenses with companies (Bach and Llerena 2007), academic spin-offs (Colombo et al. 2010), innovative sales from licensees (Vincett 2010), as well as employment and income (Armstrong 1993). Third, the theoretical frameworks adopted varied widely from resource-based theory (Colombo et al. 2010) and CDM (Barajas et al. 2012) to input-output model (Roessner et al. 2013) and total factor productivity (Fischer and Varga 2003), while the methodologies ranged from individual case study (Brownrigg 1973) to econometric analysis (Felsenstein 1996). The use of different theoretical and methodological approaches has certainly enriched the literature with multi-disciplinary perspectives. However, these have also fragmented the literature, making it difficult for scholars and policy makers to perform meaningful international benchmarking of their innovation impact to inform policy and direct future research (Roessner et al. 2013, Penfield et al. 2014).

This paper aims to consolidate all parts of the proverbial elephants by reviewing the extant literature for university/PRI impact studies, analysing and synthesising the erstwhile disparate studies, discussing the current metrics and their issues, and identifying gaps among the different research strands. Building on the endogenous growth theory (Romer 1990) which advocates knowledge-based economic growth strategy and the Triple Helix modelFootnote 1 (Etzkowitz et al. 2005), we contribute to the body of knowledge by offering a multi-level framework for assessing the economic impact of public research through structuring the indicators into four groups of stakeholders: (a) inventive output of universities/PRIs, (b) commercialisation output of universities/PRIs by their technology transfer organisations (TTOs), (c) innovation of industry/firms and (d) community/ economy.

According to the Transaction Cost Economics (TCE) theory, market transactions are not costless and firms come into existence due to market failures. Buyers may have to bear various types of transaction costs ranging from costs of coordination to hazard of hold-upFootnote 2 arising from asset specificity or IP appropriabilityFootnote 3 (Williamson 1979). Before firms decide to procure external intellectual property (IP), such as in-license technologies from universities/PRIs, they will typically consider the “make internal” versus “buy external” options and weigh the relative costs and benefits of each option. Prior studies have shown that firms with satisfactory prior licensing experience and perception of high licensing benefits relative to costs have a higher propensity for in-licensing technology (Atuahene-Gima 1993).

Adopting the ‘objectivist’ approach (Klaes 2000) of objectively measuring the value of IP transaction to firms, we used the net present value (NPV) methodology (Fisher 1930; Magni 2009) that is regarded robust in theory and prevalent in practice for financial investment decisions to quantify the benefit of in-licensing PRIs’ IPs to the firms. This is done by totalling the sum of projected or extrapolated commercial revenue streams from the royalty-bearing IP licensing transactions - also known as Realised Imputed Commercial Value (RICV).

RICV could thus be objectively computed for every one-time or royalty-bearing IP licensing deal to estimate the sales revenue that firms generate from IP commercialisation. Unlike the resource-intensive methods that could estimate project benefits upon project completion (e.g. RVMFootnote 4, BETAFootnote 5 or NISTFootnote 6), the RICVFootnote 7 method uses annual revenue streams received by the TTO/PRIs as units of observations by relaxing the assumption that the R&D project benefits should or could be attributed to the percentage of PRI’s IP contribution. RICV also has the other advantages of being aggregated for firms at industry and country levels for micro-, meso- and macro-perspectives.

Building on the TCE theory and earlier works that established the positive relationship among licensing experience, benefits and propensity, we postulated in this study that firms’ benefits from in-licensing of university/PRIs’ IPs, represented by RICV, would be positively associated with their propensity to repeat in-licensing transactions with PRIs.

To examine empirically the relative significance of the concept of RICV, we posed the following research question: Is RICV a suitable predictor of firms’ propensity to repeat in-licensing agreements with PRIs, compared to the traditional licensing revenue indicator? Two hypotheses were formulated and binary logistic regression was used to test the model. The first hypothesis (H1) postulates that firms that incur higher IP licensing fees (costs) with universities/PRIs have lower propensity to repeat their IP licensing transactions, thereby reducing academic innovation impact on firms. The second hypothesis (H2) hence posits that firms that attain higher levels of RICV with universities/PRIs (benefits) have higher propensity to repeat their license transaction, indicating an increase in academic innovation impact on firms. With results confirming our hypotheses, we contribute to the literature of impact studies by proposing RICV as new secondary indicators to estimate the economic impact of public-funded technological innovation at the firm, industry and national levels.

We conclude the paper with discussion of the results and their implications.

Methodology

This study comprises two parts: (a) development of an Impact Assessment Framework and (b) the empirical investigation of the RICV concept as part of the framework. An exploration process was used for the first part while regression analysis of archived data of Singapore PRIs was performed for the second.

To develop a comprehensive framework for impact studies on public-funded research, we applied an exploration process comprising three stages: planning, execution and reporting (Tranfield et al. 2003). In the first stage of planning, we identified the key data source as the ISI Web of Knowledge’s Social Sciences Citation Index (SSCI) as it provides peer-reviewed journals which are regarded as validated knowledge of high impact (Podsakoff et al. 2005). During the second stage of execution, we determined initial selection criteria in the form of keywords and various combinationsFootnote 8. With these search keys, we refined our search of SSCI database to articles in English language under the research categories as business economics, educational research, public administration, urban studies, social sciences and science and technology for the period from 1973 to 2013. This results in an initial list of 299 papers. After reviewing their abstracts, we identified 192 relevant papers. To avoid reference bias, we searched Google ScholarFootnote 9 with similar keyword combination queries to arrive at an initial list of 108 peer-reviewed journal papers, of which 89 were found relevant. After checking against duplicates, we compiled our consideration setFootnote 10 of 275 papers for analysis and classification by the types of the impact indicators used and synthesis of conceptual framework. The framework will be presented in the next section.

For the empirical investigation of the RICV concept as part of the framework, we used licensing agreement transactions as the units of analysis in non-contrived settings. To control for inter-nation and inter-PRI variations, this study focused on PRIs governed within the same IP, science and technology and organisational framework of the national research agency of Singapore. Operating under the same agency, the PRIs are assumed to have the same propensity to out-license their technologies to enterprises. This study used the IP commercialisation database the agency maintains on organisations that had signed licensing agreements with them to deploy technologies developed by their PRIs for in-house use or commercial purposes for a stipulated period of time. The dataset comprised 835 licensing agreement transactions that took place between 2006 and 2012, innovation outcome of the firms in terms of innovative product count and RICV, as well as their status as repeat licensees.

In H1, the independent variable is the annual IP licensing fees incurred by a firm for in-licensing IP developed by the PRI, payable from the firm to the TTO representing the PRI, as executed in the TTO firm licensing agreement. The dependent variable is the firm’s propensity to repeat licensing, which is used as a proxy indicator of the impact that prior PRIs’ IP licensing and knowledge transfer has on the firm’s innovation output. This variable is binary with a value of 1 if the firm has repeated licensing agreements with the TTO or 0 if it has not.

In H2, the dependent variable is the same as that for H1, and the independent variable is annual RICV generated by the firm for executing an IP licensing agreement with the TTO. The annual RICV is made up of projected or extrapolated commercial revenue streams from royalty-bearing IP licensing deals for a particular year. Please refer to the Appendix for computation details and examples.

To control for variations contributed by other factors, control variables such as licensee organisation type and technological orientation of IP licensed are included in the study.

To validate H1, positing that firms that incur higher IP licensing fees with universities/PRIs have lower propensity to repeat licensing transactions, binary logistic regression is performed, as the dependent variable, firm’s propensity to repeat licensing is a binary variable, to examine how the independent variable, IP licensing fees, affect the dichotomous outcome. This approach of binary logistic regression had been used in prior study by Ambos et al. (2008) on the factors affecting the commercial output of academic research, also coded as a binary variable. In H2, the same binary-dependent variable used in H1, firm’s propensity to repeat licensing, is regressed using binary logistic regression on the independent variable, RICV.

Results

Public research impact studies

This section presents a descriptive analysis of our consideration set of 275 articles and reviews the dimensions of public research impact assessment posited in these studies. Figure 1 shows that 275 articles were published in a total of 50 journals.

Fig. 1
figure 1

Public research impact articles published per academic journal 1973–2013

Full size image

Given that impact studies may be undertaken at various levels of analysis, ranging from individual through institutional to national, the range of journals publishing such studies is correspondingly wide, from the fields of business economics to public administration. However, a closer examination of the journal titles shows that the distribution of journal titles is rather skewed, with three journals accounting for more than half of the journal population: Research Policy (35 %), Journal of Technology Transfer (12 %) and Technovation (5 %). While Research Policy has consistently topped the chart, Journal of Technology Transfer saw an explosion of publications in this area in the recent decade. There may be several explanations for this phenomenon. Using the lens of life cycle model to the development of new inquiry topic, the topic of technology transfer was in the nascent stage of development. It is noteworthy that following the US Bayh–Dole Act of 1980 and similar developments in other innovating economies, which encouraged universities and PRIs to patent their inventions with government funds, a rise in technology transfer from public research to the industry spear-headed a concomitant growth in the literature of technology transfer, particularly in the USA and Europe. As new topics of inquiry gain attention, the editorial teams of more prestigious journals that might be initially sceptical of their validity became more willing to publish them or new journals such as the Journal of Technology Transfer being developed to carry the scholastic conversations (Kuhn 1962). During the period of 1973–2013, we observe an increasing volume of impact studies, as presented in Fig. 2.

Fig. 2
figure 2

Public research impact articles published per year

Full size image

Our review of public research and innovation impact studies shows that interest in this topic started back in the 1970s, when the US tertiary institutions were recommended to assess the direct and indirect consequences of their investment and expenditures on their regional economy by the American Council on Education (Goldstein 1989).

In the 1980s, the national innovation system (NIS) concept gained attention with the works of Freeman (1987) in Japan and Nelson (1993) in the USA in the area of public-funded research. Their works highlighted the significant roles that universities, PRIs and government agencies play in pursuing technological progress and contributing to economic growth. The public policy makers’ rising expectations of academic innovation impact on the industry and society may be seen in the proliferation of studies on this topic as politicians, economists and technology management scholars questioned about the diverse aspects of public research impact ranging from the rationale and feasibility of measuring such impact to the conceptual framework and methodology of monitoring and managing it.

Brownrigg’s (1973) case study of UK University of Stirling found that its impact on the local economy in terms of income and employment might be marginal due to the significance of other factors such as local industrial structure and its ability to promote trade and investment flow. However, Brownrigg highlighted the limitations of traditional input-out modelling used by the economists in such impact studies of local economy. A similar case study approach coupled with university panel data was adopted by Keane and Allison (1999) in their study of Australian University of Sunshine Coast, Queensland, to establish the strength of university-industry connection and the degree of its “embeddedness” in the region.

In the USA, Bozeman and his colleagues developed the research value mapping (RVM) approach to assess the socio-technical impact of applied R&D projects with the Department of Energy (Bozeman and Kingsley 1997; Bozeman and Klein 1999). Using a combination of qualitative (case studies) and quantitative techniques (data developed from case studies), Bozeman and his team spent a considerable amount of effort to track the knowledge flow of R&D projects and model the possible project outcomes such as technologies patented or licensed, new companies created, new or improved products or processes developed or enhanced human capital. The RVM was found to work well for summative evaluation; however, its adoption beyond the USA has been limited due to the extensive resources required to trace and perform value mapping. According to Hicks et al. (2002) report submission to the Japanese National Institute of Science and Technology Policy, the RVM’s identification of projects with significant milestone achievement to be units of analysis could provide administrative convenience from the funding perspective but did not reflect how the researchers view their work and capture other non-economic payoffs such as the development of human capital (Georghiou and Roessner 2000).

Jaffe (1989, p. 957), on the other hand, used US state-level time-series data to present the significance of “geographically mediated spill-overs from university research to commercial innovation” in the form of corporate patents, particularly in the drugs, chemicals and electronics industrial sectors. These findings were corroborated by Fischer and Varga’s (2003) work on 72 Austrian districts using knowledge production function framework, as well as Beugelsdijk and Cornet’s (2002) study of 1510 Dutch firms, who found that their proximity to technical universities had significant spill-over effects on the firms in the form of new products. Mansfield’s (1991) study of 76 major US firms attributed 11 % of their new products and 9 % of new processes to academic research.

Harhoff’s (1999) econometric analysis of German engineering industry sectors yielded similar evidence of regional spill-over effects from academic institutions on the sectors in the form of new business formation. However, Lööf and Broström’s (2008) study in Sweden found significant academic impact only on large manufacturing firms in the form of innovative sales, while the average service firms did not seem to benefit from university collaboration.

More recent studies are also found in Spain by Barajas et al. (2012) who used the CDM (Crépon et al. 1998) model to measure academic spill-over effects in the form of sales per employee; and Italy by Colombo et al. (2010) who applied the resource-based theory (Wernerfelt 1984) to assess innovation impact in terms of spin-offs.

In France, Bach and Llerena (2007), in their study with BETA, urged to look beyond traditional technology innovation indicators, such as R&D expenditures and licensing revenue, to capture the interaction dynamics among the institutes. It is noteworthy that BETA of France and NIST of the USA had introduced their respective methods in the 1990s to assess the impacts of large-scale R&D projects such as the European Space Programme (Bach and Lambert 1992) and Measurement and Standards Laboratory Programme (Tassey 1999). These methods were subsequently adopted by other countries such as Japan in their study of techno-economic impact of large-scale R&D projects (Kondo 2012). While these methods have been useful for estimating the impact of R&D projects, their limitation on the timing and size of the impact studies has been highlighted as the projects selected for studies must have either completed or achieved some significant benefits. As the time elapsed between project start-up and project benefit realisation might range two to seven  years, based on past studies (Tassey 1999), BETA and NIST used these methods only for large-scale projects, while employing other forms of intermediate indicators, such as new companies started and product or process innovation for periodic and more pragmatic assessment of their innovation impact.

To build a comprehensive multi-dimensional framework, we applied the R&D-based endogenous growth theory (Romer 1990) and Triple Helix (Etzkowitz et al. 2005) to model the flow of knowledge among the major stakeholders of the industrialised economies: universities and PRIs, firms and industry, as well as the community and society. The endogenous growth theory (Romer 1990) postulates that knowledge and technology, unlike physical objects, are characterised by increasing returns, thereby driving economic growth. The theory forms the basis of knowledge-based economy and emphasises the critical role that the economic processes of knowledge creation and diffusion play in shaping growth of firms, industries, communities and nations. The Triple Helix, on the other hand, underscores the central role of universities and its relations with the industry and government in advancing the knowledge-based society (Etzkowitz et al. 2005).

To synthesise the data gathered into the framework, we categorised the dimensions of impact assessment which emerged from our literature review by the different types of knowledge generated and associated output indicators from the various stakeholders: (1) inventive output of universities/PRIs, (2) commercialisation output of universities/PRIs by their TTOs, (3) innovation of industry/firms and (4) community/economy. Figure 3 shows the distribution of impact study papers by categories 1 to 4.

Fig. 3
figure 3

Distribution of papers by output category

Full size image

While the majority of papers (84 %) fall into only one of the four categories, some articles belong to more than one category. For example, some studies (e.g. Derrick and Bryant 2013) used both the inventive output of the universities (publications—category 1) and commercialisation output of universities/PRIs (licenses generated—category 2) to measure the impact of research incentives on Australian medical research organisations, thereby falling into in category (1/2) as illustrated in Fig. 3.

Figure 4 presents the conceptual framework depicting the processes and associated output captured by studies among these four stakeholder output categories.

Fig. 4
figure 4

Impact Assessment Framework

Full size image

The categories are represented as blocks and the knowledge flows among the blocks depicted by bilateral arrows. These bilateral flows among the blocks are better representation of knowledge flow directions than the traditional linear knowledge flows from universities/PRIs through the firms/industry to the economy as illustrated by most studies in the current literature (Gulbrandsen and Rasmussen 2012). In practice, there is much knowledge exchange between universities/PRIs and the industry, as the inventive output from the former could be inspired by the problems encountered by the latter through research collaboration. In a similar vein, the commercialisation output of the former could be shaped and refined by the requirements of the latter through the open innovation approach adopted by firms (Chesbrough 2007). In addition, universities/PRIs could have direct impact on the economy through their direct expenditures and employment. This direct interaction between universities/PRIs and economic development is represented in our conceptual framework by enveloping universities/PRIs/TTOs (block 1 and block 2) and industry/firms (block 3) with the larger community/economy (block 4). The knowledge flows are therefore indicative of the four phases of the RIE value chain.

It is important to note that this framework is distinct from the chain-linked model postulated by Kline and Rosenberg (1986), which focuses on the relationships among research, invention, innovation and production within the boundaries of an organisation. The Impact Assessment Framework, in contrast, looks beyond the organisational boundaries to access complementary assets and open innovation. The categorisation of block 1 and block 2 is supported by Gulbrandsen and Rasmussen (2012), who made similar distinction between inventive output of universities/PRIs and their commercialisation output, articulating that the former measures the potential rather than the actual commercial value of public science. Block 3 is consistent with the findings of Crépon et al. (1998), who distinguished between innovation input (R&D) and innovation outcome (product and process innovation). Block 4 is corroborated by the work of Goldstein and Renault (2004, p. 735), who advocated that universities could have potential impacts including “direct and indirect spending impacts”.

Case of Singapore PRIs

In the case of Singapore PRIs, all the indicators highlighted in blocks 1 and 2 were used to measure their inventive and commercialisation output for tracking progress and benchmarking with local and international counterparts, as indicated in Fig. 5.

Fig. 5
figure 5

Impact Assessment Framework used by Singapore PRIs

Full size image

To assess the commercialisation impact of PRIs’ research at firm, industry and economy levels, the PRIs actively tracked since 2006 the RICV of the intellectual property (IPs) generated from the licensees’ commercialisation of the IPs and aggregated them at the various levels. RICV was captured for two purposes: (a) obtain the actual commercial value generated by the IP and (b) validate their pre-deal assumptions of projected cash flows as feedback mechanism.

According to the descriptive statistics in Table 1, more than 50 % of the licensing agreement transactions were with small and medium enterprises (SMEs), 40 % with multi-national corporations (MNCs) and large enterprises and the remaining with start-ups. About one third of the licensing agreement transactions came from the biomedical sciences sector, one third from info-communication sector and the rest from engineering, manufacturing, electronics, materials, chemicals and other sectors.

Table 1 Descriptive statistics of organisations participating in licensing transactions
Full size table

Our regression results for H1 are presented in models 1 and 2 of Table 2. Model 1 is a baseline model that examines the variances contributed by the control variables. Model 2 describes how IP licensing fees affects a firm’s propensity to repeat licensing pertaining to H1. Model 2 does not appear to be an improvement over model 1, as evident from the similar pseudo R square values, Cox and Snell R square (0.08) and Nagelkerke R square (0.13), as well as the percentage of correct classification (78 %). However, with a coefficient (b = −0.23, p < 0.1), IP licensing fees is found to be a significant but negative predictor of the firm’s propensity to repeat licensing, indicating support for H1 that firms that incur higher IP licensing costs have lower propensity to repeat their IP licensing transactions, thereby reducing academia innovation impact on firms.

Table 2 Regression results
Full size table

The regression results for H2 are presented in model 3 of Table 2. Model 3 is an improvement over the baseline model 1, as seen from the increase in the Cox and Snell R square (from 0.08 to 0.17), Nagelkerke R square (0.13 to 0.27) and the percentage of correct classification (78 to 82 %).

With a positive coefficient (b = 1.67, p < 0.05), RICV is established to be a significant and positive predictor of the firm’s propensity to repeat licensing, thereby validating H2 that firms that attain higher levels of RICV with universities/PRIs have higher propensity to repeat their license transaction.

To explore possible interaction effects between IP licensing fees and RICV, model 4 is a full model demonstrating the main and interaction effects of all variables, including the interaction term between IP licensing fees and RICV. The interaction term is found to be not significant. In models 1 to 4, some control variables such as firm size and engineering and manufacturing sector seem to be significant. After controlling for them, there is no change in the significant contributions of the independent variables to the dependent variable, confirming support for H1 and H2. Combining the results of H1 and H2, we can establish that RICV is a better indicator than IP licensing fees in predicting a firm’s propensity to repeat licensing and indicating academic innovation impact on firms. With more than 100 cases per predictor variable, these models satisfy Field’s (2005) guidelines of minimum 10 cases per predictor for reliable regression results. With a variance inflation factor (VIF) value of 1.234 that is lower than the cut-off value of 10 recommended by Chatterjee and Price (1991), the probability of multi-collinearity issues affecting the findings is relatively low.

Combining the results of H1 and H2, we establish that while IP licensing fees and RICV have opposite effects on firm’s propensity to repeat licensing with PRIs and academic innovation impact on them. The findings are consistent with Atuahene-Gima’s (1993) findings that firms’ ITL propensity was determined by their perceived value, namely, their benefits relative to their costs.

Although this study involves PRIs, its results may be seamlessly extended to universities for several reasons. First, since the implementation of the US Bayh–Dole Act and its equivalents in other knowledge-based economies, PRIs and universities in most countries have been charged with similar missions to create socio-economic impact through promoting knowledge generation and transfer to the industry (Link et al. 2011). Second, although PRIs are expected to work on more applied research while universities on basic research, both share similar strategic roles in setting future R&D directions, especially in areas critical to their nations, e.g., energy or climate. As a result of their longer planning horizon, their technologies are at an earlier stage of development compared to the industrial R&D centres, making their time to commercialisation longer and more risky than that of their industrial counterparts (Link and Scott 2013). As both universities and PRIs share similar goals, roles, pressures and challenges in their technology transfer processes, many studies in the technology transfer literature have treated them as a homogeneous group and extend their findings across them (Park et al. 2010).

Discussion

In the field of public research impact, the definition of the term “impact” differed across various stakeholders. To academia, the term is usually associated with intellectual contribution to one’s discipline within academia while socio-economic impact is regarded as beyond academia. From the government perspective, the definition is much broader as that adopted by the Australian RQF and the UK Research Excellence Framework (REF). According to the UK Research Excellence Framework (REF2014 2011), impact is an effect on change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

How broad should the definition be? Its breadth really depends on what and how it is to be assessed. To countries that focus on science, technology and innovation (STI) to fuel economic growth and generate wealth (e.g. Singapore), impact is typically defined and assessed with economic measures and STI indicators (Donovan 2011)

We made several contributions in this paper. First, we consolidated a large body of knowledge on impact studies with a theoretically grounded, multi-level framework for assessing public research and innovation impact, connecting the four stakeholders of the knowledge production process—PRIs/universities, TTOs, firms/industry, and community/economy.

Second, we identified literature gaps that create opportunities for future research. In this section, we will discuss several gaps, including the relationships among the various categories of stakeholder outputs, depicted as Impact Assessment Framework blocks 1 to 4 of Fig. 4.

Third, we established empirically the relative significance of RICV as a better indicator than IP licensing fees in predicting firms’ propensity to repeat licensing from PRIs, thereby indicating some form of academic innovation impact on firms. We will elaborate in this section our findings of the RICV concept in the context of the integrated framework.

Inventive output of universities/PRI

In the first block of the framework in Fig. 4, the studies generally adopted the approach of using the inventive output of universities/PRIs as proxy indicators to estimate their social and economic impact. The notion of knowledge as key driver for the competitive advantage of firms and nations has its roots in the literature of economics building on Romer’s endogenous growth theory (Romer 1990; Nonaka and Takeuchi 1995; Grant 1996). The capacity and capability to develop novel ideas and translate them into new products and processes for economic growth form the knowledge base of an economy (Huggins and Johnston 2009). These innovation development capabilities are increasingly being associated with universities and PRIs. As knowledge becomes publicly accessible either in the form of spill-overs or diffusion, it is widely regarded as a desirable outcome for the greater good of the society, spurring greater innovation and economic growth (Fischer and Varga 2003). Table 3 provides a summary of key studies using inventive output indicators.

Table 3 Summary of studies using university/PRI inventive output indicators
Full size table

As centres of knowledge production, universities/PRIs consistently engage in inventive process, where researchers develop ideas that could be novel and inventive. These inventions are then submitted as required by the Bayh–Dole Act or equivalent in the form of invention disclosures for evaluation by the TTOs for the appropriate intellectual property (IP) strategy to pursue. For invention disclosures that satisfy both internal and external patenting guidelines, the TTOs would proceed to file patent applications, while the researchers would submit their findings for publications. Some high-technology industries may find these publications or patents relevant and cite them or even participate in research collaboration with the universities/PRIs in the course of their R&D work.

Given the significant government R&D expenditures in universities and PRIs to develop the knowledge base of the economy, the studies of this category frequently built on the total factor productivity or knowledge production function frameworks to examine the various outputs in the inventive process as indicators to determine the economic benefits directly or indirectly generated (Jaffe 1989). Building on Cobb–Douglas production function, Pakes and Griliches (1984) pioneered the development and use of the full model of knowledge production function and the final output production function. In their study, they used R&D expenditures as knowledge creation investment and patent counts as economically valuable knowledge produced. Besides economic impact, many researchers have also adopted bibliometric approaches to assess the degree of international scientific influence using citation, mapping and network analysis of papers and patents (Tornquist and Hoenack 1996; Spencer 2001). Others have studied co-authored publications in their bibliometric examination of public-private research collaboration (Abramo et al. 2009a). While citation indicators may reflect productivity and impact, co-authorship indicators may illustrate informal network access, knowledge diffusion and skill transfer.

Cohen et al. (2002) found that publications are important knowledge channels from the public sector to the manufacturing sector. This is supported by Agrawal and Henderson’s (2002) interview of MIT faculty, who established that publications were two-and-one-half times more significant than patents as knowledge channels. Jaffe (1989) found that the proximity between universities/PRIs and firms could have mediating effects on spill-overs from public-funded research to corporate patents. Subsequent studies suggested similar association between patent citation patterns (as indicator of “economic benefit”) and geographical proximity in Austria (Fischer and Varga 2003) and in the USA (Jaffe et al. 1993).

Building on the RVM project, Dietz and Bozeman (2005) discovered significant differences between publication and patent productivity in their socio-technical human capital project. Career spent in industrial settings was found to be negatively related to publication productivity but positively related to patent productivity. In a similar vein, the source of industry support was identified to be unrelated to publication productivity but positively related to patent productivity. These findings suggest that governments that aim to improve the commercial relevance of their public-funded research should encourage “revolving door” policy between academia and industry in their development of science and technical human capital to enhance their university/PRI inventive output.

Pakes and Griliches (1984) established a strong association between R&D and patent counts at the firm and industry levels to demonstrate that patents are appropriate measures of inventive output. Although patent and patent citations have frequently been used as economic indicators, there is concern that patents do not provide the complete picture of economic contribution by universities/PRIs (Fischer and Varga 2003). First, patents do not capture all forms of knowledge, such as codified knowledge (trade secrets and copyrights) and tacit knowledge (know-how and experience), which is important for spill-over. Second, patent count is dependent on the intensity of patenting activity, which is in turn determined by the patenting policy of the institutions. Therefore, patent counts and citations are not necessarily reflective of the commercial value of underlying IP.

According to a study by Intellectual Ventures in 2008, although 60 % of the patents were developed by academia, PRIs and individual inventors, they contributed to less than 1 % of the licensing revenue. On the other hand, large enterprises which were the originators of 40 % of the patents received more than 99 % of the licensing revenue (Hagiu et al. 2011). The stark contrast in licensing revenue between these groups that is disproportionate to that in patent origination signals critical issues with patents as economic indicators.

This demonstrates the urgency for more commercialisation-related indicators that policy makers can implement if they are committed to encouraging more firms to license and commercialise public science (Cheah and Zalan 2013).

Commercialisation output of universities/PRIs by TTOs

The second block of the Impact Assessment Framework in Fig. 4 dwells on the output of commercialisation process that TTOs use in taking IPs from their universities/PRIs to the market. Since the 1990s, there has been an increasing awareness that technological innovation may not be confined within the boundaries of an organisation. It has been widely acknowledged that technology is an essential resource that can be converted into a capability and competitive advantage for firms and nations. Firms have been giving more attention to the sourcing of external knowledge to complement and enhance their existing R&D stock and the acquisition of external technology as part of their R&D process. A summary of key studies using university/PRI commercialisation output indicators is listed in Table 4.

Table 4 Summary of studies using university/PRI commercialisation output indicators
Full size table

According to Hemmert (2004), external technology acquisition takes place by (a) licensing technology, (b) research collaboration with other organisations and (c) purchasing technology from other organisations. TTOs typically do not sell the technologies originating from universities/PRIs, but prefer to assign non-exclusive licenses to firms if the inventions are broad in scope and wide in applications across industries, so as to encourage product development in more fields for maximum economic impact. Notwithstanding this, TTOs may consider exclusive licensing terms for inventions that require substantial investments for commercialisation. Nelson’s (2009, p. 994) comparison of IP licenses, publications and patents found IP licenses to be the most reliable indicator of “downstream knowledge utilisation” as the “competing economic interests” between licensors and licensees ensure that the licensing agreement is a fair representation of industry demand. Besides licensing IPs to existing firms, TTOs may facilitate the formation of spin-offs by providing technology and business incubation services to their researchers.

As both developed and developing nations put in place the appropriate science, technology and education policies to facilitate technology commercialisation and academic entrepreneurship from universities/PRIs, there has been much scrutiny on their commercialisation output, namely, the number of IP licenses signed, the IP licensing revenues collected and the number spin-offs generated.

To date, more than 14 periodic surveys had been launched to capture such output indicators across different regions such as the USA (AUTM 1991–2012), Canada (AUTM 2000–2012), Europe (ProTon 1991–2012) and Singapore (A*STAR 2002–2011). The data collected from these surveys form the dataset for many studies in this literature strand. For example, AUTM data were used by Di Gregorio and Shane (2003) to show that the rate of university spin-offs depends on inventors’ quality and equity stake in the spin-offs. Calls have been made to both AUTM and ASTP to standardise their variable definitions and methodological approaches to facilitate comparability of PRI indicators across the studies. However, there is a dearth of impact studies in the Asia.

One frequently used theoretical framework in these studies is the resource-based theory (Wernerfelt 1984), which postulates that a firm’s performance is determined by its resources. Barney (1991, p. 117) defined resources as assets that are specific to the firm, “valuable, rare, imperfectly imitable, and non-substitutable”. Building on the resource-based theory, O’Shea et al. (2005), Colombo et al. (2010) and Powers and McDougall (2005b) established that the quality of researcher and research has significant effect on spin-off formation and performance, which were commercialisation output indicators used to estimate public research economic impact.

While licensing number and revenue are useful and relevant indicators to measure commercialisation process efficiency, several issues have been raised about their adequacy in reflecting the economic benefits of public-funded research. First, as license data collected by TTOs only reflect commercialisation activities during the licensing period, which is capped between patent grant date and expiration date, they do not capture any further innovative activity that may continue to impact the economy beyond the licensing period. Second, it is widely recognised that the pathway of commercialisation from invention (that the firm has licensed from universities/PRIs) to innovation (that the firm can generate sales in the form of products and services) requires significant investment as it involves early-stage technology development. This pathway is often alluded to as the valley of death due to the high failure rate, and the role of the public sector in bridging this valley has been much debated upon in the UK and USA (Link and Scott 2013).

The current issues with commercialisation output indicators call for other indicators that would better reflect the downstream innovation process taking place in the firms to provide a more complete picture of public science impact on the industry.

Innovation of industry or firms

The third block of the Impact Assessment Framework focuses on the innovation of industry or firms. Table 5 summarises the studies using firm’s innovation input and outcome indicators.

Table 5 Summary of studies using industry/firm innovation input/outcome indicators
Full size table

Some of the studies referred to the knowledge production function (Jaffe 1989). Acs et al. (1994) built on knowledge production function to ascertain that small firms could benefit (with increase in innovation count) from the knowledge of universities and the R&D centres of large firms. Varga (2000), on the other hand, applied the Griliches-Jaffe knowledge production function framework to establish that firms located in close proximity to the research university can benefit economically with an increase in their number of innovative products.

In 1998, Crépon et al. (1998) integrated both Cobb–Douglas production function and knowledge production function to develop the CDM model, which depicts two key relationships pursuant to a firm’s decision to invest in innovation. The first relationship describes the link between innovation input (primarily R&D) and innovation output (product or process innovation) as accepted in the knowledge production function (Pakes and Griliches 1984). The second relationship established the connection between innovation output and productivity based on Cobb–Douglas production function. Since then, many studies have built on the CDM model using labour productivity (sales per employee) in Spain (Barajas et al. 2012) and in the Netherlands (Lööf and Heshmati 2002) and turnover growth in Italy (Evangelista and Vezzani 2010).

Based on the CDM model, we have classified the following indicators as input to the innovation process of firms: (a) corporate R&D stocks, (b) innovation expenditures, (c) corporate patents, (d) investment in building and equipment for R&D and (e) induced investments. In particular, Pressman et al. (1995, p. 30) surveyed a sample of MIT licensees by collecting information on their induced investment defined as “Money spent developing new products and efficient ways to produce and market these products. It excludes the costs of producing (or investment required to produce) mature products”. From the survey, they extrapolated their data to estimate an economic impact of US$922 million.

Using a similar approach, Kramer et al. (1997) estimated that University of Pennsylvania’s licensees created an economic impact of US$4.6 billion in the form of induced investments. Although Pressman et al. (1995) and Kramer et al. (1997) have used induced investments to measure economic impact, these values are actually input to the firm’s innovation process that should be distinguished from its innovation outcome. According to Eurostat (2012), innovation outcome may be in the form of product innovation (e.g. sales from innovative products) or process innovation (e.g. productivity gain from innovative process) (Goldstein and Drucker 2006). The distinction between innovation input and innovation outcome indicators is clearly depicted in block 3 of the framework in Fig. 4.

While many studies continue to use innovative product sales or proportion of new products as indicators of firms’ innovative outcome, there are recommendations to refine innovative outcome modes and indicators. In a recent study, Evangelista and Vezzani (2010) expanded Eurostat’s (2012) definition of innovation outcome from two to four modes to examine economic impact: (a) pure product innovation, (b) pure process innovation, (c) organisation innovation and (d) mix of product, process and organisational innovations. Penfield et al. (2014) further articulated the need to distinguish among output, outcome and impact.

Community/economy

The fourth block of the framework comprises studies that use economic indicators such as employment, income, output and GDP growth, as summarised in Table 6.

Table 6 Summary of studies related to economic impact
Full size table

The impact studies in this category generally applied the input-output modelling or Keynesian multiplier approach. In UK, Harris’ (1997) study of the University of Portsmouth with the input-output approach estimated that the direct expenditure by the university sector resulted in an output multiplier effect of 1.24 to 1.73. Glasson’s (2003) input-output model found an output multiplier of between 0.70 and 1.12 for Sunderland University in Northeast England.

Using Keynesian multiplier approach, Brownrigg (1973) arrived at income multiplier between 1.45 and 1.80 based on a case study of the University of Stirling, while Huggins and Cooke (1997) found the corresponding multiplier at 1.46 to 1.52 at the Cardiff University. In the USA, Felsenstein (1996) applied econometric model on the input-output model to quantify the contribution of the Northwestern University to the Chicago metropolitan area at an output multiplier of 3.1 and employment multiplier of 1.55 in 1993. In a more recent study, Roessner et al. (2013) estimated the impact of university licensing on the whole US economy with a range of hypothetical licensing royalty rates at 2, 5 and 10 %, based on AUTM data (1996–2010) and product substitution effect adjustment.

Despite the popularity of the input-output and Keynesian multiplier approaches, these methodologies have raised some concerns. First, these techniques are restricted to estimating the backward linkages arising only from university expenditures and investment. The dynamic impact of university knowledge production and other forms of output on the business community are not captured (Martin 1998). Second, there is a lack of counterfactual hypotheses for comparison of impact with and without the university’s presence, giving rise to wide variation in results (Goldstein and Drucker 2006, p. 24). Third, the input-output model assumes that product sales arising from university/PRI-licensed IPs are all contained within the local economy to contribute to GDP. This assumption becomes invalid when a significant proportion of IPs are commercialised overseas due to the small domestic market size. Fourth, the input-output model treats university/PRI licensing and research income as expenditures that can have direct and indirect effect on the output and employment of other industries within the local economy. However, in a small country with an open economy, it is unlikely that it has the capability and capacity to provide all the equipment, materials and talent required to conduct R&D.

Toward an integrated framework and the role of RICV

We understand that different indicators are useful and relevant for tracking the effects of intermediate output or outcome at the different stages of the long journey of research, innovation and enterprise (RIE) value-added activities involving multiple stakeholders from science produced by universities/PRIs through the innovation outcome generated by the industry/firms to economic development:

  1. 1.

    Inventive output of universities/PRIs, e.g. R&D intensity, invention or technology disclosures, publications and patents;

  2. 2.

    Commercialisation output of universities/PRIs, e.g. licenses executed, spin-offs;

  3. 3.

    Firm/industry’s innovation input, e.g. corporate R&D stocks and its innovation outcome, e.g. innovative product sales and RICV;

  4. 4.

    Economic development in the form of employment, income and RICV at the economy level.

Given that RIE is a complex and iterative value chain with multiple feedback loops, it is apparent that no single indicator can furnish a complete picture of public-funded research and innovation impact. We therefore propose an integrated Impact Assessment Framework for universities/PRIs as a starting point to manage their value chain. The framework provides the policy makers and scholars the flexibility to define and refine indicators at different stages of the RIE journey while keeping a holistic view of the entire value chain, to facilitate periodic monitoring, analytics and adaptation at micro-, meso- and macro-levels. In particular, the framework includes a relatively new indicator, RICV, to estimate innovation outcome at firm or industry level, as well as economic impact at a regional or national level.

Implications and conclusion

In this paper, we addressed our research question whether RICV is a suitable predictor of firms’ propensity to repeat their in-licensing agreements with PRIs, compared to the traditional licensing revenue indicator. Our results show that firms that incur higher IP licensing fees with universities/PRIs have lower propensity to repeat their IP licensing transactions, thereby reducing academic innovation impact on firms. On the other hand, firms that attain higher levels of RICV with universities/PRIs are found to have higher propensity to repeat their license transactions, indicating an increase in academic innovation impact on firms. With results confirming our hypotheses, we contribute to the literature of impact studies by proposing new secondary RICV indicators to estimate the economic impact of public-funded technological innovation at the firm, industry and national levels.

Our finding of RICV as a predictor of firms’ propensity to repeat licensing engagement with universities/PRI has contributed to the TCE theory. In most studies of the current literature, the TCE is used to explain the propensity of firms to license technologies from other firms (pecuniary sources as traditional licensors). Given that these public institutions are non-pecuniary sources as they have socio-economic rather than pure financial objectives, the determinants of in-licensing propensity in firm-public institution engagement are likely to be different from that in firm-firm transaction. This study therefore adds a new dimension of RICV to the TCE theory.

In addition, our results show that the NPV theory from corporate finance, which has erstwhile been useful in evaluating investment decisions at the firm level, can be extended to the technology transfer discipline to estimate the impact of public research and innovation at the industry and economy level.

This study has made several managerial implications. As shown in Fig. 2, Singapore PRIs started using RICV since 2010 as part of their balanced scorecard strategy for performance management to measure and monitor the quality of the licenses signed and their impact at the firm level. Totalling the RICVs of all licensees for a particular year allows impact estimation at the economy level for the year. From 2010 to 2012, the PRIs have seen a significant increase in total RICV by more than 80 % per annum, with corresponding increase in total licensing revenues. Second, by grouping the RICV of firms by industry, the PRIs could obtain a better grasp of research and innovation impact at the industry level. Trending analysis of RICV by technology has enabled the identification of promising and complementary technologies with rising RICVs to form innovation clusters to serve growing industry sectors. To date, three innovation clusters have been formed and increased their contribution from less than 5 % of total RICV in 2010 to more than one third in 2012.

The findings of this study indicate that RICV is a better indicator than the traditional licensing revenue metric used by universities/PRIs to measure and monitor innovation impact in terms of firm innovation output. Although this study was carried out with the public research organisations of Singapore, the findings are generalisable to those of other countries that pursue the strategy of innovation-driven economic development. Universities/PRIs of knowledge-based economies face similar challenges in measuring the intermediate input and output of their respective RIE value chain activities, as evident in their adoption of common traditional indicators for benchmarking despite their limitations (AUTM 2014). To address the challenges and limitations, the universities/PRIs of the other innovation-driven economies are invited to use the RICV approach to inform strategic decision-making in shaping research agenda, allocating resources, influencing inventive outputs and determining the effectiveness of commercialisation outputs. The implementation of the RICV indicator in various universities/PRIs will build confidence in the indicator.

The policy makers of innovating economies may use the RICV and related indicators to benchmark the impact of research and innovation on enterprise and economy. Firms could also adopt and adapt the RICV approach to actively manage their IP and technology portfolio so as to increase the net present value of revenue streams generated by their products and services enabled by their IP and technology portfolio.

As an extension for this study, we propose development of a practical implementation guide for institutions seeking to use the framework and RICV indicator. The guide should comprise the development of tools, processes and training to enable the setup of a scalable and flexible system that can cater to the needs and objectives of the institutions.

Our study has identified a few areas for future research. First, while our paper focuses on the economic impact and STI indicators gleaned from our review of 275 articles, we acknowledge that there is much value in combining narratives with both qualitative and quantitative metrics. Although qualitative approaches such as field interviews and case study interrogations could provide rich and useful data, they could be very resource-intensive. The public institutions may therefore consider embarking on in-depth case studies (e.g. RVM) for completed large-scale R&D projects once every few years, not only to study about the performance of specific projects but also to use them to baseline and compare the relative significance of the current set of applicable indicators (as synthesised in our proposed Impact Assessment Framework), as well as analyse their relationships to evaluate their effectiveness.

Second, we are aware that RIE process from PRIs/universities’ receipt of R&D grant award (to perform R&D) to their TTOs’ successful commercialisation of IP generated (with licensing firms) may take an average of 3 to 8 years, involving multiple organisations and departments. Although it is widely acknowledged that the process is non-linear, there is increasing interest among PRIs and governments (Smith et al. 2011) to trace the efficiency of the various stages of the process to evaluate its overall performance. Future studies may embark on this area to identify the determinants of the conversion efficiency among the intermediate outputs, for example, from invention disclosures to patents, from patents to licenses and from licenses to products.