Abstract
The returns to R&D literature is large and has been surveyed on several occasions. We complement previous surveys by discussing the scope for publication bias and illustrate how formal meta analytic techniques can be used to analyse the problem. We find evidence consistent with positive publication bias and discuss possible interpretations. The bias appears to be particularly strong in the part of the literature that controls for unobserved firm fixed effects. The reason may be that fixed effects specifications are particularly susceptible to measurement errors and therefore have a high probability of producing implausibly low return estimates. Implausible estimates are likely to be filtered out before being reported, and our analysis suggests that 23 % of a hypothetical complete literature is missing. Future reviews should take into account that the full effect of negative specifications biases may be masked by reporting and publication bias.
Similar content being viewed by others
Notes
This is supported by a recent and more formal analysis by Castellacci and Lie (2015).
Wieser (2005) constructed his meta sample from earlier narrative reviews.
This being said we fully acknowledge the advantages of systematic reviews, and that this is the gold standard in meta analysis, cf. Stanley et al. (2013). Systematic literature reviews recently made available by Ugur et al. (2015) and OECD (2015) find that there exists a large number of results that have been ignored in the earlier surveys on the returns to R&D literature.
This was pointed out to us by an early referee, “aware of the research and publication history of many of the papers”. Brodeur et al. (2013) find that young researchers distort their reported results more in order to achieve statistical significance than tenured and older researchers.
The model can also be specified with labor productivity or total factor productivity as dependent variable.
In equation (2), R is the net investment in R&D, while only gross R&D is available in empirical work. Using an R&D measure that is too large, obviously causes ρ to be underestimated. Hall et al. (2010) show that this bias may be substantial. Eberhardt et al. (2013), on the other hand, show that private returns to R&D estimates may suffer from a positive bias as they are based on specifications that ignore the effect of R&D spillovers. R&D spillovers are positively correlated with private R&D investments. Much of this bias, however, is likely to be absorbed by firm fixed effects.
See Karlsson et al. (2013) for a formal meta-analysis of the spatial knowledge spillover litterature.
Strictly speaking, there may also be symmetric publication selection in which case Light and Pillemer (1984) suggest using the ‘hollowness’ of the funnel graph to indicate selection bias. Symmetric selection is less of a concern, since the mean effect found in meta-analyses will remain largely unbiased.
A common caveat in this literature, however, is that publication bias is only one possible cause of funnel plot asymmetry, see Sterne and Egger (2005). A cautious interpretation of results based on funnel graph asymmetry, therefore, is to consider them sensitivity analyses showing the potential effect of missing studies. We return to this issue in the “Robustness and Caveats” section.
The algorithm works as follows: The most extreme right-hand side effect sizes are removed one by one. Between each iteration, the overall effect size is estimated, and the procedure goes on until the funnel plot is symmetric around the latest computed overall effect size. This trimming gives an unbiased effect size estimate, but also reduces the variance of the effects. Therefore, the algorithm adds back the original results and augments the sample with their mirror images before computing the confidence interval.
In the empirical part of this paper, we use the distribution free moment-based estimator of DerSimonian and Laird (1986). Kontopantelis and Reeves (2012), in a comprehensive simulation study, conclude that researchers may have “confidence that, whichever method they adopt, results are highly robust against even very severe violations of the assumption of normally distributed effect sizes”.
The algorithm uses the non-iterative moment-based estimator of DerSimonian and Laird (1986) to estimate the between studies variance. Hence, our results in Table 3 do not depend strictly on any distributional assumption regarding the random effect, cf. the discussion in the section “Correcting for Publication Bias”.
Testing the random effects model against the fixed effects model using the so-called Q-test presented in Shadish and Haddock (1994, p. 266), clearly rejects the fixed effects model in all three samples. The fixed effects point estimates before adjusting for potential publication bias are 0.151, 0.099, and 0.021 for columns (1), (2), and (3), respectively. The corresponding estimates with correction for publication bias are 0.126, 0.097, and 0.007. Hence, our results are not highly sensitive to the choice between the fixed effects and the random effects model.
As a partial robustness test, we have divided our meta sample in two according to the median sample size of the primary studies. It turns out that our main conclusions based on Table 2 do not depend on combining samples with large and small n in one regression. We get similar results using only the larger half of the samples as using the smaller half of the samples.
This possibility is—as far as we know—not previously discussed in the meta-analysis literature.
We are grateful to Jonas Andersson for providing these simulations.
References
Abreu, M., de Groot, H.L.F., & Florax, R.J.G.M. (2005). A meta-analysis of β-convergence: the legendary 2 %. Journal of Economic Surveys, 19(3), 389–420.
Ashenfelter, O., Harmon, C., & Oosterbeek, H. (1999). A review of estimates of the schooling/earnings relationship, with tests for publication bias. Labour Economics, 6(4), 453–470.
Australian Industry Commission. (1995). Research and development report No. 44 Vol. 3. Canberra: Australian Government Publishing Service.
Bartelsman, E., van Leeuwen, G., Nieuwenhuijsen, H., & Zeelenberg, K. (1996). R&D, and productivity growth: evidence from firm-level data in the Netherlands. Netherlands Official Statistics, 11, 52–69.
Bond, S., Harhoff, D., & van Reenen, J. (2003). Corporate R&D and productivity in Germany and the United Kingdom, CEP Discussion Papers 0599.
Borenstein, M., Hedges, L.V., & Rothstein, H.R. (2007). Introduction to meta-analysis. Section: fixed vs. random effects models. http://www.Meta-Analysis.com.
Brodeur, A., Le, M., Sangnier, M., & Zylberberg, Y. (2013). Star wars: the empirics strike back, IZA Discussion Papers 7268.
Capron, H., & Cincera, M. (1998). Exploring the spillover impact on productivity of world-wide manufacturing firms. Annales d’Economie et de Statistiques, 49/50, 565–588.
Card, D., & Krueger, A.B. (1995). Time-series minimum-wage. American Economic Review, 85(2), 238–243.
Castellacci, F., & Lie, C. (2015). Do the effects of R&D tax credits vary across industries? A meta-regression analysis. Research Policy, 44(4), 819–832.
Cincera, M. (1998). Technological and economic performances of international firms. PhD Thesis, Université Libre de Bruxelles, Belgium.
Clark, K.B., & Griliches, Z. (1984). Productivity growth and R&D at the business level: results from the PIMS data base. In Z. Griliches (Ed.), R&D, patents and productivity. Chicago: University of Chicago Press.
Crepon, B., Duguet, E., & Mairesse, J. (1998). Research, innovation, and productivity: an econometric analysis at the firm level. Economics of Innovation and New Technology, 7(2), 115–156.
Cuneo, P., & Mairesse, J. (1984). Productivity and R&D at the firm level in French manufacturing. In Z. Griliches (Ed.), R&D, patents and productivity. Chicago: University of Chicago Press.
DeLong, J.B., & Lang, K. (1992). Are all economic hypotheses false? Journal of Political Economy, 100(6), 1257–1272.
DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clincal Trials, 7, 177–188.
DerSimonian, R., & Kacker, R. (2007). Random-effects model for meta-analysis of clinical trials: an update. Contemporary Clincal Trials, 28, 105–114.
Doraszelski, U., & Jaumandreu, J. (2008). R&D and productivity: estimating production functions when productivity is endogenous, CEPR Discussion Papers 6636.
Doucouliagos, H., & Stanley, T.D. (2009). Publication selection bias in minimum-wage research? A meta-regression analysis. British Journal of Industrial Relations, 47(2), 406–428.
Duval, S., & Tweedie, R. (2000a). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95(449), 89–98.
Duval, S., & Tweedie, R. (2000b). Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56 (2), 455–463.
Eberhardt, M., Helmers, C., & Strauss, H. (2013). Do spillovers matter when estimating private returns to R&D? Review of Economics and Statistics, 95(2), 436–448.
Egger, M., Smith, G.D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315 (7109), 629–34.
Fecher, F. (1990). Effects directs et inderects de la R&D sur la productivite: une analyse de l’industrie manufacturiere Belge. Cahiers Economiques de Bruxelles, 128 (4), 459–482.
Goto, A., & Suzuki, K. (1989). R&D capital, rate of return on R&D investment and spillover of R&D in Japanese manufacturing industries. Review of Economics and Statistics, 71(4), 555–564.
Griffith, R., Harrison, R., & van Reenen, J. (2006). How special is the special relationship? Using the impact of U.S. R&D spillovers on U.K. firms as a test of technology sourcing. American Economic Review, 96(5), 1859–1875.
Griliches, Z. (1973). Research expenditures and growth accounting. In B.R. Williams (Ed.), Science and Technology in Economic Growth. Macmillan, New York.
Griliches, Z. (1979). Issues in assessing the contribution of research and development to productivity growth. Bell Journal of Economics, 10(1), 92–116.
Griliches, Z. (1980). R&D and the productivity slowdown. American Economic Review, 70(2), 343–348.
Griliches, Z. (1986). Productivity, R&D and basic research at the firm level in the 1970s. American Economic Review, 76(1), 141–154.
Griliches, Z. (1992). The Search for R&D Spillovers. Scandinavian Journal of Economics, 94, S29–47.
Griliches, Z. (1995). R&D and productivity: econometric results and measurement issues. In P. Stoneman (Ed.), Handbook of the economics of innovation and technical change. Oxford: Blackwell.
Griliches, Z. (2000). R&D, education, and productivity: a retrospective. Cambridge: Harvard University Press.
Griliches, Z., & Hausman, J.A. (1986). Errors in variables in panel data. Journal of Econometrics, 31(1), 93–118.
Griliches, Z., & Mairesse, J. (1983). Comparing productivity growth: an exploration of French and US industrial and firm data. European Economic Review, 21(1–2), 89–119.
Griliches, Z., & Mairesse, J. (1984). Productivity and R&D at the firm level. In Z. Griliches (Ed.), R&D, patents and productivity. Chicago: University of Chicago Press.
Griliches, Z., & Mairesse, J. (1990). R&D and productivity growth: comparing Japanese and US manufacturing firms. In C.R. Hulten (Ed.), Productivity growth in Japan and the United States. Chicago: University of Chicago Press.
Görg, H., & Strobl, E.A. (2001). Multinational companies and productivity spillovers: a meta-analysis. Economic Journal, 111(475), 723–739.
Haelermans, C., & Borghans, L. (2012). Wage effects of on-the-job-training: a meta analysis. British Journal of Industrial Relations, 50(3), 502–528.
Hall, B.H. (1993). Industrial research during the 1980s: did the rate of return fall? Brookings papers on economic activity. Microeconomics, (2), 289–344.
Hall, B.H. (1996). The private and social returns to research and development. In B.L.R. Smith, & C.E. Barfield (Eds.), Technology, R&D, and the economy. Washington: Brookings Institution and American Enterprise Institute.
Hall, B.H., Foray, D., & Mairesse, J. (2009). Pitfalls in estimating the returns to corporate R&D using accounting data. Revised version of a paper presented at the First European Conference on Knowledge for Growth, October 8–9, 2007, Seville, Spain.
Hall, B.H., & Mairesse, J. (1995). Exploring the relationship between R&D and productivity in French manufacturing firms. Journal of Econometrics, 65(1), 263–293.
Hall, B.H., Mairesse, J., & Mohnen, P. (2010). Measuring the returns to R&D. In B.H. Hall, & N. Rosenberg (Eds.), Handbook of the economics of innovation, Vol. 2. New York: Elsevier.
Harhoff, D. (1998). R&D and productivity in german manufacturing firms. Economics of Innovation and New Technology, 6(1), 29–50.
Ientile, D., & Mairesse, J. (2009). A policy to boost R&D: Does the R&D tax credit work? EIB Papers 6/2009, European Investment Bank.
Kafouros, M. (2005). R&D and productivity growth: evidence from the UK. Economics of Innovation and New Technology, 14(6), 479–497.
Karlsson, C., Warda, P., & Gråsjö, U. (2013). Spatial knowledge spillovers in Europe: a meta-analysis. In E.G. Carayannis, & G.M. Korres (Eds.), The innovation union in europe a socio-economic perspective on EU integration, science, innovation, technology and entrepreneurship series (pp. 144–175). Cheltenham: Edward Elgar Publishing.
Klette, T.J. (1991). On the importance of R&D and ownership for productivity growth: evidence from Norwegian Micro-Data 1976–1985, Statistics Norway Discussion Paper No. 60.
Kontopantelis, E., & Reeves, D. (2012). Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: a simulation study. Statistical Methods in Medical Research, 21(4), 409–426.
Kwon, H.U., & Inui, T. (2003). R&D and productivity growth in Japanese manufacturing firms. Economic and Social Research Institute Discussion Paper No. 44.
Lau, J., Ioannidis, J.P.A., Terrin, N., Schmid, C.H., & Olkin, I. (2006). Evidence based medicine. The case of the misleading funnel plot. BMJ: British Medical Journal, 333(7568), 597–600.
Lichtenberg, F.R., & Siegel, D. (1991). The impact of R&D investment on productivity—New evidence using linked R&D – LRD data. Economic Inquiry, 19(2), 535–551.
Light, Richard J., & Pillemer, David B. (1984). Summing up: the Science of Reviewing Research. Cambridge: Harvard University Press.
Link, A.N. (1981). Research and development activity in U. S. manufacturing. New York: Praeger.
Link, A.N. (1983). Inter-firm technology flows and productivity growth. Economics Letters, 11(2), 179–184.
Los, B., & Verspagen, B. (2000). R&D spillovers and productivity: evidence from U.S. manufacturing industries. Empirical Economics, 25(1), 127–148.
Mairesse, J., & Cunéo, P. (1985). Recherche-développement et performances des entreprises: une etude economé trique sur données individuelles. Revue Economique, 36(5), 1001–1042.
Mairesse, J., & Hall, B.H. (1996). Estimating the productivity of research and development: an exploration of GMM methods using data on French and United States manufacturing firms, NBER Working Paper No. 5501.
Mairesse, J., & Mohnen, P. (1990). Recherche-Développement et productivité: un survol de la littérature économétrique, Economie et Statistique. Programme National Persée, 237(1), 99–108.
Mairesse, J., & Sassenou, M. (1991). R&D productivity: a survey of econometric studies at the firm level, NBER Working Paper No. 3666.
Mansfield, E. (1980). Basic research and productivity increase in manufacturing. American Economic Review, 70(5), 863–873.
Medda, G., Piga, C.A., & Siegel, Donald S. (2003). On the relationship between R&D and productivity: a treatment effect analysis, Fondazionae Eni Enrico Mattei Nota di Lacoro 34-2003, Milano, Italy.
Minasian, J.R. (1962). The Economics of research and development. In R. Nelson (Ed.), The rate and direction of inventive activity: Economic and social factors (pp. 93–141). Cambridge: National Bureau of Economic Research.
Moreno, S.G., Sutton, A.J., Ades, A.E., Stanley, T.D., Abrams, K.R., Peters, J.L., & Cooper, N.J. (2009). Assessment of regression based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 9(2).
Nadiri, M.I. (1993). Innovations and technological spillovers, NBER Working Paper No. 4423.
Nelson, J.P. (2014). Estimating the price elasticity of beer: Meta-analysis of data with heterogeneity, dependence, and publication bias. Journal of Health Economics, 33, 180–187.
Neumark, D., & Wascher, W. (1998). Is the time-series evidence on minimum wage effects contaminated by publication bias? Economic Inquiry, 36(3), 458–470.
Odagiri, H. (1983). R&D expenditures, royalty payments, and sales growth in Japanese manufacturing corporations. Journal of Industrial Economics, 32(1), 61–71.
Odagiri, H., & Iwata, H. (1986). The impact of R&D on productivity increase in Japanese manufacturing companies. Research Policy, 15(1), 13–19.
OECD (2015). The impact of R&D investment on economic performance: A review of the econometric evidence, DSTI/EAS/STP/NESTI(2015)8, OECD, Paris.
O’Mahony, M., & Vecchi, M. (2000). Tangible and intangible investment and economic performance: evidence from company accounts. In P. Buigues, A. Jacquemin, & J.-F. Marchipont (Eds.) Competitiveness and the Value of Intangible Assets. Cheltenham: Edward Elgar Publishing Limited.
Ortega-Argiles, R., Piva, M., Potters, L., & Vivarelli, M. (2009). Is corporate R&D investment in high-tech sectors more effective? Some guidelines for European research policy. IZA Discussion Paper No. 3945.
Rogers, M. (2010). R&D and productivity: using UK firm-level data to inform polity. Empirica, 37(3), 329–359.
Sassenou, M. (1988). Recherche-developpement et productivite dans les entreprises Japonaises: Une Etude econometrique sur donnees de panel. Doctoral dissertation, Ecole des Hautes Etudes en Sciences Sociales, Paris.
Schankerman, M. (1981). The effects of double-counting and expensing on the measured returns to R&D. Review of Economics and Statistics, 63(3), 454–458.
Scherer, F.M., & Harhoff, D. (2000). Technology policy for a world of skew-distributed outcomes. Research Policy, 29(4–5), 559–566.
Shadish, W.R., & Haddock, C.K. (1994). Combining estimates of effect size. In H.M. Cooper, & L.V. Hedges (Eds.) The handbook of research synthesis. New York: Russel Sages Foundation.
Stanley, T.D. (2008). Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxford Bulletin of Economics and Statistics, 70(1), 103–127.
Stanley, T.D., & Doucouliagos, H. (2007). Identifying and correcting publication selection bias in the efficiency-wage literature: Heckman meta-regression. Deakin University School Working Paper, Economics Series 2007–11.
Stanley, T.D., & Doucouliagos, H. (2010). Picture this: a simple graph that reveals much ado about research. Journal of Economic Surveys, 24(1), 170–191.
Stanley, T.D., & Doucouliagos, H. (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 5(1), 60–78.
Stanley, T.D., & Doucouliagos, H. (2013). Better than random: weighted least squares meta-regression analysis. Deakin University School Working Paper, Economics Series 2013–2.
Stanley, T.D., Doucouliagos, H., Giles, M., Heckemeyer, J.H., Johnston, R.J., Laroche, P., Nelson, J.P., Paldam, M., Poot, J., Pugh, G., Rosenberger, R.S., & Rost, K. (2013). Meta-analysis of economics research reporting guidelines. Journal of Economic Surveys, 27(2), 390–94.
Steichen, T.J. (2001). Nonparametric “trim and fill” analysis of publication bias in meta-analysis. Stata Technical Bulletin, 10(57), 8–14.
Sterne, J.A., & Egger, M. (2005). Regression methods to detect publication and other bias in meta-analysis. In H.R. Rothstein, A.J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis—prevention, assessment and adjustments. Chichester: Wiley.
Sutton, A., Duval, S., Tweedie, R., Abrams, K., & Jones, D. (2000). Empirical assessment of effect of publication bias on meta-analyses. British Medical Journal, 320(7249), 1574–1577.
Tsai, K.-H., & Wang, J.-C. (2005). Productivity Growth and R&D expenditure in Taiwan’s manufacturing firms. Research Policy, 34(6), 966–976.
Ugur, M., Solomon, E., Guidi, F., & Trushinb, E. (2015). R&D and productivity in OECD firms and industries: a hierarchical meta-regression analysis. Mimeo, University of Greenwich Business School. Available at www.econstor.eu.
Wakelin, K. (2001). Productivity growth and R&D expenditure in UK manufacturing firms. Research Policy, 30(7), 1079–1090.
Wang, J.-C., & Tsai, K.-H. (2003). Productivity Growth and R&D expenditure in Taiwan’s manufacturing firms, NBER Working Paper No. 9724.
Wieser, R. (2005). Research and development productivity and spillovers: empirical evidence at the firm level. Journal of Economic Surveys, 19(4), 587–621.
Author information
Authors and Affiliations
Corresponding author
Additional information
The project is financed by the Research Council of Norway. We are grateful to Jonas Andersson, referees and participants at the MAER-Net 2013 colloquium at the University of Greenwich for useful comments.
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Møen, J., Thorsen, H.S. Publication Bias in the Returns to R&D Literature. J Knowl Econ 8, 987–1013 (2017). https://doi.org/10.1007/s13132-015-0309-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13132-015-0309-9