, Volume 94, Issue 1, pp 247–262 | Cite as

Benchmarking regional innovative performance: composite measures and direct innovation counts

  • Teemu MakkonenEmail author
  • Robert P. van der Have


There is a considerable amount of discussion, but still no consensus, about which indicator should be used to measure innovation. To participate in this debate, a unique innovation database, SFINNO, is introduced. Innovation counts from the database are used as the baseline, to which individual proxy indicators (patent- and research and development statistics) of innovation and innovation indexes, constructed here with principal component analysis, are compared. The local administrative units of Finland serve as the regional units benchmarked. The study results show that innovation is a complex phenomenon which cannot be entirely explained through the use of proxy statistics, as the linkages between innovation input- and output-indicators are fuzzy. We also show that the strength of these linkages varies by field of technology. Furthermore, different innovation measures produce highly divergent rankings when they are used as benchmarking tools of regional innovative performance. Although the produced innovation indexes perform slightly better, their superiority is marginal. Therefore, caution should be taken before drawing too drastic policy conclusions depending on a single measure of regional innovative performance.


Innovation Composite indicators Patents R&D Regional innovative performance 

MSC Classification

62H25 62H20 

JEL Classification

O18 O30 R11 



This work is partly funded by the Academy of Finland (project 127213). We acknowledge the financial support from VTT Technical Research Centre of Finland and the Finnish Funding Agency for Technology and Innovation (Tekes) that have enabled the construction of the SFINNO database. Finally, we are grateful for the anonymous reviewers’ suggestions to improve the paper. Remaining errors are ours.


  1. Acs, Z. (2002). Innovation and the growth of cities. Cheltenham: Edward Elgar Publishers.Google Scholar
  2. Acs, Z., Anselin, L., & Varga, A. (2002). Patents and innovation counts as measures of regional production of new knowledge. Research Policy, 31(7), 1069–1085.CrossRefGoogle Scholar
  3. Acs, Z., & Audretsch, D. (1993). Analysing innovation output indicators: The US experience. In A. Kleinknecht & D. Bain (Eds.), New concepts in innovation output measurement (pp. 10–41). New York: Saint Martin’s Press.Google Scholar
  4. Acs, Z., Audretsch, D., & Feldman, M. (1992). Real effects of academic research: Comment. The American Economic Review, 82(1), 363–367.Google Scholar
  5. Acs, Z., & Varga, A. (2002). Geography, endogenous growth and innovation. International Regional Science Review, 25(1), 132–148.CrossRefGoogle Scholar
  6. Altvater-Mackensen, N., Balicki, G., Bestakowa, L., Bocatius, B., Braun, J., et al. (2005). Science and technology in the region: The output of regional science and technology, its strengths and its leading institutions. Scientometrics, 63(3), 463–529.CrossRefGoogle Scholar
  7. Archibugi, D., & Coco, A. (2005). Measuring technological capabilities at the country level: A survey and a menu for choice. Research Policy, 34(2), 175–194.CrossRefGoogle Scholar
  8. Archibugi, D., & Pianta, M. (1996). Measuring technological change through patents and innovation surveys. Technovation, 16(9), 451–468.CrossRefGoogle Scholar
  9. Arundel, A. (2001). The relative effectiveness of patents and secrecy for appropriation. Research Policy, 30(4), 611–624.CrossRefGoogle Scholar
  10. Arundel, A., & Kabla, I. (1998). What percentage of innovations are patented? Empirical estimates for European firms. Research Policy, 27(2), 127–141.CrossRefGoogle Scholar
  11. Beard, T., Ford, R., Koutsky, T., & Spiwak, L. (2009). A valley of death in the innovation sequence: An economic investigation. Research Evaluation, 18(5), 343–356.CrossRefGoogle Scholar
  12. Beneito, P. (2006). The innovative performance of in-house and contracted R&D in terms of patents and utility models. Research Policy, 35(4), 502–517.CrossRefGoogle Scholar
  13. Booysen, F. (2002). An overview and evaluation of composite indices of development. Social Indicators Research, 59(2), 115–151.CrossRefGoogle Scholar
  14. Buesa, M., Heijs, J., & Baumert, T. (2010). The determinants of regional innovation in Europe: A combined factorial and regression knowledge production function approach. Research Policy, 39(6), 722–735.CrossRefGoogle Scholar
  15. Buesa, M., Heijs, J., Martínez Pellitero, M., & Baumert, T. (2006). Regional systems of innovation and the knowledge production function: The Spanish case. Technovation, 26(4), 463–472.CrossRefGoogle Scholar
  16. Camacho, J., & Rodríguez, M. (2005). How innovative are services? An empirical analysis for Spain. The Service Industries Journal, 25(2), 253–271.CrossRefGoogle Scholar
  17. Carayannis, E., & Provance, M. (2008). Measuring firm innovativeness: Towards a composite innovation index built on firm innovative posture, propensity and performance attributes. International Journal of Innovation and Regional Development, 1(1), 90–107.CrossRefGoogle Scholar
  18. Chen, P., & Popovich, P. (2002). Correlation: Parametric and nonparametric measures. Quantitative Applications in the Social Sciences, 139.Google Scholar
  19. Coad, A., & Rao, R. (2008). Innovation and firm growth in high-tech sectors: A quantile regression approach. Research Policy, 37(4), 633–648.CrossRefGoogle Scholar
  20. Coombs, R., Narandren, P., & Richards, A. (1996). A literature-based innovation output indicator. Research Policy, 25(3), 403–413.CrossRefGoogle Scholar
  21. Damanpour, F. (1991). Organizational innovation: A meta-analysis of effects of determinants and moderators. Academy of Management Journal, 34(3), 555–590.CrossRefGoogle Scholar
  22. David, M., & Sutton, C. (2004). Social research: The basics. London: SAGE Publications.Google Scholar
  23. de Bruijn, P., & Lagendijk, A. (2005). Regional innovation systems in the Lisbon strategy. European Planning Studies, 13(8), 1153–1172.CrossRefGoogle Scholar
  24. Edwards, K., & Gordon, T. (1984). Characterization of innovations introduced on the US market in 1982. Washington: US Small Business Administration.Google Scholar
  25. Fagerberg, J., & Srholec, M. (2008). National innovation systems, capabilities and economic development. Research Policy, 37(9), 1417–1435.CrossRefGoogle Scholar
  26. Fagerberg, J., & Verspagen, B. (2009). Innovation studies—The emerging structure of a new scientific field. Research Policy, 38(2), 218–233.CrossRefGoogle Scholar
  27. Feeny, S., & Rogers, M. (2003). Innovation and performance: Benchmarking Australian firms. The Australian Economic Review, 36(3), 253–264.CrossRefGoogle Scholar
  28. Gössling, T., & Rutten, R. (2007). Innovation in regions. European Planning Studies, 15(2), 253–270.CrossRefGoogle Scholar
  29. Griliches, Z. (Ed.). (1984). R and D, patents and productivity. Chicago: University of Chicago Press.Google Scholar
  30. Grupp, H., & Mogee, M. (2004). Indicators for national science and technology policy: How robust are composite indicators? Research Policy, 33(9), 1373–1384.CrossRefGoogle Scholar
  31. Grupp, H., & Schubert, T. (2010). Review and new evidence on composite innovation indicators for evaluating national performance. Research Policy, 39(1), 67–78.CrossRefGoogle Scholar
  32. Gu, W., & Tang, J. (2004). Link between innovation and productivity in Canadian manufacturing industries. Economics of Innovation and New Technology, 13(7), 671–686.CrossRefGoogle Scholar
  33. Hagedoorn, J., & Cloodt, M. (2003). Measuring innovative performance: Is there and advantage in using multiple indicators? Research Policy, 32(8), 1365–1379.CrossRefGoogle Scholar
  34. Hall, B., Jaffe, A., & Trajtenberg, M. (2005). Market value and patent citations. RAND Journal of Economics, 36(1), 16–38.Google Scholar
  35. Hipp, C., & Grupp, H. (2005). Innovation in the service sector: The demand for service-specific innovation measurement concepts and typologies. Research Policy, 34(4), 517–535.CrossRefGoogle Scholar
  36. Hollenstein, H. (1996). A composite indicator of firm’s innovativeness: An empirical analysis based on survey data for Swiss manufacturing. Research Policy, 25(4), 633–645.CrossRefGoogle Scholar
  37. Huizingh, E. (2011). Open innovation: State of the art and future perspectives. Technovation, 31(1), 2–9.CrossRefGoogle Scholar
  38. Jaffe, A. (1989). Real effects of academic research. The American Economic Review, 79(5), 957–970.Google Scholar
  39. Jolliffe, I. (2002). Principal component analysis. New York: Springer.zbMATHGoogle Scholar
  40. Kaasa, A. (2009). Effects of different dimensions of social capital on innovative activity: Evidence from Europe at the regional level. Technovation, 29(3), 218–233.CrossRefGoogle Scholar
  41. Kleinknecht, A., van Montfort, K., & Brouwer, E. (2002). The non-trivial choice between innovation indicators. Economics of Innovation and New Technology, 11(2), 109–121.CrossRefGoogle Scholar
  42. Kline, S., & Rosenberg, N. (1986). An overview of innovation. In R. Landau & N. Rosenberg (Eds.), The positive sum strategy: Harnessing technology for economic growth (pp. 275–305). Washington: National Academy Press.Google Scholar
  43. Lepori, B., Barré, R., & Filliatreau, G. (2008). New perspectives and challenges for design and production of S&T indicators. Research Evaluation, 17(1), 33–44.CrossRefGoogle Scholar
  44. Lieberman, M. (1989). The learning curve, technology barriers to entry and competitive survival in the chemical processing industries. Strategic Management Journal, 10(5), 431–447.MathSciNetCrossRefGoogle Scholar
  45. Mäkinen, I. (2007). To patent or not to patent? An innovation-level investigation of the propensity to patent. VTT Publications, 646.Google Scholar
  46. Mendonça, S., Pereira, T., & Godinho, M. (2004). Trademarks as an indicator of innovation and industrial change. Research Policy, 33(9), 1385–1404.CrossRefGoogle Scholar
  47. Michel, J., & Bettels, B. (2001). Patent citation analysis: A closer look at the basic input data from patent search reports. Scientometrics, 51(1), 185–201.CrossRefGoogle Scholar
  48. Molas-Gallart, J., & Davies, A. (2006). Toward theory-led evaluation. American Journal of Evaluation, 27(1), 64–82.CrossRefGoogle Scholar
  49. Moon, H.-S., & Lee, J.-D. (2005). A fuzzy set theory approach to national composite S&T indices. Scientometrics, 64(1), 67–83.MathSciNetCrossRefGoogle Scholar
  50. Nelson, A. (2009). Measuring knowledge spillovers: What patents, licenses and publications reveal about innovation diffusion. Research Policy, 38(6), 994–1005.CrossRefGoogle Scholar
  51. OECD. (1992). Oslo manual: Proposed guidelines for collecting and interpreting technological innovation data. Paris: OECD Publications.Google Scholar
  52. OECD. (2005). Oslo manual: Guidelines for collecting and interpreting innovation data. Paris: OECD Publications.Google Scholar
  53. OECD. (2011). OECD science, technology and industry scoreboard 2011. Paris: OECD Publications.CrossRefGoogle Scholar
  54. Palmberg, C., Leppälahti, A., Lemola, T., & Toivanen, H. (1999). Innovations and industrial renewal in Finland—A new perspective. VTT Working Papers, 41.Google Scholar
  55. Perez, C., & Soete, L. (1988). Catching up in technology: Entry barriers and windows of opportunity. In G. Dosi, C. Freeman, R. Nelson, G. Silverberg, & L. Soete (Eds.), Technical change and economic theory (pp. 458–479). London: Pinter.Google Scholar
  56. Pinto, H. (2009). The diversity of innovation in the European Union: Mapping latent dimensions and regional profiles. European Planning Studies, 17(2), 304–326.CrossRefGoogle Scholar
  57. Pinto, H., & Guerreiro, J. (2010). Innovation regional planning and latent dimensions: The case of the Algarve region. Annals of Regional Science, 44(2), 315–329.CrossRefGoogle Scholar
  58. Ratanawaraha, A., & Polenske, K. (2007). Measuring the geography of innovation: A literature review. In K. Polenske (Ed.), The economic geography of innovation (pp. 30–59). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  59. Rothwell, R. (1992). Successful industrial innovation: Critical factors for the 1990 s. R&D Management, 22(3), 221–238.CrossRefGoogle Scholar
  60. Rothwell, R., & Gardiner, P. (1988). Re-innovation and robust designs: Producer and user benefits. Journal of Marketing Management, 3(3), 372–387.CrossRefGoogle Scholar
  61. Saisana, M., Saltelli, A., & Tarantola, S. (2005). Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. Journal of the Royal Statistical Society. Series A, 168(2), 307–323.MathSciNetzbMATHGoogle Scholar
  62. Schmoch, U., & Gauch, S. (2009). Service marks as indicators for innovation in knowledge-based services. Research Evaluation, 18(4), 323–335.CrossRefGoogle Scholar
  63. Schmookler, J. (1950). The interpretation of patent statistics. Journal of the Patent Office Society, 32(2), 123–146.Google Scholar
  64. Schumpeter, J. (1934). The theory of economic development: An inquiry into profits, capital, credit, interest and the business cycle. New York: Oxford University Press.Google Scholar
  65. Slaper, T., Hart, N., Hall, T., & Thompson, M. (2011). The index of innovation: A new tool for regional analysis. Economic Development Quarterly, 25(1), 36–53.CrossRefGoogle Scholar
  66. Smith, K. (2005). Measuring innovation. In J. Fagerberg, D. Mowery, & R. Nelson (Eds.), The Oxford handbook of innovation (pp. 148–177). New York: Oxford University Press.Google Scholar
  67. Tabachnich, B., & Fidell, L. (2007). Using multivariate statistics. Boston: Pearson.Google Scholar
  68. Tang, J., & Le, C. (2007). Multidimensional innovation and productivity. Economics of Innovation and New Technology, 16(7), 501–516.CrossRefGoogle Scholar
  69. Unger, B. (2000). Innovation systems and innovative performance: Voice systems. Organization Studies, 21(5), 941–969.MathSciNetCrossRefGoogle Scholar
  70. van der Panne, G. (2007). Issues in measuring innovation. Scientometrics, 71(3), 495–507.CrossRefGoogle Scholar
  71. van der Panne, G., & van Beers, C. (2006). On the Marshall-Jacobs controversy: It takes two to tango. Industrial and Corporate Change, 15(5), 877–890.CrossRefGoogle Scholar
  72. Varga, A. (2006). The spatial dimension of innovation and growth: Empirical research methodology and policy analysis. European Planning Studies, 14(9), 1171–1186.CrossRefGoogle Scholar
  73. Zabala-Iturriagagoitia, J., Voigt, P., Gutiérrez-Gracia, A., & Jiménez-Sáez, F. (2007). Regional innovation systems: How to assess performance. Regional Studies, 41(5), 661–672.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2012

Authors and Affiliations

  1. 1.Department of Geosciences and GeographyUniversity of HelsinkiHelsinkiFinland
  2. 2.Organizations, Networks and Innovation SystemsVTT Technical Research Centre of FinlandVTTFinland
  3. 3.Department of Industrial Engineering and ManagementAalto UniversityEspooFinland

Personalised recommendations