Assessing the health research’s social impact: a systematic review

  • Matteo Pedrini
  • Valentina Langella
  • Mario Alberto Battaglia
  • Paola Zaratin
Article

Abstract

In recent year, a growing attention is dedicated to the assessment of research’s social impact. While prior research has often dealt with results of research, the last decade has begun to generate knowledge on the assessment of health research’s social impact. However, this knowledge is scattered across different disciplines, research communities, and journals. Therefore, this paper analyzes the heterogeneous picture research has drawn within the past years with a focus on the health research’s social impact on different stakeholders through an interdisciplinary, systematic review. By consulting major research databases, we have analyzed 53 key journal articles bibliographically and thematically. We argued that the adoption of a multi-stakeholder could be an evolution of the existing methods used to assess impact of research. After presenting a model to assess the health research’s social impact with a multi stakeholder perspective, we suggest the implementation in the research process of three practice: a multi-stakeholder workshop on research agenda; a multi stakeholder supervisory board; a multi-stakeholder review process.

Keywords

Impact assessment Medical research Multi-stakeholder 

References

  1. Adam, P., Solans-Domènech, M., Pons, J. M. V., Aymerich, M., Berra, S., Guillamon, I., et al. (2012). Assessment of the impact of a clinical and health services research call in Catalonia. Research Evaluation, 21(4), 319–328.CrossRefGoogle Scholar
  2. Ahmed, S., Berzon, R. A., Revicki, D. A., Lenderking, W. R., Moinpour, C. M., Basch, E., et al. (2012). The use of patient-reported outcomes (PRO) within comparative effectiveness research implications for clinical practice and health care policy. Medical Care, 50(12), 60–70.CrossRefGoogle Scholar
  3. Amara, N., Ouimet, M., & Landry, R. (2004). New evidence on the instrumental, conceptual, and symbolic utilization of university research in government agencies. Science Communication, 26(1), 75–106.CrossRefGoogle Scholar
  4. Anderson, A. R. (1998). Cultivating the Garden of Eden: environmental entrepreneuring. Journal of Organizational Change Management, 11(2), 135–144.CrossRefGoogle Scholar
  5. Anderson, M., & McCleary, K. K. (2016). On the path to a science of patient input. Science Translational Medicine, 8(336), 336.CrossRefGoogle Scholar
  6. Banzi, R., Moja, L., Pistotti, V., Facchini, A., & Liberati, A. (2011). Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Research Policy and Systems, 9, 26.CrossRefGoogle Scholar
  7. Barker, K. (2007). The UK Research Assessment Exercise: the evolution of a national research evaluation system. Research Evaluation, 16(1), 3–12.CrossRefGoogle Scholar
  8. Barré, R. (2005). S&T indicators for policy making in a changing science–society relationship. In H. Moed, W. Glänzel & U. Schmoch (Eds.), Handbook of quantitative science and technology research (pp. 115–131). Dordrecht: Springer.CrossRefGoogle Scholar
  9. Bell, S., Shaw, B., & Boaz, A. (2011). Real-world approaches to assessing the impact of environmental research on policy. Research evaluation, 20(3), 227–237.CrossRefGoogle Scholar
  10. Bensing, J. M., Caris-Verhallen, W. M., Dekker, J., Delnoij, D. M., & Groenewegen, P. P. (2003). Doing the right thing and doing it right: toward a framework for assessing the policy relevance of health services research. International Journal of Technology Assessment in Health Care, 19(04), 604–612.CrossRefGoogle Scholar
  11. Bloch, C., Sørensen, M. P., Graversen, E. K., Schneider, J. W., Schmidt, E. K., Aagaard, K., et al. (2014). Developing a methodology to assess the impact of research grant funding: A mixed methods approach. Evaluation and program planning, 43, 105–117.CrossRefGoogle Scholar
  12. Boaz, A., Fitzpatrick, S., & Shaw, B. (2009). Assessing the impact of research on policy: a literature review. Science & Public Policy (SPP), 36(4), 255–270.CrossRefGoogle Scholar
  13. Bollen, J., Van de Sompel, H., & Rodriguez, M. A. (2008, June). Towards usage-based impact metrics: first results from the mesur project. In Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries (pp. 231–240). ACM.Google Scholar
  14. Bornmann, L. (2013a). Measuring the societal impact of research: research is less and less assessed on scientific impact alone—We should aim to quantify the increasingly important contributions of science to society. EMBO Reports, 13(8), 673–676.CrossRefGoogle Scholar
  15. Bornmann, L. (2013b). What is societal impact of research and how can it be assessed? a literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233.CrossRefGoogle Scholar
  16. Bornmann, L., & Marx, W. (2014). How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons. Scientometrics, 98(1), 211–219.CrossRefGoogle Scholar
  17. Boyd, A., Cole, D. C., Cho, D. B., Aslanyan, G., & Bates, I. (2013). Frameworks for evaluating health research capacity strengthening: a qualitative study. Health Research Policy and Systems, 11(1), 46.CrossRefGoogle Scholar
  18. Bozeman, B., & Sarewitz, D. (2011). Public value mapping and science policy evaluation. Minerva, 49(1), 1–23.CrossRefGoogle Scholar
  19. Brewer, J. D. (2011). The impact of impact. Research Evaluation, 20(3), 255–256.CrossRefGoogle Scholar
  20. Bridges, J. F., & Buttorff, C. (2010). What outcomes should US policy makers compare in comparative effectiveness research? Expert Review of Pharmacoeconomics & Outcomes Research, 10(3), 217–220.CrossRefGoogle Scholar
  21. Brody, H., Croisant, S. A., Crowder, J. W., & Banda, J. P. (2015). Ethical issues in patient-centered outcomes research and comparative effectiveness research: A Pilot study of community dialogue. Journal of Empirical Research on Human Research Ethics, 10(1), 22–30.CrossRefGoogle Scholar
  22. Burdge, R. J., & Vanclay, F. (1995). Social impact assessment. In F. Vanclay & D. A. Bronstein (Eds.), Environmental and social impact assessment (pp. 31–65). Chichester, UK: Wiley.Google Scholar
  23. Buxton, M., Hanney, S., Packwood, T., Roberts, S., & Youll, P. (2000). Getting reearch into practice: Assessing benefits from department of health and national health service research & development. Public Money and Management, 20(4), 29–34.CrossRefGoogle Scholar
  24. Castelnuovo, G., Limonta, D., Sarmiento, L., & Molinari, E. (2010). A more comprehensive index in the evaluation of scientific research: the single researcher impact factor proposal. Clinical practice and epidemiology in mental health: CP & EMH, 6, 109.CrossRefGoogle Scholar
  25. Cohen, G., Schroeder, J., Newson, R., King, L., Rychetnik, L., Milat, A. J., et al. (2015). Does health intervention research have real world policy and practice impact s: testing a new impact assessment tool. Health research policy and systems, 13(1), 3.CrossRefGoogle Scholar
  26. Colugnati, F. A., Firpo, S., de Castro, P. F. D., Sepulveda, J. E., & Salles-Filho, S. L. (2014). A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: The BIOTA Program. Scientometrics, 101(1), 85–107.CrossRefGoogle Scholar
  27. Cooksey, D. A. (2006). Review of UK health research funding. Norwich: HM Treasury.Google Scholar
  28. Council of Canadian Academies (2012). Expert Panel on Science Performance and Research Funding.Google Scholar
  29. Cousins, J. B., Svensson, K., Szijarto, B., Pinsent, C., Andrew, C., & Sylvestre, J. (2015). Assessing the practice impact of research on evaluation. New Directions for Evaluation, 2015(148), 73–88.CrossRefGoogle Scholar
  30. Cyril, F. M. D., & Phil, M. (2009). Health research: Measuring the social, health and economic benefits. Canadian Medical Association Journal, 180(5), 528–534.CrossRefGoogle Scholar
  31. Czarnitzki, D., & Lopes-Bento, C. (2013). Value for money? New Microeconometric Evidence on Public R&D Grants in Flanders. Research Policy, 42, 76–89.CrossRefGoogle Scholar
  32. Dannenberg, A. L., Bhatia, R., Cole, B. L., Dora, C., Fielding, J. E., Kraft, K., et al. (2006). Growing the field of health impact assessment in the United States: an agenda for research and practice. American Journal of Public Health, 96(2), 262–270.CrossRefGoogle Scholar
  33. Darmoni, S. J., Roussel, F., Benichou, J., Thirion, B., & Pinhas, N. (2002). Reading factor: A new bibliometric criterion for managing digital libraries. Journal-Medical Library Association, 90, 323–326.Google Scholar
  34. Davies, P. (2004). Is evidence-based government possible? Jerry Lee lecture to Campbell Collaboration Colloquium, Washington DC 19 February.Google Scholar
  35. Davies, P., Walker, A. E., & Grimshaw, J. M. (2010). A systematic review of the use of theory in the design of guideline dissemination and implementation strategies and interpretation of the results of rigorous evaluations. Implementation Science, 5(1), 5–14.CrossRefGoogle Scholar
  36. De Jong, S. P., Van Arensbergen, P., Daemen, F., Van Der Meulen, B., & Van Den Besselaar, P. (2011). Evaluation of research in context: An approach and two cases. Research Evaluation, 20(1), 61–72.CrossRefGoogle Scholar
  37. Denholm, E. M., & Martin, W. J. (2008). Translational research in environmental health sciences. Translational research: The journal of laboratory and clinical medicine, 151(2), 57.CrossRefGoogle Scholar
  38. Department of Education, Science and Training. (2005). Research quality framework: Assessing the quality and impact of research in Australia (Issue paper). Canberra: Commonwealth of Australia.Google Scholar
  39. Donovan, C. (2007). The qualitative future of research evaluation. Science and Public Policy, 34(8), 585–597.CrossRefGoogle Scholar
  40. Donovan, C. (2008). The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research. New Directions for Evaluation, 2008(118), 47–60.CrossRefGoogle Scholar
  41. Donovan, C. (2011). State of the art in assessing research impact: introduction to a special issue. Research Evaluation, 20(3), 175–179.CrossRefGoogle Scholar
  42. Drew, C. H., Pettibone, K. G., Finch Iii, F. O., Giles, D., & Jordan, P. (2016). Automated Research Impact Assessment: A new bibliometrics approach. Scientometrics, 106(3), 987–1005.CrossRefGoogle Scholar
  43. Ekboir, J. (2003). Why impact analysis should not be used for research evaluation and what the alternatives are. Agricultural Systems, 78(2), 166–184.CrossRefGoogle Scholar
  44. Eric. (2010). Evaluating the societal relevance of academic research: A guide. The Hague: Rathenau Institute.Google Scholar
  45. Ernø-Kjølhede, E., & Hansson, F. (2011). Measuring research performance during a changing relationship between science and society. Research Evaluation, 20(2), 130–142.CrossRefGoogle Scholar
  46. European Commission. (2010). Assessing Europe’s university-based research. Expert group on assessment of university-based research. Brussels, Belgium: Publications Office of the European Union. Google Scholar
  47. European Commission. (2011). Assessing Europe’s university-based research. Expert group on assessment of university-based research. Brussels: Publications Office of the European Union.Google Scholar
  48. Evans, A., Strezov, V., & Evans, T. J. (2009). Assessment of sustainability indicators for renewable energy technologies. Renewable and Sustainable Energy Reviews, 13(5), 1082–1088.CrossRefGoogle Scholar
  49. Figueredo, A. J., & Sechrest, L. (2001). Approaches used in conducting health outcomes and effectiveness research. Evaluation and Program Planning, 24(1), 41–59.CrossRefGoogle Scholar
  50. Fink, A. (1998). Conducting research literature review: from paper to internet. Thousand Oaks: SagePublications.Google Scholar
  51. Fox, R. J., Thompson, A., Baker, D., Baneke, P., Brown, D., Browne, P., et al. (2012). Setting a research agenda for progressive multiple sclerosis: The International Collaborative on Progressive MS. Multiple Sclerosis Journal, 18(11), 1534–1540.CrossRefGoogle Scholar
  52. Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2015). Research quality evaluation: Comparing citation counts considering bibliometric database errors. Quality & Quantity, 49(1), 155–165.CrossRefGoogle Scholar
  53. Freeman, R. E. (1984). Strategic management, a stakeholder approach. Boston: Pitman.Google Scholar
  54. Frey, B. S., & Rost, K. (2010). Do rankings reflect research quality? Journal of Applied Economics, 13(1), 1–38.CrossRefGoogle Scholar
  55. Furman, E., Kivimaa, P., Kuuppo, P., Nykänen, M., Väänänen, P., Mela, H., & Korpinen, P. (2006). Experiences in the management of research funding programmes for environmental protection. Including recommendations for best practice. Finnish Environment Institute.Google Scholar
  56. Fusco, D., Barone, A. P., Sorge, C., D’Ovidio, M., Stafoggia, M., Lallo, A., et al. (2012). P. Re. Val. E.: Outcome research program for the evaluation of health care quality in Lazio, Italy. BMC Health Services Research, 12(1), 25.CrossRefGoogle Scholar
  57. Gershon, R., Rothrock, N. E., Hanrahan, R. T., Jansky, L. J., Harniss, M., & Riley, W. (2010). The development of a clinical outcomes survey research application: Assessment CenterSM. Quality of Life Research, 19(5), 677–685.CrossRefGoogle Scholar
  58. Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. Thousand Oaks: Sage.Google Scholar
  59. Gibson, T. B., Ehrlich, E. D., Graff, J., Dubois, R., Farr, A. M., Chernew, M., et al. (2014). Real-world impact of comparative effectiveness research findings on clinical practice. The American journal of managed care, 20(6), e208–e220.Google Scholar
  60. Godin, B., & Dore, C. (2005). Measuring the impact s of science; beyond the economic dimension, INRS Urbanisation, Culture et Société. HIST Lecture, Helsinki Institute for Science and Technology Studies, Helsinki, Finland. Available at: http://www.csiic.ca/PDF/Godin_Dore_Impacts.pdf.
  61. Göransson, B., Maharajh, R., & Schmoch, U. (2009). New activities of universities in transfer and extension: Multiple requirements and manifold solutions. Science and Public Policy, 36(2), 157–164.CrossRefGoogle Scholar
  62. Grimshaw, J. M., Eccles, M. P., Lavis, J. N., Hill, S. J., & Squires, J. E. (2012). Knowledge translation of research findings. Implementation science, 7(1), 50.CrossRefGoogle Scholar
  63. Guinea, J., Sela, E., Gómez-Núñez, A. J., Mangwende, T., Ambali, A., Ngum, N., et al. (2015). Impact oriented monitoring: A new methodology for monitoring and evaluation of international public health research projects. Research Evaluation, 24(2), 131–145.CrossRefGoogle Scholar
  64. Guthrie, S., Wamae, W., Diepeveen, S., Wooding, S., & Grant, J. (2013). Measuring research: A guide to research evaluation frameworks and tools. Santa Monica: RAND.Google Scholar
  65. Haigh, F., Harris, P., & Haigh, N. (2012). Health impact assessment research and practice: A place for paradigm positioning? Environmental Impact Assessment Review, 33(1), 66–72.CrossRefGoogle Scholar
  66. Hall, J., & Wagner, M. (2012). Editorial: The challenges and opportunities of sustainable development for entrepreneurship and small business. Journal of Small Business & Entrepreneurship, 25(4), 409–416.CrossRefGoogle Scholar
  67. Hanney, S. R., Gonzalez-Block, M. A., Buxton, M. J., & Kogan, M. (2003). The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health research policy and systems, 1(1), 2.CrossRefGoogle Scholar
  68. Hanney, S., Packwood, T., & Buxton, M. (2000). Evaluating the benefits from health research and development centres: a categorization, a model and examples of application. Evaluation, 6(2), 137–160.CrossRefGoogle Scholar
  69. Harzing, A. W. (2010). The publish or perish book. Melbourne: Tarma Software Research Pty Ltd.Google Scholar
  70. Healthcare Industries Task Force, (2004). Better health through partnership: a programme for action, Final report. London, England: Author.Google Scholar
  71. Helming, K., Diehl, K., Kuhlman, T., Jansson, T., Verburg, P., Bakker, M., Morris, J. (2011). Ex ante impact assessment of policies affecting land use, part B: application of the analytical framework. Ecology and Society, 16(1), 1–29.Google Scholar
  72. Henshall, C. (2011). The impact of payback research: Developing and using evidence in policy. Research Evaluation, 20(3), 257–258.CrossRefGoogle Scholar
  73. Hessels, L. K., & Van Lente, H. (2010). The mixed blessing of Mode 2 knowledge production. Science Technology and Innovation Studies, 6(1), 65–69.Google Scholar
  74. Holbrook, J. B. (2012). Re-assessing the science-society relation: The case of the US National Science Foundation’s broader impact s merit review criterion (1997–2011). Techonology in Society, 27(4), 437–451.MathSciNetCrossRefGoogle Scholar
  75. Holbrook, J. B., & Frodeman, R. (2010, April). Comparative Assessment of Peer Review (CAPR). In EU/US workshop on peer review: Assessing ‘‘broader impact’’in research grant applications. Brussels: European Commission, Directorate-General for Research and Innovation.Google Scholar
  76. Holbrook, J. B., & Frodeman, R. (2011). Peer review and the ex ante assessment of societal impacts. Research Evaluation, 20(3), 239–246.CrossRefGoogle Scholar
  77. Horton, K., Tschudin, V., & Forget, A. (2007). The value of nursing: A literature review. Nursing Ethics, 14(6), 716–740.CrossRefGoogle Scholar
  78. Inglesi-Lotz, R., & Pouris, A. (2011). Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa. Scientometrics, 88(3), 747–760.CrossRefGoogle Scholar
  79. Ippoliti, R., & Falavigna, G. (2014). Public health institutions, clinical research and protection system of patients’ rights: An impact evaluation of public policy. Public Organization Review, 14(2), 109–125.CrossRefGoogle Scholar
  80. Jammer, I., Wickboldt, N., Sander, M., Smith, A., Schultz, M. J., Pelosi, P., et al. (2015). Standards for definitions and use of outcome measures for clinical effectiveness research in perioperative medicine: European Perioperative Clinical Outcome (EPCO) definitions: a statement from the ESA-ESICM joint taskforce on perioperative outcome measures. European Journal of Anaesthesiology (EJA), 32(2), 88–105.CrossRefGoogle Scholar
  81. Jette, A. M., & Keysor, J. J. (2002). 3. Uses of evidence in disability outcomes and effectiveness research. Milbank Quarterly, 80(2), 325–345.CrossRefGoogle Scholar
  82. Kovacs, S. M., Turner-Bowker, D. M., Calarco, G., Mulberg, A. E., & Paty, J. (2016). Practical considerations for the use of clinical outcome assessments (COAs) in pediatric clinical research: examples from pediatric gastroenterology. Therapeutic Innovation & Regulatory Science, 50(1), 37–43.CrossRefGoogle Scholar
  83. Kryl, D., Allen, L., Dolby, K., Sherbon, B., & Viney, I. (2012). Tracking the impact of research on policy and practice: Investigating the feasibility of using citations in clinical guidelines for research evaluation. British Medical Journal Open, 2(2), e000897.Google Scholar
  84. Lähteenmäki-Smith, K., Hyytinen, K., Kutinlahti, P., & Konttinen, J. (2006). Research with an impact. Evaluation practises in public research organisations. VTT Research Notes2336 Google Scholar
  85. LaKind, J. S., Goodman, M., Barr, D. B., Weisel, C. P., & Schoeters, G. (2015). Lessons learned from the application of BEES-C: Systematic assessment of study quality of epidemiologic research on BPA, neurodevelopment, and respiratory health. Environment International, 80, 41–71.CrossRefGoogle Scholar
  86. Lamm, G. M. (2006). Innovation works. A case study of an integrated pan-European technology transfer model. BIF Futura, 21(2), 86–90.MathSciNetGoogle Scholar
  87. Lane, P. J., Koka, B. R., & Pathak, S. (2006). The reification of absorptive capacity: A critical review and rejuvenation of the construct. Academy of Management Review, 31(4), 833–863.CrossRefGoogle Scholar
  88. Leduc, P. (1994). Evaluation in the social sciences: The strategic context. Research Evaluation, 4(1), 2–5.CrossRefGoogle Scholar
  89. Lee, F. S. (2007). The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities. Cambridge Journal of Economics, 31(2), 309–325.CrossRefGoogle Scholar
  90. Liebow, E., Phelps, J., Van Houten, B., Rose, S., Orians, C., Cohen, J., et al. (2009). Toward the assessment of scientific and public health impact s of the National Institute of Environmental Health Sciences Extramural Asthma Research Program using available data. Environmental Health Perspectives, 117(7), 1147.CrossRefGoogle Scholar
  91. Luukkonen, T. (1998). The difficulties in assessing the impact of EU framework programmes. Research Policy, 27(6), 599–610.CrossRefGoogle Scholar
  92. Manion, F. J., Harris, M. R., Buyuktur, A. G., Clark, P. M., An, L. C., & Hanauer, D. A. (2012). Leveraging EHR data for outcomes and comparative effectiveness research in oncology. Current oncology reports, 14(6), 494–501.CrossRefGoogle Scholar
  93. Maredia, M. K., & Byerlee, D. (2000). Efficiency of research investments in the presence of international spillovers: Wheat research in developing countries. Agricultural Economics, 22(1), 1–16.CrossRefGoogle Scholar
  94. Martin, B. R. (2007). Assessing the impact of basic research on society and the economy. In Paper presented at the rethinking the impact of basic research on society and the economy (WF-EST international conference, 11 May 2007), Vienna, Austria.Google Scholar
  95. Martin, B. R. (2011). The research excellence framework and the’impact agenda’: Are we creating a Frankenstein monster? Research Evaluation, 20(3), 247–254.CrossRefGoogle Scholar
  96. Martin and Irvine. (1983). Assessing basic research: The case of the Isaac Newton telescope. Social Studies of Science, 13, 49–86.CrossRefGoogle Scholar
  97. Mayring, P. (2003). Qualitative inhaltsanalyse [Qualitative content analysis]. Qualitative Forschung, 3, 468–475.Google Scholar
  98. Milat, A. J., Bauman, A. E., & Redman, S. (2015). A narrative review of research impact assessment models and methods. Health Research Policy and Systems, 13(1), 18.CrossRefGoogle Scholar
  99. Milat, A. J., Laws, R., King, L., Newson, R., Rychetnik, L., Rissel, C., et al. (2013). Policy and practice impact s of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006. Health Research Policy and Systems, 11(1), 5.CrossRefGoogle Scholar
  100. Moed, H. F. (2007). The effect of “open access” on citation impact: An analysis of ArXiv’s condensed matter section. Journal of the American Society for Information Science and Technology, 58(13), 2047–2054.CrossRefGoogle Scholar
  101. Moed, H. F., & Halevi, G. (2015). Multidimensional assessment of scholarly research impact. Journal of the Association for Information Science and Technology, 66(10), 1988–2002.CrossRefGoogle Scholar
  102. Molas-Gallart, J., & Tang, P. (2011). Tracing “productive interactions” to identify social impacts: An example for the social sciences. Research Evaluation, 20(3), 219–226.CrossRefGoogle Scholar
  103. Molas-Gallart, J., Salter, A., Patel, P., Scott, A., & Duran, X. (2002). Measuring third stream activities. Final report to the Russell Group of Universities. Brighton: SPRU, University of Sussex Google Scholar
  104. Moore, S. B., & Manring, S. L. (2009). Strategy development in small and medium sized enterprises for sustainability and increased value creation. Journal of Cleaner Production, 17(2), 276–282.CrossRefGoogle Scholar
  105. Morgan, M. M., & Grant, J. (2013). Making the grade: Methodologies for assessing and evidencing research impacts. Dean, A., Wykes, M. and Stevens, H.(eds), 7, 25–43Google Scholar
  106. Morton, S. (2015). Progressing research impact assessment: A ‘contributions’ approach. Research Evaluation, rvv016.Google Scholar
  107. Mostert, S. P., Ellenbroek, S. P., Meijer, I., Van Ark, G., & Klasen, E. C. (2010). Societal output and use of research performed by health research groups. Health Research Policy and Systems, 8(1), 30.CrossRefGoogle Scholar
  108. Mullins, C. D., Onukwugha, E., Cooke, J. L., Hussain, A., Baquet, C. R. (2010). The potential impact of comparative effectiveness research on the health of minority populations. Health Affairs, 29(11), 10–1377CrossRefGoogle Scholar
  109. Nallamothu, B. K., & Lüscher, T. F. (2012). Moving from impact to influence: Measurement and the changing role of medical journals. European Heart Journal, 33(23), 2892–2896.CrossRefGoogle Scholar
  110. New Philanthropy Capital. (2010). Social return on investment: Position paper. London: New Philanthropy Capital.Google Scholar
  111. Newby, H. (1994). The challenge for social science: A new role in public policy-making. Research Evaluation, 4(1), 6–11.CrossRefGoogle Scholar
  112. Niederkrotenthaler, T., Dorner, T. E., & Maier, M. (2011). Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists. BMC Public Health, 11(1), 588.CrossRefGoogle Scholar
  113. Nightingale, P., & Scott, A. (2007). Peer review and the relevance gap: ten suggestions for policy-makers. Science & Public Policy (SPP), 34(8), 543–553.CrossRefGoogle Scholar
  114. Nutley, S. M., Walter, I., & Davies, H. T. (2007). Using evidence: How research can inform public services. Bristol: Policy Press.CrossRefGoogle Scholar
  115. O’Connor, D. P., & Brinker, M. R. (2013). Challenges in outcome measurement: Clinical research perspective. Clinical Orthopaedics and Related Research®, 471(11), 3496–3503.CrossRefGoogle Scholar
  116. OECD. (2008). OECD science, technology and industry outlook. Paris: OECD.Google Scholar
  117. OECD Report. (2016). OECD science, technology and industry outlook. Paris: OECD.Google Scholar
  118. Penfield, T., Baker, M. J., Scoble, R., & Wykes, M. C. (2014). Assessment, evaluations, and definitions of research impact: A review. Research Evaluation, 23(1), 21–32.CrossRefGoogle Scholar
  119. Perrin, E. B. (2002). Some thoughts on outcomes research quality improvement, and performance measurement. Medical Care, 40(6), 89–91.Google Scholar
  120. Pontille, D., & Torny, D. (2010). The controversial policies of journal ratings: Evaluating social sciences and humanities. Research evaluation, 19(5), 347–360.CrossRefGoogle Scholar
  121. Potì, B., & Cerulli, G. (2011). Evaluation of firm R&D and innovation support: New indicators and the ex-ante prediction of ex-post additionality-potential. Research Evaluation, 20(1), 19–29.CrossRefGoogle Scholar
  122. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76.CrossRefGoogle Scholar
  123. Punt, A., Schiffelers, M. J. W., Horbach, G. J., van de Sandt, J. J., Groothuis, G. M., Rietjens, I. M., et al. (2011). Evaluation of research activities and research needs to increase the impact and applicability of alternative testing strategies in risk assessment practice. Regulatory Toxicology and Pharmacology, 61(1), 105–114.Google Scholar
  124. Reeve, B. B., Burke, L. B., Chiang, Y. P., Clauser, S. B., Colpe, L. J., Elias, J. W., et al. (2007). Enhancing measurement in health outcomes research supported by Agencies within the US Department of Health and Human Services. Quality of Life Research, 16(1), 175–186.CrossRefGoogle Scholar
  125. Reichertz, J. (2010). Abduction: the logic of discovery of grounded theory. Forum Qualitative Social Research, 11, 1–12.Google Scholar
  126. Roessner, D. (2000). Quantitative and qualitative methods and measures in the evaluation of research. Research Evaluation, 9(2), 125–132.CrossRefGoogle Scholar
  127. Rymer, L. (2011). Measuring the impact of research—The context for metric development. Turner, Australia: The Group of Eight.Google Scholar
  128. Sarli, C. C., Dubinsky, E. K., & Holmes, K. L. (2010). Beyond citation analysis: A model for assessment of research impact. Journal of the American Medical Library Association, 98(1), 17.CrossRefGoogle Scholar
  129. Schaltegger, S. (2002). A framework for ecopreneurship e leading bioneers and environmental managers to ecopreneurship. Greener Management International Journal, 38, 45–58.CrossRefGoogle Scholar
  130. Schulz, K. F., Chalmers, I., Hayes, R. J., & Altman, D. G. (1995). Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA, 273(5), 408–412.CrossRefGoogle Scholar
  131. Seuring, S., & Müller, M. (2008). From a literature review to a conceptual framework for sustainable supply chain management. Journal of Cleaner Production, 16(15), 1699–1710.CrossRefGoogle Scholar
  132. Seuring, P. D. S., Müller, P. D. M., Westhaus, M., Morana, R. (2005). Conducting a literature review—the example of sustainability in supply chains. In H. Kotzab, S. Seuring, M. Muller & G. Reiner (Eds.), Research methodologies in supply chain management (pp. 91–106). Physica-Verlag HD.Google Scholar
  133. Social Sciences and Humanities Scientific Committees. (2013). Humanities and social sciences in horizon 2020 societal challenges: Implementation and monitoring.Google Scholar
  134. Sombatsompop, N., Markpin, T., Yochai, W., & Saechiew, M. (2005). An evaluation of research performance for different subject categories using Impact Factor Point Average (IFPA) index: Thailand case study. Scientometrics, 65(3), 293–305.CrossRefGoogle Scholar
  135. Spaapen, J., & van Drooge, L. (2011). Introducing “productive interactions” in social impact assessment. Research Evaluation, 30(3), 211–218.CrossRefGoogle Scholar
  136. Spaapen, J., Dijstelbloem, H., & Wamelink, F. (2007). Evaluating research in context. A method for comprehensive assessment (2nd ed.). The Hague: COS.Google Scholar
  137. Stein, T. V., Anderson, D. H., & Kelly, T. (1999). Using stakeholders’ values to apply ecosystem management in an upper Midwest landscape. Environmental Management, 24(3), 399–413.CrossRefGoogle Scholar
  138. Stryer, D., Tunis, S., Hubbard, H., & Clancy, C. (2000). The outcomes of outcomes and effectiveness research: Impact s and lessons from the first decade. Health Services Research, 35(5 Pt 1), 977.Google Scholar
  139. Taylor, J., & Bradbury-Jones, C. (2011). International principles of social impact assessment: Lessons for research? Journal of Research in Nursing, 16(2), 133–145.CrossRefGoogle Scholar
  140. Technopolis. (2009). Impact Europese Kaderprogramma’s in Nederland. Woluwe-Saint-Pierre: Technopolis Group.Google Scholar
  141. Thorpe, R., Holt, R., Macpherson, A., & Pittaway, L. (2005). Using knowledge within small and medium-sized firms: A systematic review of the evidence. International Journal of Management Reviews, 7(4), 257–281.CrossRefGoogle Scholar
  142. Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207–222.CrossRefGoogle Scholar
  143. Tremblay, G., Zohar, S., Bravo, J., Potsepp, P., & Barker, M. (2010). The Canada Foundation for Innovation’s outcome measurement study: A pioneering approach to research evaluation. Research Evaluation, 19(5), 333–345.CrossRefGoogle Scholar
  144. United States Government Accountability Office. (2012). Designing evaluations. Washington, DC: Author.Google Scholar
  145. Van den Besselaar, P., & Leydesdorff, L. (2009). Past performance, peer review and project selection: A case study in the social and behavioral sciences. Research Evaluation, 18(4), 273–288.CrossRefGoogle Scholar
  146. Van der Meulen, B., & Rip, A. (2000). Evaluation of societal quality of public sector research in the Netherlands. Research Evaluation, 9(1), 11–25.CrossRefGoogle Scholar
  147. Van Raan, A. F. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.CrossRefGoogle Scholar
  148. Van Vught, F., & Ziegele, F. (2011). Design and testing the feasibility of a multidimensional global university ranking. Final Report. European Community, Europe: Consortium for Higher Education and Research Performance Assessment, CHERPA Network.Google Scholar
  149. Vanclay, F. (2003). International principles for social impact assessment. Impact assessment and project appraisal, 21(1), 5–12.CrossRefGoogle Scholar
  150. Westrich, K. D., Wilhelm, J. A., & Schur, C. L. (2016). Comparative effectiveness research in the U.S.A.: when will there be an impact on healthcare decision-making? Journal of Comparative Effectiveness Research, 5(2), 207–216.CrossRefGoogle Scholar
  151. Willis, T. A., Hartley, S., Glidewell, L., Farrin, A. J., Lawton, R., McEachan, R. R., et al. (2016). Action to Support Practices Implement Research Evidence (ASPIRE): Protocol for a cluster-randomised evaluation of adaptable implementation packages targeting ‘high impact’ clinical practice recommendations in general practice. Implementation Science, 11(1), 25.CrossRefGoogle Scholar
  152. Wooding, S., Hanney, S., Pollitt, A., Buxton, M., & Grantm, J. (2011). Project Retrosight. Understanding the Returns from Cardiovascular and Stroke Research: Policy Report. Cambridge: RAND Europe.Google Scholar
  153. Wu, Z. (2015). Average evaluation intensity: A quality-oriented indicator for the evaluation of research performance. Library & Information Science Research, 37(1), 51–60.CrossRefGoogle Scholar
  154. Pastakia, C. M. R. (1998). The rapid impact assessment matrix (RIAM)—A new tool for environmental impact assessment. In K. Jensen (Ed.), Environmental impact assessment using the rapid impact assessment matrix (RIAM). Fredensborg, Denmark: Olsen & Olsen.Google Scholar
  155. Yiend, J., Chambers, J. C., Burns, T., Doll, H., Fazel, S., Kaur, A., et al. (2011). Outcome measurement in forensic mental health research: An evaluation. Psychology, Crime & Law, 17(3), 277–292.CrossRefGoogle Scholar
  156. Zaratin, P., Battaglia, M. A., & Abbracchio, M. P. (2014). Nonprofit foundations spur translational research. Trends in Pharmacological Sciences, 35(11), 552–555.CrossRefGoogle Scholar
  157. Zaratin, P., Comi, G., Coetzee, T., Ramsey, K., Smith, K., Thompson, A., et al. (2016). Progressive MS Alliance Industry Forum: maximizing collective impact to enable drug development. Trends in Pharmacological Sciences, 37(10), 808–810.CrossRefGoogle Scholar
  158. Zelefsky, M. J., Lee, W. R., Zietman, A., Khalid, N., Crozier, C., Owen, J., et al. (2013). Evaluation of adherence to quality measures for prostate cancer radiotherapy in the United States: Results from the quality research in radiation oncology (QRRO) survey. Practical Radiation Oncology, 3(1), 2–8.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2017

Authors and Affiliations

  1. 1.ALTIS-Alta Scuola Impresa e SocietàUniversità Cattolica del Sacro CuoreMilanItaly
  2. 2.Dipartimento di Scienza della VitaUniversità di SienaSienaItaly
  3. 3.Fondazione Italiana Sclerosi MultiplaGenoaItaly

Personalised recommendations