Skip to main content
SpringerLink
Log in

Assessing the health research’s social impact: a systematic review

Scientometrics volume 114, pages 1227–1250 (2018)Cite this article

Abstract

In recent year, a growing attention is dedicated to the assessment of research’s social impact. While prior research has often dealt with results of research, the last decade has begun to generate knowledge on the assessment of health research’s social impact. However, this knowledge is scattered across different disciplines, research communities, and journals. Therefore, this paper analyzes the heterogeneous picture research has drawn within the past years with a focus on the health research’s social impact on different stakeholders through an interdisciplinary, systematic review. By consulting major research databases, we have analyzed 53 key journal articles bibliographically and thematically. We argued that the adoption of a multi-stakeholder could be an evolution of the existing methods used to assess impact of research. After presenting a model to assess the health research’s social impact with a multi stakeholder perspective, we suggest the implementation in the research process of three practice: a multi-stakeholder workshop on research agenda; a multi stakeholder supervisory board; a multi-stakeholder review process.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  • Adam, P., Solans-Domènech, M., Pons, J. M. V., Aymerich, M., Berra, S., Guillamon, I., et al. (2012). Assessment of the impact of a clinical and health services research call in Catalonia. Research Evaluation, 21(4), 319–328.

    Article  Google Scholar 

  • Ahmed, S., Berzon, R. A., Revicki, D. A., Lenderking, W. R., Moinpour, C. M., Basch, E., et al. (2012). The use of patient-reported outcomes (PRO) within comparative effectiveness research implications for clinical practice and health care policy. Medical Care, 50(12), 60–70.

    Article  Google Scholar 

  • Amara, N., Ouimet, M., & Landry, R. (2004). New evidence on the instrumental, conceptual, and symbolic utilization of university research in government agencies. Science Communication, 26(1), 75–106.

    Article  Google Scholar 

  • Anderson, A. R. (1998). Cultivating the Garden of Eden: environmental entrepreneuring. Journal of Organizational Change Management, 11(2), 135–144.

    Article  Google Scholar 

  • Anderson, M., & McCleary, K. K. (2016). On the path to a science of patient input. Science Translational Medicine, 8(336), 336.

    Article  Google Scholar 

  • Banzi, R., Moja, L., Pistotti, V., Facchini, A., & Liberati, A. (2011). Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Research Policy and Systems, 9, 26.

    Article  Google Scholar 

  • Barker, K. (2007). The UK Research Assessment Exercise: the evolution of a national research evaluation system. Research Evaluation, 16(1), 3–12.

    Article  Google Scholar 

  • Barré, R. (2005). S&T indicators for policy making in a changing science–society relationship. In H. Moed, W. Glänzel & U. Schmoch (Eds.), Handbook of quantitative science and technology research (pp. 115–131). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Bell, S., Shaw, B., & Boaz, A. (2011). Real-world approaches to assessing the impact of environmental research on policy. Research evaluation, 20(3), 227–237.

    Article  Google Scholar 

  • Bensing, J. M., Caris-Verhallen, W. M., Dekker, J., Delnoij, D. M., & Groenewegen, P. P. (2003). Doing the right thing and doing it right: toward a framework for assessing the policy relevance of health services research. International Journal of Technology Assessment in Health Care, 19(04), 604–612.

    Article  Google Scholar 

  • Bloch, C., Sørensen, M. P., Graversen, E. K., Schneider, J. W., Schmidt, E. K., Aagaard, K., et al. (2014). Developing a methodology to assess the impact of research grant funding: A mixed methods approach. Evaluation and program planning, 43, 105–117.

    Article  Google Scholar 

  • Boaz, A., Fitzpatrick, S., & Shaw, B. (2009). Assessing the impact of research on policy: a literature review. Science & Public Policy (SPP), 36(4), 255–270.

    Article  Google Scholar 

  • Bollen, J., Van de Sompel, H., & Rodriguez, M. A. (2008, June). Towards usage-based impact metrics: first results from the mesur project. In Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries (pp. 231–240). ACM.

  • Bornmann, L. (2013a). Measuring the societal impact of research: research is less and less assessed on scientific impact alone—We should aim to quantify the increasingly important contributions of science to society. EMBO Reports, 13(8), 673–676.

    Article  Google Scholar 

  • Bornmann, L. (2013b). What is societal impact of research and how can it be assessed? a literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233.

    Article  Google Scholar 

  • Bornmann, L., & Marx, W. (2014). How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons. Scientometrics, 98(1), 211–219.

    Article  Google Scholar 

  • Boyd, A., Cole, D. C., Cho, D. B., Aslanyan, G., & Bates, I. (2013). Frameworks for evaluating health research capacity strengthening: a qualitative study. Health Research Policy and Systems, 11(1), 46.

    Article  Google Scholar 

  • Bozeman, B., & Sarewitz, D. (2011). Public value mapping and science policy evaluation. Minerva, 49(1), 1–23.

    Article  Google Scholar 

  • Brewer, J. D. (2011). The impact of impact. Research Evaluation, 20(3), 255–256.

    Article  Google Scholar 

  • Bridges, J. F., & Buttorff, C. (2010). What outcomes should US policy makers compare in comparative effectiveness research? Expert Review of Pharmacoeconomics & Outcomes Research, 10(3), 217–220.

    Article  Google Scholar 

  • Brody, H., Croisant, S. A., Crowder, J. W., & Banda, J. P. (2015). Ethical issues in patient-centered outcomes research and comparative effectiveness research: A Pilot study of community dialogue. Journal of Empirical Research on Human Research Ethics, 10(1), 22–30.

    Article  Google Scholar 

  • Burdge, R. J., & Vanclay, F. (1995). Social impact assessment. In F. Vanclay & D. A. Bronstein (Eds.), Environmental and social impact assessment (pp. 31–65). Chichester, UK: Wiley.

    Google Scholar 

  • Buxton, M., Hanney, S., Packwood, T., Roberts, S., & Youll, P. (2000). Getting reearch into practice: Assessing benefits from department of health and national health service research & development. Public Money and Management, 20(4), 29–34.

    Article  Google Scholar 

  • Castelnuovo, G., Limonta, D., Sarmiento, L., & Molinari, E. (2010). A more comprehensive index in the evaluation of scientific research: the single researcher impact factor proposal. Clinical practice and epidemiology in mental health: CP & EMH, 6, 109.

    Article  Google Scholar 

  • Cohen, G., Schroeder, J., Newson, R., King, L., Rychetnik, L., Milat, A. J., et al. (2015). Does health intervention research have real world policy and practice impact s: testing a new impact assessment tool. Health research policy and systems, 13(1), 3.

    Article  Google Scholar 

  • Colugnati, F. A., Firpo, S., de Castro, P. F. D., Sepulveda, J. E., & Salles-Filho, S. L. (2014). A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: The BIOTA Program. Scientometrics, 101(1), 85–107.

    Article  Google Scholar 

  • Cooksey, D. A. (2006). Review of UK health research funding. Norwich: HM Treasury.

    Google Scholar 

  • Council of Canadian Academies (2012). Expert Panel on Science Performance and Research Funding.

  • Cousins, J. B., Svensson, K., Szijarto, B., Pinsent, C., Andrew, C., & Sylvestre, J. (2015). Assessing the practice impact of research on evaluation. New Directions for Evaluation, 2015(148), 73–88.

    Article  Google Scholar 

  • Cyril, F. M. D., & Phil, M. (2009). Health research: Measuring the social, health and economic benefits. Canadian Medical Association Journal, 180(5), 528–534.

    Article  Google Scholar 

  • Czarnitzki, D., & Lopes-Bento, C. (2013). Value for money? New Microeconometric Evidence on Public R&D Grants in Flanders. Research Policy, 42, 76–89.

    Article  Google Scholar 

  • Dannenberg, A. L., Bhatia, R., Cole, B. L., Dora, C., Fielding, J. E., Kraft, K., et al. (2006). Growing the field of health impact assessment in the United States: an agenda for research and practice. American Journal of Public Health, 96(2), 262–270.

    Article  Google Scholar 

  • Darmoni, S. J., Roussel, F., Benichou, J., Thirion, B., & Pinhas, N. (2002). Reading factor: A new bibliometric criterion for managing digital libraries. Journal-Medical Library Association, 90, 323–326.

    Google Scholar 

  • Davies, P. (2004). Is evidence-based government possible? Jerry Lee lecture to Campbell Collaboration Colloquium, Washington DC 19 February.

  • Davies, P., Walker, A. E., & Grimshaw, J. M. (2010). A systematic review of the use of theory in the design of guideline dissemination and implementation strategies and interpretation of the results of rigorous evaluations. Implementation Science, 5(1), 5–14.

    Article  Google Scholar 

  • De Jong, S. P., Van Arensbergen, P., Daemen, F., Van Der Meulen, B., & Van Den Besselaar, P. (2011). Evaluation of research in context: An approach and two cases. Research Evaluation, 20(1), 61–72.

    Article  Google Scholar 

  • Denholm, E. M., & Martin, W. J. (2008). Translational research in environmental health sciences. Translational research: The journal of laboratory and clinical medicine, 151(2), 57.

    Article  Google Scholar 

  • Department of Education, Science and Training. (2005). Research quality framework: Assessing the quality and impact of research in Australia (Issue paper). Canberra: Commonwealth of Australia.

    Google Scholar 

  • Donovan, C. (2007). The qualitative future of research evaluation. Science and Public Policy, 34(8), 585–597.

    Article  Google Scholar 

  • Donovan, C. (2008). The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research. New Directions for Evaluation, 2008(118), 47–60.

    Article  Google Scholar 

  • Donovan, C. (2011). State of the art in assessing research impact: introduction to a special issue. Research Evaluation, 20(3), 175–179.

    Article  Google Scholar 

  • Drew, C. H., Pettibone, K. G., Finch Iii, F. O., Giles, D., & Jordan, P. (2016). Automated Research Impact Assessment: A new bibliometrics approach. Scientometrics, 106(3), 987–1005.

    Article  Google Scholar 

  • Ekboir, J. (2003). Why impact analysis should not be used for research evaluation and what the alternatives are. Agricultural Systems, 78(2), 166–184.

    Article  Google Scholar 

  • Eric. (2010). Evaluating the societal relevance of academic research: A guide. The Hague: Rathenau Institute.

    Google Scholar 

  • Ernø-Kjølhede, E., & Hansson, F. (2011). Measuring research performance during a changing relationship between science and society. Research Evaluation, 20(2), 130–142.

    Article  Google Scholar 

  • European Commission. (2010). Assessing Europe’s university-based research. Expert group on assessment of university-based research. Brussels, Belgium: Publications Office of the European Union.

  • European Commission. (2011). Assessing Europe’s university-based research. Expert group on assessment of university-based research. Brussels: Publications Office of the European Union.

    Google Scholar 

  • Evans, A., Strezov, V., & Evans, T. J. (2009). Assessment of sustainability indicators for renewable energy technologies. Renewable and Sustainable Energy Reviews, 13(5), 1082–1088.

    Article  Google Scholar 

  • Figueredo, A. J., & Sechrest, L. (2001). Approaches used in conducting health outcomes and effectiveness research. Evaluation and Program Planning, 24(1), 41–59.

    Article  Google Scholar 

  • Fink, A. (1998). Conducting research literature review: from paper to internet. Thousand Oaks: SagePublications.

    Google Scholar 

  • Fox, R. J., Thompson, A., Baker, D., Baneke, P., Brown, D., Browne, P., et al. (2012). Setting a research agenda for progressive multiple sclerosis: The International Collaborative on Progressive MS. Multiple Sclerosis Journal, 18(11), 1534–1540.

    Article  Google Scholar 

  • Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2015). Research quality evaluation: Comparing citation counts considering bibliometric database errors. Quality & Quantity, 49(1), 155–165.

    Article  Google Scholar 

  • Freeman, R. E. (1984). Strategic management, a stakeholder approach. Boston: Pitman.

    Google Scholar 

  • Frey, B. S., & Rost, K. (2010). Do rankings reflect research quality? Journal of Applied Economics, 13(1), 1–38.

    Article  Google Scholar 

  • Furman, E., Kivimaa, P., Kuuppo, P., Nykänen, M., Väänänen, P., Mela, H., & Korpinen, P. (2006). Experiences in the management of research funding programmes for environmental protection. Including recommendations for best practice. Finnish Environment Institute.

  • Fusco, D., Barone, A. P., Sorge, C., D’Ovidio, M., Stafoggia, M., Lallo, A., et al. (2012). P. Re. Val. E.: Outcome research program for the evaluation of health care quality in Lazio, Italy. BMC Health Services Research, 12(1), 25.

    Article  Google Scholar 

  • Gershon, R., Rothrock, N. E., Hanrahan, R. T., Jansky, L. J., Harniss, M., & Riley, W. (2010). The development of a clinical outcomes survey research application: Assessment CenterSM. Quality of Life Research, 19(5), 677–685.

    Article  Google Scholar 

  • Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. Thousand Oaks: Sage.

    Google Scholar 

  • Gibson, T. B., Ehrlich, E. D., Graff, J., Dubois, R., Farr, A. M., Chernew, M., et al. (2014). Real-world impact of comparative effectiveness research findings on clinical practice. The American journal of managed care, 20(6), e208–e220.

    Google Scholar 

  • Godin, B., & Dore, C. (2005). Measuring the impact s of science; beyond the economic dimension, INRS Urbanisation, Culture et Société. HIST Lecture, Helsinki Institute for Science and Technology Studies, Helsinki, Finland. Available at: http://www.csiic.ca/PDF/Godin_Dore_Impacts.pdf.

  • Göransson, B., Maharajh, R., & Schmoch, U. (2009). New activities of universities in transfer and extension: Multiple requirements and manifold solutions. Science and Public Policy, 36(2), 157–164.

    Article  Google Scholar 

  • Grimshaw, J. M., Eccles, M. P., Lavis, J. N., Hill, S. J., & Squires, J. E. (2012). Knowledge translation of research findings. Implementation science, 7(1), 50.

    Article  Google Scholar 

  • Guinea, J., Sela, E., Gómez-Núñez, A. J., Mangwende, T., Ambali, A., Ngum, N., et al. (2015). Impact oriented monitoring: A new methodology for monitoring and evaluation of international public health research projects. Research Evaluation, 24(2), 131–145.

    Article  Google Scholar 

  • Guthrie, S., Wamae, W., Diepeveen, S., Wooding, S., & Grant, J. (2013). Measuring research: A guide to research evaluation frameworks and tools. Santa Monica: RAND.

    Google Scholar 

  • Haigh, F., Harris, P., & Haigh, N. (2012). Health impact assessment research and practice: A place for paradigm positioning? Environmental Impact Assessment Review, 33(1), 66–72.

    Article  Google Scholar 

  • Hall, J., & Wagner, M. (2012). Editorial: The challenges and opportunities of sustainable development for entrepreneurship and small business. Journal of Small Business & Entrepreneurship, 25(4), 409–416.

    Article  Google Scholar 

  • Hanney, S. R., Gonzalez-Block, M. A., Buxton, M. J., & Kogan, M. (2003). The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health research policy and systems, 1(1), 2.

    Article  Google Scholar 

  • Hanney, S., Packwood, T., & Buxton, M. (2000). Evaluating the benefits from health research and development centres: a categorization, a model and examples of application. Evaluation, 6(2), 137–160.

    Article  Google Scholar 

  • Harzing, A. W. (2010). The publish or perish book. Melbourne: Tarma Software Research Pty Ltd.

    Google Scholar 

  • Healthcare Industries Task Force, (2004). Better health through partnership: a programme for action, Final report. London, England: Author.

  • Helming, K., Diehl, K., Kuhlman, T., Jansson, T., Verburg, P., Bakker, M., Morris, J. (2011). Ex ante impact assessment of policies affecting land use, part B: application of the analytical framework. Ecology and Society, 16(1), 1–29.

    Google Scholar 

  • Henshall, C. (2011). The impact of payback research: Developing and using evidence in policy. Research Evaluation, 20(3), 257–258.

    Article  Google Scholar 

  • Hessels, L. K., & Van Lente, H. (2010). The mixed blessing of Mode 2 knowledge production. Science Technology and Innovation Studies, 6(1), 65–69.

    Google Scholar 

  • Holbrook, J. B. (2012). Re-assessing the science-society relation: The case of the US National Science Foundation’s broader impact s merit review criterion (1997–2011). Techonology in Society, 27(4), 437–451.

    Article  Google Scholar 

  • Holbrook, J. B., & Frodeman, R. (2010, April). Comparative Assessment of Peer Review (CAPR). In EU/US workshop on peer review: Assessing ‘‘broader impact’’in research grant applications. Brussels: European Commission, Directorate-General for Research and Innovation.

  • Holbrook, J. B., & Frodeman, R. (2011). Peer review and the ex ante assessment of societal impacts. Research Evaluation, 20(3), 239–246.

    Article  Google Scholar 

  • Horton, K., Tschudin, V., & Forget, A. (2007). The value of nursing: A literature review. Nursing Ethics, 14(6), 716–740.

    Article  Google Scholar 

  • Inglesi-Lotz, R., & Pouris, A. (2011). Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa. Scientometrics, 88(3), 747–760.

    Article  Google Scholar 

  • Ippoliti, R., & Falavigna, G. (2014). Public health institutions, clinical research and protection system of patients’ rights: An impact evaluation of public policy. Public Organization Review, 14(2), 109–125.

    Article  Google Scholar 

  • Jammer, I., Wickboldt, N., Sander, M., Smith, A., Schultz, M. J., Pelosi, P., et al. (2015). Standards for definitions and use of outcome measures for clinical effectiveness research in perioperative medicine: European Perioperative Clinical Outcome (EPCO) definitions: a statement from the ESA-ESICM joint taskforce on perioperative outcome measures. European Journal of Anaesthesiology (EJA), 32(2), 88–105.

    Article  Google Scholar 

  • Jette, A. M., & Keysor, J. J. (2002). 3. Uses of evidence in disability outcomes and effectiveness research. Milbank Quarterly, 80(2), 325–345.

    Article  Google Scholar 

  • Kovacs, S. M., Turner-Bowker, D. M., Calarco, G., Mulberg, A. E., & Paty, J. (2016). Practical considerations for the use of clinical outcome assessments (COAs) in pediatric clinical research: examples from pediatric gastroenterology. Therapeutic Innovation & Regulatory Science, 50(1), 37–43.

    Article  Google Scholar 

  • Kryl, D., Allen, L., Dolby, K., Sherbon, B., & Viney, I. (2012). Tracking the impact of research on policy and practice: Investigating the feasibility of using citations in clinical guidelines for research evaluation. British Medical Journal Open, 2(2), e000897.

    Google Scholar 

  • Lähteenmäki-Smith, K., Hyytinen, K., Kutinlahti, P., & Konttinen, J. (2006). Research with an impact. Evaluation practises in public research organisations. VTT Research Notes, 2336

  • LaKind, J. S., Goodman, M., Barr, D. B., Weisel, C. P., & Schoeters, G. (2015). Lessons learned from the application of BEES-C: Systematic assessment of study quality of epidemiologic research on BPA, neurodevelopment, and respiratory health. Environment International, 80, 41–71.

    Article  Google Scholar 

  • Lamm, G. M. (2006). Innovation works. A case study of an integrated pan-European technology transfer model. BIF Futura, 21(2), 86–90.

    MathSciNet  Google Scholar 

  • Lane, P. J., Koka, B. R., & Pathak, S. (2006). The reification of absorptive capacity: A critical review and rejuvenation of the construct. Academy of Management Review, 31(4), 833–863.

    Article  Google Scholar 

  • Leduc, P. (1994). Evaluation in the social sciences: The strategic context. Research Evaluation, 4(1), 2–5.

    Article  Google Scholar 

  • Lee, F. S. (2007). The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities. Cambridge Journal of Economics, 31(2), 309–325.

    Article  Google Scholar 

  • Liebow, E., Phelps, J., Van Houten, B., Rose, S., Orians, C., Cohen, J., et al. (2009). Toward the assessment of scientific and public health impact s of the National Institute of Environmental Health Sciences Extramural Asthma Research Program using available data. Environmental Health Perspectives, 117(7), 1147.

    Article  Google Scholar 

  • Luukkonen, T. (1998). The difficulties in assessing the impact of EU framework programmes. Research Policy, 27(6), 599–610.

    Article  Google Scholar 

  • Manion, F. J., Harris, M. R., Buyuktur, A. G., Clark, P. M., An, L. C., & Hanauer, D. A. (2012). Leveraging EHR data for outcomes and comparative effectiveness research in oncology. Current oncology reports, 14(6), 494–501.

    Article  Google Scholar 

  • Maredia, M. K., & Byerlee, D. (2000). Efficiency of research investments in the presence of international spillovers: Wheat research in developing countries. Agricultural Economics, 22(1), 1–16.

    Article  Google Scholar 

  • Martin, B. R. (2007). Assessing the impact of basic research on society and the economy. In Paper presented at the rethinking the impact of basic research on society and the economy (WF-EST international conference, 11 May 2007), Vienna, Austria.

  • Martin, B. R. (2011). The research excellence framework and the’impact agenda’: Are we creating a Frankenstein monster? Research Evaluation, 20(3), 247–254.

    Article  Google Scholar 

  • Martin and Irvine. (1983). Assessing basic research: The case of the Isaac Newton telescope. Social Studies of Science, 13, 49–86.

    Article  Google Scholar 

  • Mayring, P. (2003). Qualitative inhaltsanalyse [Qualitative content analysis]. Qualitative Forschung, 3, 468–475.

    Google Scholar 

  • Milat, A. J., Bauman, A. E., & Redman, S. (2015). A narrative review of research impact assessment models and methods. Health Research Policy and Systems, 13(1), 18.

    Article  Google Scholar 

  • Milat, A. J., Laws, R., King, L., Newson, R., Rychetnik, L., Rissel, C., et al. (2013). Policy and practice impact s of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006. Health Research Policy and Systems, 11(1), 5.

    Article  Google Scholar 

  • Moed, H. F. (2007). The effect of “open access” on citation impact: An analysis of ArXiv’s condensed matter section. Journal of the American Society for Information Science and Technology, 58(13), 2047–2054.

    Article  Google Scholar 

  • Moed, H. F., & Halevi, G. (2015). Multidimensional assessment of scholarly research impact. Journal of the Association for Information Science and Technology, 66(10), 1988–2002.

    Article  Google Scholar 

  • Molas-Gallart, J., & Tang, P. (2011). Tracing “productive interactions” to identify social impacts: An example for the social sciences. Research Evaluation, 20(3), 219–226.

    Article  Google Scholar 

  • Molas-Gallart, J., Salter, A., Patel, P., Scott, A., & Duran, X. (2002). Measuring third stream activities. Final report to the Russell Group of Universities. Brighton: SPRU, University of Sussex

  • Moore, S. B., & Manring, S. L. (2009). Strategy development in small and medium sized enterprises for sustainability and increased value creation. Journal of Cleaner Production, 17(2), 276–282.

    Article  Google Scholar 

  • Morgan, M. M., & Grant, J. (2013). Making the grade: Methodologies for assessing and evidencing research impacts. Dean, A., Wykes, M. and Stevens, H.(eds), 7, 25–43

  • Morton, S. (2015). Progressing research impact assessment: A ‘contributions’ approach. Research Evaluation, rvv016.

  • Mostert, S. P., Ellenbroek, S. P., Meijer, I., Van Ark, G., & Klasen, E. C. (2010). Societal output and use of research performed by health research groups. Health Research Policy and Systems, 8(1), 30.

    Article  Google Scholar 

  • Mullins, C. D., Onukwugha, E., Cooke, J. L., Hussain, A., Baquet, C. R. (2010). The potential impact of comparative effectiveness research on the health of minority populations. Health Affairs, 29(11), 10–1377

    Article  Google Scholar 

  • Nallamothu, B. K., & Lüscher, T. F. (2012). Moving from impact to influence: Measurement and the changing role of medical journals. European Heart Journal, 33(23), 2892–2896.

    Article  Google Scholar 

  • New Philanthropy Capital. (2010). Social return on investment: Position paper. London: New Philanthropy Capital.

    Google Scholar 

  • Newby, H. (1994). The challenge for social science: A new role in public policy-making. Research Evaluation, 4(1), 6–11.

    Article  Google Scholar 

  • Niederkrotenthaler, T., Dorner, T. E., & Maier, M. (2011). Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists. BMC Public Health, 11(1), 588.

    Article  Google Scholar 

  • Nightingale, P., & Scott, A. (2007). Peer review and the relevance gap: ten suggestions for policy-makers. Science & Public Policy (SPP), 34(8), 543–553.

    Article  Google Scholar 

  • Nutley, S. M., Walter, I., & Davies, H. T. (2007). Using evidence: How research can inform public services. Bristol: Policy Press.

    Book  Google Scholar 

  • O’Connor, D. P., & Brinker, M. R. (2013). Challenges in outcome measurement: Clinical research perspective. Clinical Orthopaedics and Related Research®, 471(11), 3496–3503.

    Article  Google Scholar 

  • OECD. (2008). OECD science, technology and industry outlook. Paris: OECD.

    Google Scholar 

  • OECD Report. (2016). OECD science, technology and industry outlook. Paris: OECD.

    Google Scholar 

  • Penfield, T., Baker, M. J., Scoble, R., & Wykes, M. C. (2014). Assessment, evaluations, and definitions of research impact: A review. Research Evaluation, 23(1), 21–32.

    Article  Google Scholar 

  • Perrin, E. B. (2002). Some thoughts on outcomes research quality improvement, and performance measurement. Medical Care, 40(6), 89–91.

    Google Scholar 

  • Pontille, D., & Torny, D. (2010). The controversial policies of journal ratings: Evaluating social sciences and humanities. Research evaluation, 19(5), 347–360.

    Article  Google Scholar 

  • Potì, B., & Cerulli, G. (2011). Evaluation of firm R&D and innovation support: New indicators and the ex-ante prediction of ex-post additionality-potential. Research Evaluation, 20(1), 19–29.

    Article  Google Scholar 

  • Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76.

    Article  Google Scholar 

  • Punt, A., Schiffelers, M. J. W., Horbach, G. J., van de Sandt, J. J., Groothuis, G. M., Rietjens, I. M., et al. (2011). Evaluation of research activities and research needs to increase the impact and applicability of alternative testing strategies in risk assessment practice. Regulatory Toxicology and Pharmacology, 61(1), 105–114.

    Google Scholar 

  • Reeve, B. B., Burke, L. B., Chiang, Y. P., Clauser, S. B., Colpe, L. J., Elias, J. W., et al. (2007). Enhancing measurement in health outcomes research supported by Agencies within the US Department of Health and Human Services. Quality of Life Research, 16(1), 175–186.

    Article  Google Scholar 

  • Reichertz, J. (2010). Abduction: the logic of discovery of grounded theory. Forum Qualitative Social Research, 11, 1–12.

    Google Scholar 

  • Roessner, D. (2000). Quantitative and qualitative methods and measures in the evaluation of research. Research Evaluation, 9(2), 125–132.

    Article  Google Scholar 

  • Rymer, L. (2011). Measuring the impact of research—The context for metric development. Turner, Australia: The Group of Eight.

    Google Scholar 

  • Sarli, C. C., Dubinsky, E. K., & Holmes, K. L. (2010). Beyond citation analysis: A model for assessment of research impact. Journal of the American Medical Library Association, 98(1), 17.

    Article  Google Scholar 

  • Schaltegger, S. (2002). A framework for ecopreneurship e leading bioneers and environmental managers to ecopreneurship. Greener Management International Journal, 38, 45–58.

    Article  Google Scholar 

  • Schulz, K. F., Chalmers, I., Hayes, R. J., & Altman, D. G. (1995). Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA, 273(5), 408–412.

    Article  Google Scholar 

  • Seuring, S., & Müller, M. (2008). From a literature review to a conceptual framework for sustainable supply chain management. Journal of Cleaner Production, 16(15), 1699–1710.

    Article  Google Scholar 

  • Seuring, P. D. S., Müller, P. D. M., Westhaus, M., Morana, R. (2005). Conducting a literature review—the example of sustainability in supply chains. In H. Kotzab, S. Seuring, M. Muller & G. Reiner (Eds.), Research methodologies in supply chain management (pp. 91–106). Physica-Verlag HD.

  • Social Sciences and Humanities Scientific Committees. (2013). Humanities and social sciences in horizon 2020 societal challenges: Implementation and monitoring.

  • Sombatsompop, N., Markpin, T., Yochai, W., & Saechiew, M. (2005). An evaluation of research performance for different subject categories using Impact Factor Point Average (IFPA) index: Thailand case study. Scientometrics, 65(3), 293–305.

    Article  Google Scholar 

  • Spaapen, J., & van Drooge, L. (2011). Introducing “productive interactions” in social impact assessment. Research Evaluation, 30(3), 211–218.

    Article  Google Scholar 

  • Spaapen, J., Dijstelbloem, H., & Wamelink, F. (2007). Evaluating research in context. A method for comprehensive assessment (2nd ed.). The Hague: COS.

    Google Scholar 

  • Stein, T. V., Anderson, D. H., & Kelly, T. (1999). Using stakeholders’ values to apply ecosystem management in an upper Midwest landscape. Environmental Management, 24(3), 399–413.

    Article  Google Scholar 

  • Stryer, D., Tunis, S., Hubbard, H., & Clancy, C. (2000). The outcomes of outcomes and effectiveness research: Impact s and lessons from the first decade. Health Services Research, 35(5 Pt 1), 977.

    Google Scholar 

  • Taylor, J., & Bradbury-Jones, C. (2011). International principles of social impact assessment: Lessons for research? Journal of Research in Nursing, 16(2), 133–145.

    Article  Google Scholar 

  • Technopolis. (2009). Impact Europese Kaderprogramma’s in Nederland. Woluwe-Saint-Pierre: Technopolis Group.

    Google Scholar 

  • Thorpe, R., Holt, R., Macpherson, A., & Pittaway, L. (2005). Using knowledge within small and medium-sized firms: A systematic review of the evidence. International Journal of Management Reviews, 7(4), 257–281.

    Article  Google Scholar 

  • Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207–222.

    Article  Google Scholar 

  • Tremblay, G., Zohar, S., Bravo, J., Potsepp, P., & Barker, M. (2010). The Canada Foundation for Innovation’s outcome measurement study: A pioneering approach to research evaluation. Research Evaluation, 19(5), 333–345.

    Article  Google Scholar 

  • United States Government Accountability Office. (2012). Designing evaluations. Washington, DC: Author.

    Google Scholar 

  • Van den Besselaar, P., & Leydesdorff, L. (2009). Past performance, peer review and project selection: A case study in the social and behavioral sciences. Research Evaluation, 18(4), 273–288.

    Article  Google Scholar 

  • Van der Meulen, B., & Rip, A. (2000). Evaluation of societal quality of public sector research in the Netherlands. Research Evaluation, 9(1), 11–25.

    Article  Google Scholar 

  • Van Raan, A. F. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.

    Article  Google Scholar 

  • Van Vught, F., & Ziegele, F. (2011). Design and testing the feasibility of a multidimensional global university ranking. Final Report. European Community, Europe: Consortium for Higher Education and Research Performance Assessment, CHERPA Network.

  • Vanclay, F. (2003). International principles for social impact assessment. Impact assessment and project appraisal, 21(1), 5–12.

    Article  Google Scholar 

  • Westrich, K. D., Wilhelm, J. A., & Schur, C. L. (2016). Comparative effectiveness research in the U.S.A.: when will there be an impact on healthcare decision-making? Journal of Comparative Effectiveness Research, 5(2), 207–216.

    Article  Google Scholar 

  • Willis, T. A., Hartley, S., Glidewell, L., Farrin, A. J., Lawton, R., McEachan, R. R., et al. (2016). Action to Support Practices Implement Research Evidence (ASPIRE): Protocol for a cluster-randomised evaluation of adaptable implementation packages targeting ‘high impact’ clinical practice recommendations in general practice. Implementation Science, 11(1), 25.

    Article  Google Scholar 

  • Wooding, S., Hanney, S., Pollitt, A., Buxton, M., & Grantm, J. (2011). Project Retrosight. Understanding the Returns from Cardiovascular and Stroke Research: Policy Report. Cambridge: RAND Europe.

    Google Scholar 

  • Wu, Z. (2015). Average evaluation intensity: A quality-oriented indicator for the evaluation of research performance. Library & Information Science Research, 37(1), 51–60.

    Article  Google Scholar 

  • Pastakia, C. M. R. (1998). The rapid impact assessment matrix (RIAM)—A new tool for environmental impact assessment. In K. Jensen (Ed.), Environmental impact assessment using the rapid impact assessment matrix (RIAM). Fredensborg, Denmark: Olsen & Olsen.

    Google Scholar 

  • Yiend, J., Chambers, J. C., Burns, T., Doll, H., Fazel, S., Kaur, A., et al. (2011). Outcome measurement in forensic mental health research: An evaluation. Psychology, Crime & Law, 17(3), 277–292.

    Article  Google Scholar 

  • Zaratin, P., Battaglia, M. A., & Abbracchio, M. P. (2014). Nonprofit foundations spur translational research. Trends in Pharmacological Sciences, 35(11), 552–555.

    Article  Google Scholar 

  • Zaratin, P., Comi, G., Coetzee, T., Ramsey, K., Smith, K., Thompson, A., et al. (2016). Progressive MS Alliance Industry Forum: maximizing collective impact to enable drug development. Trends in Pharmacological Sciences, 37(10), 808–810.

    Article  Google Scholar 

  • Zelefsky, M. J., Lee, W. R., Zietman, A., Khalid, N., Crozier, C., Owen, J., et al. (2013). Evaluation of adherence to quality measures for prostate cancer radiotherapy in the United States: Results from the quality research in radiation oncology (QRRO) survey. Practical Radiation Oncology, 3(1), 2–8.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. ALTIS-Alta Scuola Impresa e Società, Università Cattolica del Sacro Cuore, Via San Vittore 18, 20123, Milan, Italy

    Matteo Pedrini & Valentina Langella

  2. Dipartimento di Scienza della Vita, Università di Siena, San Miniato, Via Aldo Moro, 2, Via Pier Andrea Mattioli, 4, 53100, Siena, Italy

    Mario Alberto Battaglia

  3. Fondazione Italiana Sclerosi Multipla, Via Operai 40, 16149, Genoa, Italy

    Paola Zaratin

Authors
  1. Matteo Pedrini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Valentina Langella
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mario Alberto Battaglia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Paola Zaratin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matteo Pedrini.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pedrini, M., Langella, V., Battaglia, M.A. et al. Assessing the health research’s social impact: a systematic review. Scientometrics 114, 1227–1250 (2018). https://doi.org/10.1007/s11192-017-2585-6

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-017-2585-6

Keywords

search

Navigation