Abstract
Research for development (R4D) aims to make a tangible difference to development challenges, but these effects typically take years to emerge. Evaluation (especially impact evaluation) often takes place before there is evidence of development impact. In this paper, we focus on opportunities for assessing the potential for impact at earlier stages in the research and innovation process. We argue that such a focus can help research programme managers and evaluators learn about the pre-conditions for impact and adjust accordingly. Using the Global Challenges Research Fund (GCRF) as a large-scale case of R4D evaluation, we identify and explore some of the building blocks that can increase impact potential. Guided by GCRF’s theory of change, we explore emerging evidence that highlights the importance of ways of working that supports positioning for impact. We conclude by drawing out a unifying construct around standards of development excellence; to sit alongside notions of scientific excellence for research intended to have an impact. Standards can help programme managers, researchers and evaluators learn and adapt to increase the likelihood of impact.
Résumé
L'impact de la recherche prend généralement du temps à se matérialiser. Souvent, l'évaluation des impacts vient en fin de course. Dans cet article, nous nous concentrons sur la possibilité d'évaluer le potentiel d'impact aux premières étapes du processus de recherche et d'innovation. Nous soutenons qu'une telle focalisation peut aider les gestionnaires de programmes de recherche et les évaluateurs à connaître les conditions préalables à l'impact et à s'adapter en conséquence. En nous appuyant sur l'évaluation du Fonds de recherche sur les défis mondiaux (GCRF), nous identifions et explorons certains des éléments constitutifs qui peuvent accroître le potentiel d'impact. En nous appuyant sur la théorie du changement du GCRF, nous nous appuyons sur des preuves émergentes qui soulignent l'importance des méthodes de travail qui permettent de se positionner pour avoir un impact. Nous concluons en dégageant une construction fédératrice relative aux normes d'« excellence du développement» pour accompagner les notions d'excellence scientifique pour la recherche. Les normes peuvent aider les gestionnaires de programme, les chercheurs et les évaluateurs à apprendre et à s'adapter pour augmenter la probabilité d'avoir un impact.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The GCRF evaluation is implemented by an international consortium led by Itad Ltd, with RAND Europe, AFIDEP, Athena Infonomics and NIRAS-LTS. The authors would like to acknowledge and highlight the contributions of the evaluation team members to this paper, through their work on the GCRF evaluation modules from 2020–22: Melanie Punton, Jeevan Raj Lohani, Ekaterina Shaleva, Giorgia Giambi, Eve Mackinnon, Barbora Sladkova, Doug Elsey, Victoria Sword-Daniels, Mary Ann Brocklesby, David Walker, Douglas Elsey, Yannick Vuylesteke, Henry Cust, Nateisha Decruz-Young and Danielle Freed (Itad); Susan Guthrie, Hamish Evans, Joe Francombe, Cagla Stevenson and Mann Virdee (RAND Europe); Salome Wawire, Rose Oronje and Violet Murunga (AFIDEP); Anupama Ramaswamy (Athena Infonomics); Bouchra Atkinson, Valeria Izzi, Rebecca Murray, Diana Mataya and Colleen Sullivan (NIRAS-LTS).
In addition to the regular lines of scrutiny (such as departmental reporting to HM Treasury, and the National Audit Office studies), the International Commission for Aid Impact (ICAI) provides specific scrutiny for the UK’s aid spend. ICAI reports to Parliament through the House of Commons’ International Development Committee.
At the time of its launch in 2016, GCRF aligned with the 2015 Aid Strategy. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/478834/ODA_strategy_final_web_0905.pdf.
GCRF is delivered through 17 delivery partners including the seven Research Councils and Innovate UK; its umbrella organisation, the UK Research and Innovation (UKRI); the four National Academies; Innovate UK; the UK Space Agency (UKSA); plus, the four higher education funding councils. These DPs manage and disburse finding through the existing system of universities and other research organisations, as well as to their partners in low and middle -income countries. Higher education funding is devolved to the four nations of the UK, and administered by the governments of Scotland, Wales and Northern Ireland. In England, this funding stream is administered by Research England.
The Haldane principle, which ensures that research funding decisions are made by experts in the field. This is enshrined in law in the 2017 Higher Education and Research Act.
The use of ‘ODA’ (Official Development Assistance) rather than ‘development’ reflects the framing commonly used by BEIS and the UK Delivery Partners. ‘ODA research and innovation excellence’ is a working concept that describes the quality of approaches used to manage research for development impact (such as integrating a focus on gender, inclusion and poverty, fairness, relevance and coherence into the design and delivery of R&I projects).
For the purposes of this evaluation, relevance is framed in relation to OECD DAC criteria, where it is defined as ‘The extent to which the intervention objectives and design respond to beneficiaries, global, country, and partner/institution needs, policies, and priorities and continue to do so if circumstances change’. It is also framed in relation to the Canadian International Research Centre’s research quality instrument, RQ+, around research importance: ‘[T]he importance and value to key intended users of the new knowledge and understanding generated by the research’, and how far ‘research processes and products’ are relevant to the needs and priorities of potential users (Ofir et al. 2016).
UKRI and the UK Collaborative on Development Research developed a webpage of resources to support standards and good practice around equitable partnerships https://www.ukri.org/about-us/policies-standards-and-data/good-research-resource-hub/research-in-a-global-setting/
These resources include links to the ‘Equitable Partnerships Resource Hub. https://www.ukcdr.org.uk/guidance/equitable-partnerships-hub/;
‘Global code of conduct for research in resource-poor settings’, https://ec.europa.eu/research/participants/data/ref/h2020/other/hi/coc_research-resource-poor-settings_en.pdf.
The principles developed by the Rethinking Research Collaborative on promoting fair and equitable research partnerships for global challenges. https://rethinkingresearchcollaborative.com/2018/10/04/research-report-promoting-fair-and-equitable-research-partnerships-to-respond-to-global-challenges/
The web page also includes links to model partnership agreements, due diligence guidance and ethical guidance for research in developing countries. There were also links to good practice examples of working in a fair and equitable way with partners.
IDRC, 2020. Research Quality Plus. Available at: https://www.idrc.ca/en/research-in-action/research-quality-plus.
References
Alla, K., W.D. Hall, H.A. Whiteford, B.W. Head, and C.S. Meurk. 2017. How do we define the policy impact of public health research? A systematic review. Health Research Policy and Systems 15: 84. https://doi.org/10.1186/s12961-017-0247-z.
Barr, J., B. Bryan, P. Kolarz, X. Potau, M. Punton, P. Simmonds, and I. Vogel. 2018. GCRF evaluation foundation stage, final report for BEIS. Itad and Technopolis: Brighton, UK. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/810137/GCRF_Evaluation_Foundation_Stage_Final_Report.pdf
BEIS. 2017a. Global Challenges Research Fund (GCRF): How the Fund Works. https://www.gov.uk/government/publications/global-challenges-research-fund/global-challenges-research-fund-gcrf-how-the-fund-works#:~:text=GCRF%20forms%20part%20of%20the,the%20poorest%20people%20and%20countries.
BEIS. 2021. Gender Equality in Research and Innovation Official Development Assistance (ODA). https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/989934/research-innovation-oda-gender-equality.pdf.
Blume, S. S. 1977. Policy as Theory: A Framework for Understanding the Contribution of Social Science to Welfare Policy 1. Acta Sociologica 20(3): 247–262. https://doi.org/10.1177/000169937702000302
Boaz, A., S. Fitzpatrick, and B. Shaw. 2009. Assessing the impact of research on policy: a literature review. Science and Public Policy 36(4): 255–270. https://doi.org/10.3152/030234209X436545
Boaz, A., S. Hanney, R. Borst, A. O’Shea, and M. Kok. 2018. How to engage stakeholders in research: design principles to support improvement. Health Research Policy and Systems 16(1). https://doi.org/10.1186/s12961-018-0337-6.
Boswell, C., and K. Smith. 2017. Rethinking policy ‘impact’: four models of research-policy relations. Palgrave Communications 3: 44. https://doi.org/10.1057/s41599-017-0042-z.
Brocklesby, M. A., D. Walker, D. Elsey. E. Shaleva, and V. Fernandez. 2022. GCRF evaluation gender equality, social inclusion and poverty audit, Itad Ltd. https://www.itad.com/wp-content/uploads/2022/12/GCRF-GESIP-Staqe-1a_Full-Report_formatted.pdf.
Cairney, P. 2016. The politics of evidence-based policymaking. London: Palgrave Pivot.
Fransman, J. 2018. Charting a course to an emerging field of 'research engagement studies': A conceptual meta-synthesis. Research for All 2(2): 185–229. https://doi.org/10.18546/RFA.02.2.02.
Fransman, Jude, Budd Hall, Rachel Hayman, Pradeep Narayanan, Kate Newman, and Rajesh Tandon. 2018. Promoting fair and equitable research partnerships to respond to global challenges. Rethinking Research Collaborative. http://oro.open.ac.uk/57134/1/Fair%20and%20Equitable%20Partnerships_Research%20Report%20-%20PUBLIC.pdf.
Georgalakis, J., and P. Rose. 2019. Exploring Research-Policy Partnerships in International Development. Brighton: IDS Bulletin Institute of Development Studies (IDS).
Guthrie, S., C. d'Angelo, B. Ioppolo, Y. Shenderovich, and G. R. McInroy. 2018. Evidence synthesis on measuring the distribution of benefits of research and innovation. Santa Monica, CA: RAND Corporation. https://www.rand.org/pubs/research_reports/RR2610z1.html.
Hicks, D., P. Wouters, L. Waltman, S. de Rijcke, and I. Rafols. 2015. Bibliometrics: The Leiden Manifesto for research metrics. Nature 520(7548): 429–431. https://doi.org/10.1038/520429a.
Izzi, V. 2018. Research with development impact. Lessons from the Ecosystem Services for Poverty Alleviation programme, Working Paper, ESPA. https://www.espa.ac.uk/files/espa/Research%20with%20development%20impact_WP_final.pdf.
Izzi, V., B. Murray, and C. Sullivan. 2022. Global challenges research fund evaluation, research fairness module report, Itad ltd. https://www.itad.com/wp-content/uploads/2022/12/GCRF-Fairness-Stage-1a_Full-report_formatted-version.pdf.
Langdon, J. 2013. Decolonising development studies: reflections on critical pedagogies in action. Canadian Journal of Development Studies/Revue canadienne d'études du développement 34(3): 384–399. https://doi.org/10.1080/02255189.2013.825205.
Lawrence, D. S., and L. A. Hirsch. 2020. Decolonising global health: transnational research partnerships under the spotlight. International Health 12(6): 518–523. https://doi.org/10.1093/inthealth/ihaa073.
McLean, R. K. D. and Sen, K., 2019. Making a Difference in the Real World? A Meta-analysis of the Quality of Use-oriented Research Using the Research Quality Plus Approach. Research Evaluation 28(2), pp.123–135. Available at: https://doi.org/10.1093/reseval/rvy026
McLean, Robert, Z. Ofir, A. Etherington, M. Acevedo, and O. Feinstein. 2022. Research Quality Plus (RQ+)—Evaluating research differently. International Development Research Centre (IDRC): Ottawa, Canada. https://idl-bnc-idrc.dspacedirect.org/bitstream/handle/10625/60945/IDL-60945.pdf?sequence=2&isAllowed=y.
Murray, R. et al. 2021. ‘ESRC FCDO Joint Fund for Poverty Alleviation research Programme—Phase 3 Evaluation Final Report’, UKRI https://www.ukri.org/wp-content/uploads/2022/08/ESRC-080822-JointFundPovertyAlleviationResearchProgrammePhaseThreeEvaluation-FinalReport.pdf.
O'Riordan, A.-M., J. Copestake, J. Seibold, J. and D. Smith. 2013. Challenge funds in international development. Bath Papers in International Development and Wellbeing, no. 28, Centre for Development Studies, University of Bath.
Ofir, Z., T. Schwandt, C. Duggan, and R. McLean. 2016. Research quality plus a holistic approach to evaluating research. Ottawa: IDRC.
Pinnington, R. and C. Barnett. 2020. Research for Development (R4D) indicators: A review of funder practice, GCRF and Newton Funds, report produced for the Department for Business, Energy and Industrial Strategy (BEIS), Itad and Technopolis Group: Brighton, UK.
Punton, M., and J. Lohani. 2022. GCRF evaluation: relevance and coherence module Report, Itad. https://www.itad.com/wp-content/uploads/2022/12/GCRF-Relevance-Stage-1a_Full-Report_formatted.pdf
Reed, M.S., M. Ferré, J. Martin-Ortega, R. Blanche, R. Lawford-Rolfe, M. Dallimer, and J. Holden. 2021. Evaluating impact from research: A methodological framework. Research Policy. https://doi.org/10.1016/j.respol.2020.104147.
Smit, J.P., and L. K. Hessels. 2021. The production of scientific and societal value in research evaluation: a review of societal impact assessment methods. Research Evaluation 30(3): 323–335. https://doi.org/10.1093/reseval/rvab002
Taylor, P., and C. Tremblay. 2022. Decolonising knowledge for development in the Covid-19 Era, IDS Working Paper 566, Brighton: Institute of Development Studies. https://doi.org/10.19088/IDS.2022.018
Veras de Sandes-Guimarães, L., R. Velho and G. Ary. 2022. Plonski interdisciplinary research and policy impacts: Assessing the significance of knowledge coproduction. Research Evaluation 31(3): 344–354. https://doi.org/10.1093/reseval/rvac008
Vogel, I., V. Sword-Daniels, and S. Guthrie. 2022. Stage 1a: Synthesis Report of evidence on integration of relevance, fairness, gender, poverty and social inclusion in funded activities, Evaluation of the Global Challenges Research Fund, October 2022a, produced by Itad with RAND Europe, AFIDEP, Athena Infonomics, NIRAS-LTS and Digital Science, London: BEIS, UK Government. https://www.gov.uk/government/publications/global-challenges-research-fund-gcrf-stage-1a-evaluation
Funding
Funding was provided by Department for Business, Energy and Industrial Strategy, UK Government.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors are contracted by the UK Government’s Department of Business, Energy and Industrial Strategy (BEIS) to conduct an independent evaluation of GCRF. The independent evaluation contract runs from 2016 to 2025. The research and evidence that the authors have drawn on for this article are in the public domain. The views expressed in the article are the authors’ own.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Vogel, I., Barnett, C. Laying the Foundations for Impact: Lessons from the GCRF Evaluation. Eur J Dev Res 35, 281–297 (2023). https://doi.org/10.1057/s41287-023-00579-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1057/s41287-023-00579-9