Background

There is a growing interest among both research funders [1,2,3] and academics [4,5,6,7] in identifying and measuring the social, environmental and economic benefits of research. Calls to better describe ‘impact’ are driven by the need to improve accountability, ensure relevance and inform funding [7,8,9]. Whether research outcomes can be ‘translated’ or applied in the real world is seen as one important way of assessing benefit [4, 5, 10]. Further, identifying pathways to impact during the design of research programmes can improve the quality and integrity of research by clarifying purpose and end-users [10, 11]. Interest in the benefits of health and biomedical research has been prominent in broader discussions on research impact [7, 9] due to the large amount of public funding it attracts [5] and the importance of tailoring outputs to the needs of clinicians and patients [5, 12].

Calls for evidence of impact have in turn catalysed work on how to measure it, and a range of approaches have been developed globally [8, 12,13,14]. One of the earliest and most widely-used [8, 9] approaches is the Payback model, introduced by Buxton and Hanney [15]. It aims to capture benefits in a range of areas such as knowledge generation, health services improvement and policy development, and has been adapted or modified a number of times [16]. Economic assessment (i.e. monetising research impacts) is also widely used, though typically at high levels, for example, aggregating research benefit nationally or in specific programmes over decades [17,18,19]. Project-specific approaches to measuring economic impact are emerging [8], though they have been critiqued for over-reliance on modelling and questionable assumptions [7, 17]. Narratives are a third, validated approach to describing impact, providing a summary of the research process and outcomes and have been the basis of the Research Evaluation Framework in the United Kingdom. Narratives have the advantage of being able to explain the complex (and often multi-directional) process through which impact occurs [7, 10, 11].

To date, efforts to measure impact have largely focussed on health research in high-income countries (HICs) [7, 8], reflecting where the majority of health research funding is spent as well as the limited infrastructure and capacity for health research in low- and middle-income countries (LMICs) [20, 21]. In 1990, the Commission on Health Research for Development, a consortium of global health agencies, researchers and development partners, identified the ‘90/10 gap’, i.e. that less than 10% of global health research spending is devoted to diseases or conditions that account for 90% of the global disease burden. This led to calls for a more equitable and systematic approach to prioritising health research investments [22, 23] and for development partners to devote 5% of official development assistance (aid) for health to research, as well as for LMICs to increase their own health research spending [24].

These calls have in part been answered – there is now more research in and on the health needs of low-income countries [25, 26] and a growing number of development partners are active in health research [27], including the United Kingdom’s Department for International Development, the United States Agency for International Development, and the Bill and Melinda Gates Foundation; these three agencies acknowledge the need to monitor the impact of their research investment [28,29,30]. Though none have standalone research impact frameworks, their policy documents refer to the need for research investments to create new knowledge and inform decision-making [28], build capacity for research in LMICs [28, 29] and facilitate local adaptation of evidence-based approaches (e.g. through implementation research) [29, 30] – all common elements of the ‘Payback’ model. Similarly, major funders of domestic health research in the United Kingdom, the United States and Canada now also fund research in LMICs directly [31,32,33,34] and through international collaborations [35]. These agencies do not appear to have standalone impact frameworks specific to their international collaborations (the United Kingdom Medical Research Council uses the Department for International Development framework); however, their domestic research impact models highlight knowledge generation and influence on policy and practice [36], economic growth (measured through links with business) and long-term health and environmental impact [2].

Thus, despite vast differences in health needs and research capacities in HIC and LMICs, there are similar expectations of what health research should achieve and how its impact should be measured in both contexts. As research investment in LMIC environments continues to grow, a better understanding of the challenges associated with research translation and measuring impact in LMIC contexts is likely to be useful. This study applies a research impact framework developed in a HIC (Australia) to research carried out in the Pacific and Indonesia. Our aim was to evaluate applicability, identify strengths and weaknesses, and make recommendations to support further use.

Methods

We carried out a rapid search for health research impact frameworks and selected the Hunter Medical Research Institute’s (HMRI) Framework to Assess the Impact from Translational health research (FAIT). Based on an extensive review of existing impact frameworks and with input from potential users, FAIT combines the three most commonly used approaches to impact assessment [10]. The first, based on the ‘Payback’ model, identifies ‘domains of benefit’. While each domain of FAIT is based on an existing approach to research impact, the combination of these approaches into a single tool is novel. Domains can be adapted to the research project under review but suggestions proposed by FAIT include knowledge generation, impacts on policy, clinical practice, health services or population health, and economic benefits. The second method comprises a cost–benefit analysis that compares costs (of the research itself and of implementing research recommendations), to social, environmental and economic benefits (expressed in monetary terms) that flow from implementation. Again, categories of benefit are flexible and left to the discretion of those completing the assessment. The third section is a short narrative that provides a summary of “how translation occurred and how research impact was generated” [10]. The text is structured around common sub-headings (need, research response, outcome, impact, lessons) and its purpose is to contextualise quantitative findings and explain outcomes.

We selected FAIT for three reasons. First, its mixed-methods approach, combining the three main approaches to measuring research impact provided an opportunity to test a range of impact measurement approaches in the LMIC context. Second, the framework emphasises translational health research, and is therefore well suited to the research projects we sought to review, which aimed to influence policy and practice. Third, FAIT can be applied to a range of research methods, from qualitative studies to implementation research to clinical trials, and so has a potential for wide application. FAIT is currently being applied to five projects within an Australian Centre for Research Excellence [37] but has not yet been applied in an LMIC context. HMRI colleagues agreed to engage in our study, adding value by reviewing our use of the FAIT tool.

Between March and September 2018, we applied FAIT to two recently completed research projects, namely (1) a programme to reduce salt consumption in two Pacific Island countries, Samoa and Fiji, and (2) the introduction of a digital health tool, SMARThealth, to improve the quality of diagnosis and treatment of cardiovascular diseases in East Java, Indonesia.

We chose these projects because good documentation was available from which data could be extracted. For the Pacific Salt project, there was a study protocol, process evaluations, impact evaluations and intervention costings for each Pacific country [38, 39]. For SMARThealth, an end-of-project completion report had been prepared for the funder, which included clinical results and a cost-effectiveness analysis. While FAIT is designed to be applied prospectively to encourage research translation and ensure the required evidence of impact is collected along the way, retrospective application represented a feasible approach to determine the applicability of FAIT in the LMIC context.

Our study was carried out in two stages. In stage one, we completed a first draft of the impact framework (presented in the FAIT scorecard format). We drew on existing documents to source the majority of data required, supplementing this with additional discussion and data mining where needed. The process was led by RD, who was not part of the research teams responsible for the two chosen projects, with support from BA to complete the economic analysis. The leads of each research team provided relevant documents and reviewed and amended RD’s first draft. In stage two, the lead author of the FAIT framework (AS) and the person leading its application and translation (SR) provided feedback and comment, which was critical to the process of refinement, including identifying additional areas of impact. While previous applications of FAIT have focussed on gathering impact data for the project under review, this study also considered the framework itself, and its applicability to a developing country context.

Results

Tables 1 and 2 present populated impact scorecards for the Pacific Salt project and the SMARThealth project. We found application of the ‘domains of benefit’ section to be feasible and useful – it helped to generate evidence of impact, including new data, in a range of areas not documented in existing project evaluations. For the Pacific Salt project, this included impacts on knowledge advancement and capacity-building, as well as indirect, positive impacts on the Samoan and Fijian economies through generating employment (in the research team) and spending of project funds (Table 1). For example, in relation to policy development, use of the framework drew attention to networks established with policy-makers, and the learning generated on the political economy of working with the food industry in Fiji, which research project leaders were aware of but had not previously documented. Support and prompting from HMRI was critical to the identification of these domains, and to the process of describing and quantifying specific benefits within them.

Table 1 Impact scorecard for the Pacific Salt project intervention, Fiji and Samoa, Pacific
Full size table
Table 2 Impact scorecard for SMARThealth intervention, Malang, East Java, Indonesia
Full size table

For the SMARThealth project, use of the ‘domains of benefit’ section helped to identify previously unrecognised areas of impact on knowledge generation and capacity-building of in-country partners. In addition, it prompted additional work to quantify recognised (but previously unreported) positive impacts on the health system; these included numbers of health workers trained, improvements to the medications supply, and better collection and sharing of patient data (Table 2). While these aspects were mentioned in the project evaluation, they had not been explicitly measured or identified as project benefits. For both projects, the narrative text provided a useful summary of the project and its impacts and provided an opportunity to reflect on lessons learnt.

We found generating data on the return on investment to be the most challenging aspect of FAIT, for a number of reasons. First, it required specialist input from a health economist (not available to all project teams). Second, data needed to model economic returns are often not easily available for the LMIC context, and in the case of the Pacific Salt project, were not collected during project implementation. This meant that areas of benefit identified retrospectively, such as the increased earning potential of staff in partner countries who gained skills through being involved in the project, could not be calculated. In Australia, standard pay scales for most professions are available and could be used to model such a benefit, but this is not the case for the majority of LMICs. Similarly, health gains and life years saved are commonly used measures of impact of interventions in HICs. However, they are more difficult to calculate in LMICs given the paucity of data.

Third, we found the broader context of poverty and health systems development had an impact on efficiency and hence economic return. For instance, SMARThealth provided support to allow a clinical task normally provided by a physician to be delivered by a lower-cost community health worker. In circumstances where physicians do provide this support, such a ‘shift’ would be cost saving. However, this was not the case in Indonesia, where the counter-factual was ‘no care’ so introducing the intervention represented a net cost to the health system, which in turn diminished the final social return on investment. Recognising these challenges, we attempted to monetarise the identified benefits of SMARThealth, drawing on the literature for both methods and estimates on which to base assumptions. Benefits of the intervention were modelled using estimates of the reduction in cardiovascular disease (CVD) events avoided as a result of the intervention. In summary, our assumptions were:

  • A relative risk reduction in CVD events (ischaemic heart disease, myocardial infarction or stroke) of 0.80 for every 10 mmHg reduction in systolic blood pressure, based on a recent systematic review and meta-analysis of randomised clinical trials [40].

  • One hospitalisation per CVD event.

  • Disability weights for people with CVD were adopted from the Global Burden of Disease Study, using an estimated weighted average from myocardial infarction and moderate to severe stroke weights, resulting in a disability-adjusted life year (DALY) weight of 0.39. A disability weight of 1 was used to reflect the dead health state and used to calculate the years of life lost [41].

  • Death rates resulting from CVD events were estimated using results from the literature for middle-income nations [42].

Using these assumptions, we then calculated:

  • The reduction in CVD events resulting from the reductions in blood pressure found through the trial, projected over a 5-year period.

  • The savings to the health system, based on the average cost of hospitalisations for CVD events in Indonesia.

  • The health gains for the population resulting from the intervention in terms of DALYs averted. Using estimates from the literature [43], we estimated each healthy life year gained to represent productivity gains of 6–12 months of per capita Gross National Income. We believed this to be a conservative estimate, given the average age of the cohort targeted by the intervention was 59 years, average life expectancy in Indonesia is 69 years [44], unemployment is relatively low at 6.9% [45], and almost two-thirds work in the informal sector where there is no mandatory retirement age [46].

  • Indirect and non-medical cost savings; using estimates from the literature, indirect benefits were valued as half the Gross National Income of Indonesia per capita per healthy life year gained as a result of the intervention.

These calculations were then compiled, divided by the cost of the research and delivering the intervention, and used to determine the estimated social return on investment (Table 2). Our approach was similar to one that might be used in a HIC environment, yet the resulting calculations were less robust given our reliance on estimates from outside Indonesia. We were unable to complete a similar set of calculations for the Fiji Salt Project due to lack of data collected during the project itself, though we did outline an approach for doing so (Table 1). Cost–benefit approaches are rarely used in LMICs, in part due to the types of constraints we encountered, including lack of a standardised approach and lack of data on which to base assumptions.

Discussion

Good practice in the delivery of aid and development assistance, including aid for health, has long emphasised principles of local ownership, effectiveness and sustainability [47, 48]. Accordingly, many research projects designed and implemented in low-income contexts intuitively emphasise engagement of local stakeholders, use of local systems and measurement of meaningful results (beyond academic outputs), suggesting that research impact models should be a ‘natural fit’ with LMIC-based health research.

There is recognised tension between the linear approach to impact implied by impact models, and the understanding that interactions between researchers and end-users are complex and iterative [10, 11, 49,50,51]. Indeed, reviews in HICs suggest the use of research impact models has favoured quantitative, empirical studies that can describe a clear, unambiguous outcome [52] and where economic returns are likely to be high [9]. In LMIC contexts, weak health governance and implementation environments [53, 54] mean the role and influence of research ‘evidence’ is even more problematic, and therefore measurement of impact even more challenging. If, as HIC reviews suggest, the use of impact models is cementing an existing bias in research funding towards statistical measures [5] at the expense of experimental or qualitative research design [52], this may be detrimental to LMIC research, where research infrastructure is often lacking, and qualitative methods are particularly needed to explain the poorly understood [53] governance environment in which research occurs.

Nevertheless, our experience suggests that impact models can play a useful role in hypothesising pathways through which impact is expected to occur, which can in turn prompt consideration of which stakeholders need to be engaged, what advocacy work (alongside research) may be required, and what vested interests could act as a barrier to uptake. This may be particularly relevant in LMIC environments and points to the need to consider research impact pathways prospectively, during the design of the research process, rather that retrospectively as we have done in this study.

Calculating economic benefit and return on investment

The challenge of valuing human life and calculating economic return on health investments is recognised in HIC contexts [8]. Our experiences suggest this challenge is exacerbated in the LMIC environment and, consequently, research projects in LMICs may struggle to show a positive return on investment. Issues include the following:

  • Low wages, high-levels of informal sector employment and/or unemployment mean that productivity gains (as commonly measured, in terms of income) associated with extending healthy life are difficult to estimate.

  • Poor levels of population health and low life expectancy (relative to HICs) may obscure gains in healthy life.

  • Poor coverage of essential health services means that introducing a new service, however essential and cost-effective in and of itself, may represent a net cost for the system (as no service was previously provided) diminishing the level of return.

  • The dearth of studies from LMICs on non-medical and indirect costs, such as transport to health facilities [55], make it difficult to estimate these, though they are often considerable [56, 57].

  • Overarching all of these issues is the fundamental challenge of poor quality health data in many LMICs [58] and the likely low statistical accuracy of globally standardised measures such as the DALY in LMIC contexts, which are relied on to estimate cost savings and economic benefits [59].

Cost-effectiveness analyses whereby the value of interventions is assessed in terms of natural units (for example, cost per CVD event avoided) or cost–utility analyses that assess interventions in terms of utility gained (for example, DALYs) are more common than cost–benefit analyses (where all benefits are monetised) in the health sector, including in LMIC contexts [60]. In addition to being more straightforward to estimate, cost-effectiveness allows a consideration of the relative value of an intervention, which can be used to inform a ‘business case’ on whether or not to implement the intervention more widely; such data is especially important in LMIC contexts, where resources are often scarce.

Cost effectiveness analyses are therefore a critical component of determining the broader societal ‘return’ on research investment (as FAIT attempts to do). Nevertheless, practical challenges remain in performing a full cost–benefit analysis in LMIC contexts, as we have demonstrated. Equally, in qualitative studies where research benefits cannot be monetised, such as a change in perceptions or attitudes, a cost–consequence analysis may be more applicable.

Finally, it is worth acknowledging that, while the Payback and narrative components of FAIT consider impact retrospectively, based on empirical evidence, the social return on investment models projected economic returns into the future. This may appear an anomaly; however, it is common practice for economic analysis to contain an element of forecasting given the challenge of demonstrating economic impact within the short time frame of a research project.

Suggestions for application of FAIT in LMICs

Prospective use with programme logic model

We applied the FAIT framework retrospectively, yielding important insights and new knowledge on study impacts. However, greater benefit is likely to come from applying the framework prospectively and in combination with a ‘programme logic model’ as intended by FAIT’s authors [10]. For example, prospective application of FAIT can help ensure relevant data is captured in the monitoring frameworks. In the Fiji example, we were unable to complete a social return on investment due to lack of data – a prospective application of FAIT would have indicated these data gaps. Equally, prospective application of FAIT aids consideration of potential positive and negative programme externalities. In Malang, for example, prospective application may have highlighted the potential impact of the intervention on the workload of community health workers, and led to monitoring of any adverse impact on other health tasks they performed, e.g. in maternal and child health.

Programme logic models, also called ‘theory of change’ models, are commonly used in the design of development assistance (aid) programmes [61] to identify areas of potential impact and understand the process through which change occurs. Many LMIC studies bridge the disciplines of research and development, and are thus well suited to use of programme logic methodologies. The FAIT programme logic model, in line with models commonly used in aid programmes, identifies the need or issue to be addressed, activities, expected outputs, end-users of those outputs and anticipated impact [10]. Additional file 1 published with this paper provides two examples of the application of the FAIT modified programme logic model to current LMIC research projects, demonstrating its feasibility.

Menu of metrics

The ‘domains of benefit’ section is a key strength of FAIT allowing identification of a range of benefits beyond the intervention/process that is the subject of the study. This is particularly useful in an LMIC context, where the ‘process of doing research’ may itself have positive externalities, for example, related to capacity-building for research, supporting local policy-makers, building the skills of the health workforce, or the economic impact of research project spending. However, our experience suggests that users of FAIT may need guidance (prompting) to identify and capture such benefits. The initial list of potential domains provided by FAIT is helpful; however, further suggestions on possible metrics linked to each domain would be useful, for example, on specific areas of potential economic benefit and how to calculate these, or how to measure the sustainability and impact of a knowledge network established during a study. This would help to ensure appropriate data collection is built into study design (e.g. on salary scales), in turn facilitating calculation of return on investment.

Calculating economic returns

We found calculating cost–benefit to be challenging and the results to have weak validity given issues of data quality and reliance on assumptions. Furthermore, we believe that there are many contexts where it will not be possible or meaningful to undertake cost–benefit analyses due to lack of data.

Nevertheless, as interest in measuring research impact grows, and is inevitably applied to LMIC contexts, further research on how to approach this challenge is likely to be useful.

Where cost–benefit analysis is not possible, cost-effectiveness analysis may provide a practical alternative. Especially where the focus of research relates to an intervention or service delivery change that can be costed, cost-effectiveness analysis should be routinely done as part of the intervention evaluation process. Such data can contribute to a business case to scale up interventions trialled during research by projecting future returns and can also inform future research funding investment. More broadly, incorporating any form of economic analysis into research impact assessment provides a valuable perspective, re-emphasising the imperative to ensure all spending choices deliver value for money – particularly important in LMIC contexts. As discussed above, conducting prospective analysis is important to identify (and make arrangements to collect) data required to conduct economic analysis.

Strengths and limitations

A key purpose of FAIT is to encourage research translation. To this end, it is designed for use throughout the implementation of research projects. We did not use the tool in this way – rather, we applied it retrospectively. Even so, we found it yielded useful findings. Further, we did not validate our impact claims through additional project evaluation as required by some impact templates [52]. However, an independent researcher led the process and HMRI’s involvement provided a level of external scrutiny. Indeed, our pragmatic approach to assessing impact was a strength of this study as it responds to a common critique of impact frameworks, namely that they take too long to complete and are too expensive to implement [11]. This approach was facilitated by the fact that the projects reviewed already included a significant focus on research translation through involvement of stakeholders (end users), and in the case of the Pacific Salt project, a comprehensive process evaluation [38, 39]. This points to a further limitation of our study, namely that we focussed exclusively on implementation research. Applying FAIT to other types of research project designs in LMICs will allow broader assumptions to be made about its applicability within the LMIC context.

Conclusion

Though developed to measure impact in Australian health systems, FAIT can be applied to research projects in LMICs. We found the mixed-methods approach to assessing impact to be a key benefit of FAIT. While we encountered challenges calculating return on investment, the use of the FAIT framework helped illuminate data gaps and highlighted the importance of considering affordability. We make suggestions that support further applications of FAIT in LMICs and, we hope, will contribute to an emerging conversation on how best to measure research impact in LMICs. In this context, future research that tests the applicability of other high-income research frameworks in low-income environments may be useful. Capacity-building for any staff using the framework is likely to be a worthwhile investment.