Background

Low and middle income countries (LMIC) face the highest burden of disease which they must tackle despite severe resource limitations [1]. A key aspect of this is ensuring that health services and the delivery of health care are run in the most efficient way possible. This includes deployment of the least expensive human resources compatible with competent practice, for example. However, the majority of economic evaluation evidence concerns high income countries, [2] which is unlikely to be generalizable to LMICs [3]. Furthermore, while methods for the economic evaluation of patient-level clinical interventions such as specific pharmaceuticals have been well-developed and embedded into health technology assessment in high income countries [4], very limited evidence exists for service delivery interventions. And again this mainly concerns high income countries [5, 6].

A service delivery intervention can be conceived as sitting between clinical interventions acting directly on the patient and national or regional level policy interventions [5]. More specifically, service delivery interventions affect: to whom care is provided, and by whom; where care is provided; information, communication and technology that enhance the care process; and improvements to the quality and safety of care. Specific service delivery interventions in LMICs include: task shifting from more highly trained healthcare professionals to other health workers with less training and fewer qualifications, such as community health workers (CHWs); use of secondary care providers in guideline dissemination to change outpatient referral rates; outreach clinics, such as provision of a night clinic for sex workers; and emergency care quality improvement training [7]. We exclude financing and governance arrangements, which are considered policy-level interventions.

A number of previous and ongoing attempts have been made to review and rank clinical interventions in LMICs by cost-effectiveness [8,9,10]. A notable example is the World Bank’s Disease Control Priorities in Developing Countries, which aims to identify the most effective and cost-effective interventions to tackle a wide range of disease areas across LMICs. The discussion of cost-effectiveness in these analyses is typically with respect to whether or not to provide an intervention or treatment. For example, provision of anti-retroviral therapy to prevent mother-to-child transmission of HIV/AIDS or isoniazid treatment for TB are widely considered cost-effective when the comparator is no treatment [10]. However, knowledge remains limited regarding the most cost-effective way to organise the delivery of these interventions.

There is a growing volume of literature assessing CHW programs. The use of CHWs has proven to be an important service delivery innovation in low income countries as they can provide basic health services at a low cost. Recent systematic reviews of economic evaluations of CHW programs have identified a number of relevant studies [11, 12]. However, they concluded that these were generally for the introduction of small, vertical programs and not more relevant large scale, horizontally-focussed initiatives. From an economic perspective many of these studies were furthermore of limited utility, because they did not include a measure that integrated estimates of incremental costs and incremental benefits, such as the expected net benefits or incremental cost-effectiveness ratio (ICER), which is required to inform resource allocation and service delivery decisions [13]. The review by Nkonki and colleagues incorporated many studies that reported only costs or average costs and average benefits [11]. There remains a question around the evidence from ‘full’ economic evaluations for service delivery interventions, including those for CHW programmes.

In 2008, Lewin et al. conducted an overview of the evidence of the effectiveness of service delivery interventions in LMICs with the aim of “supporting the delivery of cost-effective interventions.” They organised these interventions into three categories: delivery arrangements, governance arrangements, and financial arrangements [7]. At the time the authors identified a lack of high quality evidence in this area and did not consider whether the interventions themselves were indeed cost-effective. However, they provided a useful taxonomy and definition of service delivery interventions, which we make use of here to examine the published evidence on their incremental cost-effectiveness or cost-benefit.

We aimed to review the use of health economic evaluation for service delivery interventions in LMICs. There is often considerable heterogeneity in study design between health economic evaluations, even for the same intervention. Various forms of effectiveness evidence may be incorporated, a wide variety of models may be used to extrapolate to the endpoints of interest, and the endpoints themselves may vary from disease specific outcomes, such as cases of tuberculosis treated, to generic outcomes, such as disability adjusted life years (DALY) averted. These differences may make it difficult for a decision-maker to discern which evidence is relevant to them. There is therefore a strong need to conduct systematic searches of studies in this area and amalgamate and review them. The aims of this article are fourfold. First, we aim to describe the scope of economic evaluations in service delivery research in LMICs. Second, to describe the quality of the studies and to compare and contrast methods used. Third, to summarise as far as possible the findings from the studies. And fourth, to make recommendations for policy and practice.

Methods

We conducted a systematic review of economic evaluations of service delivery interventions in LMICs.

Definition of a service delivery intervention

There exists no universally accepted definition of a service delivery intervention, which we discern to lie between clinical interventions and policy interventions in their proximity to the patient. Acknowledging inevitable overlap between intervention types, we considered (for the purpose of this review) service delivery interventions as those that could be judged as being classified in the taxonomy specified by Lewin et al. as affecting ‘Delivery arrangements’, but not financial and governance arrangements, which were considered policy interventions [7]. Specifically, service delivery interventions affect: (i) to whom care is provided and efforts made to reach them, (ii) by whom care is provided, (iii) where care is provided, (iv) with what information and communication and technology is care provided, and/or (v) how the quality and safety of care is improved and monitored.

Search strategy

We adapted the search strategy from Lewin et al., who conducted a systematic review of reviews of evidence of the effectiveness of service delivery interventions in primary health care in LMICs. To adapt this strategy we removed terms pertaining specifically to reviews and specifically to primary care. We added terms to identify economic evaluations including cost-benefit analysis, cost-effectiveness analysis, cost per DALY, cost*, economic evaluation and so forth. The full search strategy is available in the Supplementary Information. We searched PUBMED, MEDLINE, EconLit, and NHS EED for studies published between 1st January 2000, when the millennium development goals were initiated, and 30th October 2016. There were no language restrictions. LMICs were defined according to the World Bank [14]. This review was initiated as part of an internal project and the protocol was submitted to and approved by the medical school but was not registered with PROSPERO. It is provided in the Supplementary Information. Each abstract was reviewed independently by two reviewers. Full text articles were obtained for all studies selected by at least one reviewer. Disagreements about inclusion were resolved through team discussion.

Inclusion criteria

Studies were included if: (i) it was an economic evaluation that reported both incremental outcomes and incremental costs. These included studies reporting incremental cost-effectiveness ratios (ICER; e.g. cost per case cured), incremental cost-benefit ratios (e.g. cost per DALY), and expected incremental net benefit estimates (net monetary value); (ii) the study examined a service delivery intervention as defined above; (iii) the study considered only intervention(s) implemented in a LMIC, and (iv) the study was published between 1st January 2000 and 30th October 2016.

Exclusion criteria

Evaluations reporting only cost differences, effectiveness estimates, or average costs or effects were excluded, including cost-minimisation analyses. No restrictions on language were applied.

Data extraction

The following data were extracted from each study where reported: author(s); publication date; study date; description of intervention(s) and comparator(s); study country(ies); methods; source(s) of effectiveness evidence and their sample size(s), unit of effectiveness, mean age(s) of participants, and sex; source(s) of costs evidence; mean and median costs for treatment and comparator and their differences and 95% confidence intervals; currency; cost utility; cost benefit or cost-effectiveness ratio and 95% confidence intervals or expected net benefits; probability cost-effective; time horizon, and perspective. HS extracted the data and SW independently cross-checked the data against the original studies.

Quality assessment

The quality of the economic evaluations was judged using the ten point Drummond checklist. [15] Each study was scored independently by two authors (SW and HS) out of a maximum of ten points. Disagreements were resolved by discussion.

Synthesis and evaluation

Interventions were grouped according to the taxonomy in Lewin et al. (Panel 1) discussed above [7]. Estimates of ICERs or equivalent outcomes were grouped by both disease area and outcome. Permutation plots were completed for interventions judged through discussion to be similar and with similar comparators [16]. To aid comparability between results from different countries and over different time periods, ICERs, or equivalent outcomes, were converted into multiples of GDP per capita for the relevant year and country. This also permits comparability with World Health Organisation (WHO) recommendations of cost-effectiveness thresholds; less that one times GDP per capita per DALY averted is considered highly cost-effective and less than three times is considered cost-effective [17]. GDP per capita was obtained from the World Bank for the year of the relevant study in nominal, purchasing power parity terms to enable comparability between studies of different years in this metric. Where study year was not reported, publication year was used instead. A meta-analysis was not deemed appropriate for these data, as service delivery interventions are typically adapted to specific contexts, and methods and data differ widely between studies.

Results

Studies included and excluded

Overall, 3818 potentially relevant abstracts were identified of which 101 studies were selected for full text review. Sixty-five studies were excluded: 15 were considered clinical interventions and 10 were considered policy (governance or finance) interventions, nine studies were not economic evaluations; an incremental ratio or net benefits could not be calculated from 23 studies, two studies were not in an LMIC country, and the remaining six did not report on a specific intervention. In total, 36 full text articles were included in the review (Fig. 1). The included studies are reported in Table 1. The clinical topics covered were HIV (ten studies), obstetric care (four studies), child health (three studies), while the rest covered other or multiple areas of care.

Fig. 1
figure 1

PRISMA diagram

Full size image
Table 1 Summary of studies included in the review
Full size table

Comparison of methods

The 36 included studies differed widely in terms of the measures of effectiveness, perspective, time horizon, and other methods. Table 2 details the methods and approaches used in the included studies. Nine studies were economic evaluations alongside a randomised trial, while the other studies were a mix of economic evaluations from observational evidence or modelling studies. The majority (21) of the included studies used basic arithmetic approaches to calculate ICERs or equivalent, the others used more complex model structures to extrapolate from trial-based or other sources of evidence. Most studies (26) reported conducting some form of sensitivity analysis although only 13 conducted probabilistic analyses.

Table 2 Methods used by the studies included in the review
Full size table

As an illustration of methodological heterogeneity between studies, ten studies examined delivery interventions for HIV/AIDS in Sub-Saharan Africa (Tables 1 and 2) [18,19,20,21,22,23,24,25,26,27]. None of these studies used the same effectiveness measures. Some used health outcomes: QALYs, DALYs, ‘favourable’ immune response, life expectancy, death rate, cases averted, and undetectable viral load. The others used ‘surrogate’ endpoints: cases identified, uptake of testing, patients retained, and “in care and responding”. As shown in Table 2 eight of these ten studies took a provider or health care perspective [18,19,20, 22, 24,25,26,27] and two a societal perspective [21, 23]; the time horizon varied from less than one year to 20 years; and only four included cost savings contingent on reduced morbidity. This heterogeneity was reflected across all the studies included in the review rendering any further comparisons between studies and interventions inappropriate.

Comparison of study quality

Quality scores varied widely according to the Drummond ten point checklist. Scores ranged from three to ten out of ten (Table 1). Aside from the difficulties with comparability between studies, the most common weaknesses in the included studies were the establishment of effectiveness evidence (question 3: 17/36 studies meeting the criterion) and the credible valuation of costs and consequences (question 6: 21/36 studies). Poor reporting also prevented determination of whether criteria were met. As reported in Table 2, it was not possible to establish discount rates for a number of studies (Question 7: 14/36 studies); discount rates were potentially not reported for short time horizons, but may have still been required for the valuation of capital.

Assessment of task shifting interventions

The most widely studied types of delivery intervention were ones that altered “by whom and where” care was provided. We classified 11 and 12 of the 36 included studies in these categories, respectively. The predominant focus of these interventions was provision of typically doctor-led care by nurses or CHWs (eight studies [18, 19, 28,29,30,31,32,33]), or shifting place of care, such as providing treatment at home or in the community instead of at a facility (12 studies [20,21,22,23,24,25,26, 34,35,36,37,38]). The remaining three studies compared guardian to CHW supervised treatment for TB [39], increasing midwifery staffing, [40] and an integrated community-based programme for infectious disease (as compared to standard facility based care) [41].

Twenty of these 23 studies evaluated shifting care to potentially less costly settings: specifically either from doctors to nurses or CHWs, or from a facility to the home or community. Nine of these 20 reported the incremental cost per DALY averted or cost per QALY gained so that the results could be compared to WHO thresholds [20, 24, 29,30,31, 35,36,37, 41]. Seven of these nine that reported DALYs were considered highly cost-effective or cost-saving, i.e. less costly and at least as effective, when compared to WHO thresholds. A further three of the 20 studies were likely to be considered cost-effective on the basis of life years saved [23, 32, 38], and four more studies reporting other effectiveness outcomes were estimated to be cost-saving overall compared to a relevant alternative [18, 22, 28, 33]. Fig. 2(a) shows a permutation plot of the 20 studies examining shifting care to potentially less costly settings grouped by their estimated changes to costs and health outcomes and the resulting implications for decision makers. Taken together the evidence suggests that shifting tasks from doctors to CHWs, nurses, and/or from facility into the community is likely to be cost-effective and even potentially cost-saving.

Fig. 2
figure 2

Permutation plots summarising the number of economic evaluations according to their findings. Numbers in cells represent numbers of studies. a Shifting care to less costly settings: Community based care or task shifting versus facility based or doctor-led care. b Quality improvement: QI initiatives for community, primary, and secondary care

Full size image

Assessment of quality improvement interventions

Nine studies reported an economic evaluation of a quality improvement initiative. Their incremental costs and benefits are summarised in Fig. 2(b) [27, 42,43,44,45,46,47,48,49]. Five of these examined quality improvement in health care facilities, all of which were considered highly cost-effective [43, 45,46,47,48]. These schemes were comprised typically of staff training and improved supervision. Medical equipment and quality assurance was also included in two of these studies [46, 47]. Beyond quality improvement initiatives in formal health care facilities, one study compared high and low staff to patient ratios, but did not find that well-staffed HIV clinics were cost-effective compared to the standard care with lower staff-patient ratios [27]. A large rural primary care quality improvement initiative in Ethiopia that implemented and improved community and primary care infrastructure, funding, and services simultaneously also failed to demonstrate cost-effectiveness [46]. However, drug safety training for drug-dispensing shopkeepers was estimated to be highly cost-effective in Uganda [42]. This latter finding reflects the results from similar, non-quality focussed initiatives included in the review but classified in other categories, such as interventions to improve the knowledge of lay trauma first-responders in Uganda [32], and providing cardio-vascular disease training to CHWs in South Africa [29]. Taken altogether there appears to be good evidence that quality improvements through training and supervision are highly cost-effective. Only four quality improvement studies appeared to take into account cost-savings to the health service contingent on reduced morbidity [43, 45, 47, 49].

Assessment of other types of service delivery intervention

There were four further studies included in the review that examined a range of different interventions. Two considered information and communication technology interventions for hospitals, one in China [50] and one in India [51]. Two articles considered interventions expanding helminth control programs: one to schools [52] and the other to the whole community [53]. All four of these interventions were estimated to be cost-effective or cost-saving.

Discussion

For two broad classes of intervention identified in this review the evidence suggests that they are cost-effective or potentially cost-saving. Firstly, shifting basic care from facilities to the community or the home, or from doctors to nurses or CHWs has been shown to be cost-effective across a broad range of conditions. Testing, counselling, and management of HIV at home was generally more costly but also more effective than doing so in facilities. Similarly, involving and training lay members of the community in emergency care, CVD management, and mental health all proved cost-effective. However, the task shifting programs were generally small-scale, local initiatives by researchers or healthcare providers, Further evidence is required to assess these interventions at scale. Secondly, quality improvement initiatives in community, primary, and secondary care involving staff training, supervision, and equipment upgrades were also generally considered cost-effective. However, the significant heterogeneity in the design and reporting of the economic evaluations rendered more specific quantitative comparisons of results between interventions inappropriate.

A key purpose of economic evaluation is to inform decisions. But even within the same disease area in the same region, a decision maker would have difficulty synthesising the available studies to come to an informed decision on the basis of the studies reviewed here. For example, service delivery interventions for HIV in Sub-Saharan Africa used a wide range of different outcomes, perspectives, methods, and time horizons. Without knowledge of how the results compare in terms of general cost-effectiveness measures, such as a cost per DALY, a decision maker would not be able to choose from among the many similar but ultimately mutually exclusive alternative organisational arrangements. There is a strong case for a more standardised way of conducting and reporting economic evaluations in this area, while respecting the differences between context-specific interventions. This is a key area for future research and reporting guidelines.

Lewin and colleagues studied service delivery research across LMICs ten years ago and noted a ‘dearth of evidence in low and middle income countries’ [7]. More recent reviews of service delivery interventions, such as CHW programs, [11, 12] show that the evidence base is growing in this area. Nevertheless, these recent reviews of CHW programs and the results presented here demonstrate that there is still insufficient good quality evidence to support policy makers’ decisions in this area. The subject is clearly still at an early stage of development and the time is propitious to influence the development of the field.

There is no clear line that delineates service delivery interventions from policy or clinical interventions. Rather these interventions exist on a spectrum. A clinical intervention is applied directly to a patient whereas a service delivery intervention is applied to the organisation of a clinical intervention and its delivery [5]. As such, there may be some disagreement with regard to the eligibility of studies selected for inclusion here, for example, whether the extension of helminth control programs or public health behavioural change interventions are service delivery interventions. Governance and finance arrangements were excluded as they were classed as policy interventions and concern decision makers at a different level. For example, we excluded an economic evaluation of a private sector capitation model for the public sector for diabetes in South Africa [54]. The economic evaluation of health care policy in LMICs is important for effective decision making on health care organisation and should be the subject of future research. We did not formally consider publication bias given the lack of an agreed method to do so for economic evaluations. It is possible that there is a bias towards publishing results that appear cost-effective, or that economic evaluations are undertaken only for interventions thought to be effective. This is another important area for future research.

The design and organisation of health services is a growing and important issue. The severe resource limitations in many LMICs necessitate high quality economic evaluation. Guidelines and data resources for economic evaluation in LMICs such as WHO Choice are available [55]. However, there is a high level of methodological heterogeneity between studies in LMICs and when compared to studies in high income countries. This may be due to levels of health economics research capacity or different expectations of funders, journals, or peer reviewers [56]. Previous research has also demonstrated the wide differences in results from different approaches, such as between top-down and bottom up costing approaches [57]. A key recommendation of this study is therefore standardisation of research methods and ensuring adherence to reporting guidelines. This can be achieved by funders, journals, and policy makers. Standardising outcomes may be hindered by data limitations, although DALY values for many diseases are published, such as by the Global Burden of Disease studies [58]. Other studies have emphasized the need for broad measures of general health and well-being to both capture benefits across multiple dimensions and ensure comparability between studies [59].

Conclusions

There is a large evidence base supporting effective and cost-effective treatment of the diseases afflicting LMIC such as HIV, TB, and malaria. However, the evidence supporting the optimal configuration of services is highly limited and is an important direction for future research.

Available evidence generally supports shifting care to less costly settings, such as in the community instead of a health care facility, or task shifting from doctors to CHWs. Quality improvement initiatives were generally found to be more costly and more effective. However, there is significant heterogeneity in the literature, both in methodological approaches and in quality. This renders further comparisons difficult and limits the utility of the available evidence to decision makers. Given the severe resource limitations in LMICs, there is a pressing need for high quality economic evaluation of service delivery interventions and a need for standardisation of methods and reporting to facilitate its use to decision makers.