Background

Healthcare system sustainability is the ongoing capacity to deliver affordable and effective care that contributes to better health outcomes over time. There are many threats and challenges to the sustainability of healthcare systems across the world, including an ageing population, increasing costs of delivering healthcare, costly new medical technologies and growing consumer demand [1,2,3,4]. Crises, including natural disasters, large-scale accidents, epidemics, and pandemics add further challenges to already over-stretched healthcare systems [5, 6]. A sustainable healthcare system is one that is also resilient, such that it constantly adapts and endures despite these ever-changing pressures while maintaining performance in terms of health outcomes [7].

As healthcare systems strive towards delivering value-based care within these challenging contexts, improvement programs to increase quality, safety, effectiveness, and efficiency of healthcare have proliferated [8]. Quality improvement programs are now ubiquitous across healthcare sectors and facilities. Such programs are important to support innovations where new and more effective health technologies, models of care delivery, and financing are adopted and ideally, while old ineffective and inefficient ones are phased out.

To sustain the benefits from innovations in healthcare, innovations must be empirically evaluated to ensure they are indeed effective and deliver the outcomes that they promise, at scale and across different contexts. However, large-scale innovations are rare in healthcare systems and most innovations consist of improvement projects that tend to be short-term and proscribed—implemented in single centres or regions [1, 2]. Despite its importance, the sustainability of implemented improvement programs is under-researched with a limited evidence base to support decisions [9]. For example, discontinuing effective programs because of a lack of ongoing investment is wasteful and unethical [10]. Our understanding of how and why programs implemented in the real-world are sustained or discontinued is also limited, often because evaluations of improvement programs are almost always performed over the short term [11, 12]. The need for continuing investment in effective programs is well recognised as an important factor of sustainability, however, it is not the only factor [11, 12]. Shelton et al. [12] proposed the Integrated Sustainability Framework (ISF), which identifies important factors that help or hinder program sustainability. The ISF includes inner contextual factors (i.e., program champions, leadership/support, organisational resources/funding, staffing/turnover) and outer contextual factors (i.e., socio-political context, funding environment, external leadership, and values, needs and priorities), characteristics of the interventions or programs (i.e., perceived benefit/need, adaptability, and fit with context and population), characteristics of people or institutions implementing these programs (i.e., implementer/provider characteristics, implementer skills/expertise) and the processes used for implementation (i.e., partnership/engagement, training/supervision, program evaluation/data, adaptation) [12].

Healthcare system sustainability as applied to programs that are implemented in the healthcare delivery system are poorly defined and understood conceptually. In their systematic review of 125 studies of program sustainability published up until 2011, Stirman et al. [11] identified gaps in the application of definitions of sustainability when developing and implementing programs. Studies included in their review seldom reported definitions in sufficient detail to be able to assess sustainability in a systematic manner [11]. A recent systematic integrative review [1] and scoping review [2] also found gaps and inconsistencies around the definitions of program sustainability, with less than 30% of studies providing the definitions in both reviews. Sustainability was often discussed as an extension of implementation, with many studies reporting that at the end of a 1-to-2-year implementation project, the program “was sustainable” without providing long-term outcomes or specific measures of sustainability [1].

Rationale

The factors that act as barriers/facilitators for the sustainability of healthcare programs are inadequately reported and are poorly understood. The determinants of successful implementation are often reported; however, these are likely to be quite different to the factors related to sustainability [10, 13]. For example, factors such as trialability, intervention fidelity and factors associated with the inner setting are often talked about with reference to successful implementation. Different factors are more likely to be important for health program sustainability and program scaling, including outer setting factors such as socio-political and funding environment, external leadership, and values, needs and priorities of communities and populations, data and evaluation to demonstrate value and to support adaptations as contexts change [13]. Stirman and colleagues [11] identified gaps in research on public health program adaptations and factors that drive sustainability such as organisational context and capacity, processes and characteristics of implemented programs. A deeper understanding of barriers to, and facilitators of, program sustainability is essential to support the development, implementation, and evaluation of innovative healthcare programs, to support decision-making around program continuation, adaptation, scale-up, and diffusion, and to maximise the long-term benefits of programs. It is similarly important to understand the encountered barriers and contexts that lead to program discontinuation. To develop this understanding, a review and synthesis of current knowledge, guided by a theoretical framework such as the ISF [12] is needed.

Objectives

Our primary objective was to build on the studies of Stirman et al. [11] and Braithwaite et al. [1] to identify the barriers and facilitators associated with the sustainability of implemented and evaluated improvement programs in healthcare delivery systems, and then to map them to the ISF. Our study also aimed to discern the extent to which the discontinuation of healthcare programs was reported in the literature and to identify factors that led to these programs being discontinued.

Methods

The review forms part of a body of research investigating the sustainability of healthcare programs, seeking to bring it up to date [1]. The search strategy, study selection and quality assessment mirror those outlined in a published integrative review on this topic [1], with the present updated review conducted in June 2022. The data relating to barriers and facilitators, the analysis and synthesis of data, the results, and the implications and conclusions drawn from these findings are unique to the current review. The review was guided by the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) statement (Table S1) [14].

Protocol and registration

The published protocol for this review can be found at https://bmjopen.bmj.com/content/7/11/e018568 [15]. Modifications to the protocol have been previously published and the details of the updated search are described here [1].

Search strategy

The search strategy was developed in consultation with two medical librarians and included six academic databases: CINAHL, EMBASE, Ovid MEDLINE, Emerald Management, Scopus and Web of Science [1]. Additional studies were identified by hand searching reference lists of relevant systematic reviews. The search strategies for all databases are provided in Table S2.

Study selection

The selection process has been previously described [1]. The reviewers had varying degrees of experience in conducting systematic reviews, and 13 out of the 16 authors had previously been an author on at least one systematic review study. A blinded review of 5% of titles and abstracts was undertaken, and discrepancies were discussed among the reviewers (KL, LT, HA, JHD, GL, EM, KH, AC, CLS, LVB, LAE, and GD), with two reviewers (YZ and JB) acting as arbitrators, until a consensus was reached. The remaining screening of abstracts and titles was undertaken in Rayyan [16], a web and mobile app for systematic reviews, according to the inclusion criteria with records randomly allocated among the reviewers [1]. Publications were assessed against the following inclusion criteria: (1) English-language, (2) peer-reviewed journal article, (3) primary empirical research, (4) published July 2011–June 2022, (5) healthcare setting, (6) evaluation of program, (7) assessment of program sustainability, and (8) focus on changes/improvements to the healthcare system.

Studies reporting on public health programs including population-based prevention programs, community-level outcomes only, or patient-level outcomes only were excluded. Studies that did not identify barriers to, or facilitators of, program sustainability, and studies that reported barriers or facilitators of implementation only were also excluded. Studies included at the abstract-review stage were re-assessed against the inclusion criteria during full-text review. Based on our previous integrative review [1] of 92 studies and drawing on the work of Stirman et al. [11], Shelton et al. [12], and Scheirer and Dearing [10], health program sustainability was conceptualised from a systems or organisational view-point. Therefore, studies were included if they reported on the following:

  1. a)

    Evaluation of a program after funding had ended, or after the initial staff training or implementation phase; and

  2. b)

    Explicitly assessed sustainability, either using qualitative, quantitative, or mixed methods, for example, stakeholders’ views of sustainability, evidence of ongoing care delivery under the program, ongoing funding; or,

  3. c)

    Longitudinal studies, for example, evaluations conducted over multiple time points.

Data collection processes and data items

Following previously-described work [1], a purpose-designed Excel spreadsheet was utilised for data extraction. The spreadsheet was piloted by reviewers on two studies. The remaining studies were randomly assigned to the reviewers for data extraction. Verification of the accuracy and meanings of the extracted data was undertaken independently by seven reviewers (LT, AC, PNAD, CLS, NH, and YZ). Any discrepancies were resolved through team discussions during regular meetings (over 10 group meetings were held).

Data analysis and synthesis

It was often not possible to classify factors influencing sustainability in a binary way, i.e., as either a barrier or facilitator. For example, the degree to which a program was sustained may have been influenced by a high (facilitator) or low (barrier) level of leadership. As such, barriers and facilitators were conceptualised as part of a single construct, representing two ends of a spectrum.

Barriers/facilitators of program sustainability were synthesised using the ISF [12]. The ISF embodies 36 “emerging factors” grouped together under five categories: outer setting, inner setting, intervention characteristics, processes, and implementer and population characteristics. Shelton et al. [12] did not provide definitions for each of their emerging factors making it difficult for the reviewers to classify some of the barriers/facilitators. To overcome this challenge, working definitions for the ISF emerging factors were developed by two reviewers (LT and HA) based on relevant literature and other frameworks including Weiner et al. [17] and the Consolidated Framework for Implementation Research (CFIR) [18]. The proposed definitions were discussed with the broader review team before being applied during data analysis and interpretation (Table 1). Frequency counts were then performed for each emerging factor (Table 2), in addition to a qualitative narrative synthesis. For the purpose of this review, emerging factors will henceforth be referred to as “barriers/facilitators”.

Table 1 Definitions/operationalisations of emerging factors in the Integrated Sustainability Framework
Full size table
Table 2 Number of included papers reporting on barriers to and/or facilitators of program sustainabilitya, organised according to the Integrated Sustainability Framework [12]
Full size table

To identify critical barriers to program sustainability and how these might be overcome, one part of our analysis focused on programs that were discontinued. To identify critical facilitators of longer-term sustainability, the review also focussed on reports of programs that were sustained for at least 5 years after funding, training, or the implementation period ended, rather than the more commonly reported time points of 1 to 3 years at the end of trial funding when it is difficult to separate factors related with implementation from those related with sustainability [1]. There are currently no specific agreed or pre-determined time points at which a program is deemed to be sustainable. Thus, informed by the literature, especially published reviews [1, 11, 12, 20, 21] and, after team discussions, we concentrated on programs that had been sustained for 5 years or longer after funding, staff training or the implementation period or the trial had ended.

Quality assessment

The quality of included studies was assessed by ten reviewers (KL, LT, HA, JHD, GL, AC, PNAD, CLS, GD, and NS) using Hawker’s Quality Assessment tool [22] and Lorenc et al.’s quality ratings [23] (low, medium, high). A blinded quality assessment of a randomly selected 6% (n = 7) sample of included studies was conducted to ensure consistency of ratings among the reviewers. The remaining studies were randomly assigned to individual reviewers and any queries were discussed and resolved in a team meeting.

Results

Study selection

A total of 11,443 studies were screened after duplicates were removed. At the title/abstract review stage, 10,845 out of the 11,443 were excluded, leaving 598 studies progressing to full-text review, with 124 studies being retained for data analysis and synthesis (Fig. 1). The main reasons for exclusion at full-text review were that the publication did not assess sustainability or discontinuation of a program (n = 185), did not focus on change improvements in the healthcare system (n = 91), or no evaluation of a program was reported (n = 102).

Fig. 1
figure 1

PRISMA flowchart of studies identified for relevant studies for inclusion in the review

Full size image

Study characteristics and quality assessment

Fifty-two studies (41.9%) reported qualitative results, 37 (29.8%) used mixed methods and 35 (28.2%) reported quantitative results. Most studies were longitudinal with assessment of outcomes at different time points (n = 55; 44.4%), in addition to case studies (n = 32; 25.8%), and cross-sectional studies (n = 30; 24.2%). Thirty-five countries were covered by the 124 studies, with seven including more than one country. The majority of studies originated from the USA (n = 43; 34.7%), Canada (n = 14; 11.3%), the United Kingdom (n = 11; 8.9%), and Australia (n = 11; 8.9%). Eighty-four studies scored 30–36 points (high quality), 35 scored 24–29 points (moderate quality), and 5 scored less than 24 points and were considered low quality on Hawker’s Quality Assessment Tool [22, 23] (Table S3). No studies were excluded as all studies were deemed as providing sufficient information related to sustainability.

Barriers and facilitators

Table 2 summarises the number of studies that explicitly referred to the barriers/facilitators under the ISF as defined in Table 1. Detailed data for each barrier/facilitator including frequency counts are provided in Table S4. The most commonly identified barriers/facilitators were related to the ISF inner setting category (n = 99; 79.8%) and the processes category (n = 99; 79.8%), (Table 2).

Inner setting

Barriers/facilitators related to the inner setting were reported by 99 (79.8%) studies (Table 2). Organisational factors commonly identified by studies as influencing intervention sustainability were leadership (n = 61; 61.6%), staffing/turnover (n = 50; 50.5%), climate/culture (n = 42; 42.4%), and funding (n = 37; 37.4%). Committed leadership, especially from formally appointed leaders, was reported to play a critical role in sustaining interventions [24, 25]. Brewster et al. [26] described the presence of a small number of key staff members to maintain an intervention in place and demonstrable commitment by management over time as important facilitators. Staff turnover was often cited as a barrier to continuation of interventions, which related to the need to re-train new staff whilst dealing with staff shortages to deliver the program [25, 27,28,29].

The climate/culture of the inner setting were also commonly discussed, as these factors created a supportive environment for change implementation. Supportive work culture [30] and providing rewards and recognition [27, 31] were recognised facilitators. Conversely, factors including lack of a unified identity and poor accountability [32], and staff and institutional resistance [33, 34] acted as barriers to sustainability. Availability of funding facilitated the delivery of an intervention, independent of the cost of the intervention itself, however, inadequate resources to support and expand interventions presented a barrier to continuation [35].

Processes

The processes related to program sustainability were reported by 99 (79.8%) studies and included the availability of training/supervision (n = 54; 54.5%), program evaluations and data (n = 40; 40.4%), and communication (n = 40; 40.4%) (Table 2). The provision of regular staff training about new programs, for both newly hired staff and experienced staff members, was an identified facilitator for program sustainability [25, 35,36,37] whereas a lack of training was identified as a barrier [25, 35, 38]. The importance of program evaluation and regular feedback of data concerning program outcomes to staff members involved in the implementation and to stakeholders, was highlighted by at least seven studies [34, 39,40,41,42,43,44]. Positive program outcomes that were regularly communicated and visible to staff members were linked with program sustainability [37, 45,46,47] and the converse hindered program sustainability [48, 49].

Efficient and ongoing communication among stakeholders involved in the program [24, 31, 50] and strong collaborative partnerships facilitated program sustainability [51,52,53]. Clear roles and responsibilities as well as mutual trust among stakeholders to fulfil their unique responsibilities were also reported as facilitators [50, 54]. Decreased communication among stakeholders after program implementation was a recognised barrier, for example, limited communication after implementation of a multidisciplinary hospital‐based surgical program threatened sustainability in some locations [55].

Intervention characteristics

Intervention characteristics were reported in 72 studies (58.1%), with a good “fit” and alignment of interventions with existing systems and local contexts facilitating sustainability reported in 37 studies (51.4%). For example, facility-based consultations in eye care interventions in Ghana were more likely to be routinised due to a high level of compatibility with the hospitals’ mandate, whereas outreach activities were less likely to be sustained due to a low level of compatibility and lack of role clarity [56]. Intervention adaptations to overcome challenges in resource-limited settings bolstered sustainability by improving fit with the population’s needs and context. For example, across Uganda, shifting intervention delivery from physicians to other staff (nurses and pharmacists) in an anti-retroviral therapy program and adopting greater task sharing with non-physician staff, supported program sustainability [57]. Urquhart et al. [58] also reported the importance of adapting interventions to improve fit with cancer survivors’ needs in different settings, including transitioning to online delivery and tailoring of tools.

Furthermore complex interventions were less sustainable [24, 59], while simpler interventions were reported to be more sustainable [26, 53, 60].

Outer setting

Factors related to the outer setting category were reported by 46 (37.1%) studies, with funding (n = 26; 56.5%), external leadership by stakeholders (n = 16; 34.8%) and socio-political context (n = 14; 30.4%) reported most frequently (Table 2). Support and leadership from external stakeholders facilitated intervention sustainability [35, 61, 62]. External contextual factors, such as funding environment and socio-political context, were found to both positively and negatively influence intervention sustainability. For example, Bond et al. [35] found that adequate financing facilitated sustainability, whereas Fleizer et al. [32] and Olumide et al. [29] reported that insecure sources of funding challenged the continuation of programs. A high dependence on time-limited funding from external donors created a barrier to sustainability, especially when other funding sources were not planned for in the longer term [62, 63].

Ongoing involvement of leadership in the region [49] and at national level [61] were thought to facilitate sustainability while a lack of support from governments created barriers to program sustainability [38, 64]. Furthermore, the socio-political context was reported as an influencing factor by 14 studies. For example, De Neve et al. [64] found that political turnovers and instability led to discontinuation of a program as ‘political actors’ and priorities changed. A mismatch between the program activities and values, priorities and needs were also identified as a barrier [64,65,66]. Socio-political factors were also reported as important facilitators. For example, political and financial stability and perceived value of the implemented programs among external stakeholders were thought to support the sustainability of HIV/AIDS relief programs [67]. In addition, joint planning between the donor, non-governmental organisations, health facilities and government enabled stakeholders, especially local governments, develop a better understand their health system needs and therefore to sustain effective health investments [67]

Implementer and population characteristics

Factors related to implementer and population characteristics were reported by 44 studies (35.5%), and half of them reported that general attitudes of implementers of new programs (n = 23; 52.3%) both positively and negatively affected sustainability (Table 2). Implementers being realistic in their expectations, including realistic timelines, adequate resourcing, ongoing engagement with staff delivering the program, and with program recipients, facilitated sustainability [65]. Staff members’ beliefs about the advantages of programs facilitated sustainability [47, 55, 68], whereas negative attitudes and fear of change were barriers [37, 69, 70]. Staff members who perceived that a new program would have negative consequences for their autonomy and workload was identified as a barrier [27, 69].

The right skills and level of expertise were also identified as implementer characteristics that positively influenced the delivery of sustained programs. For example, a rural volunteer program in Canada underlined that volunteer coordinators with sufficient skills and expertise, who also trained and mentored others, was an important facilitator [36]. However, Fox et al. [48] found that underuse of highly experienced and skilled staff might lead to job dissatisfaction and staff attrition, posing a barrier. For example, emergency nurses were concerned about deskilling and underutilisation after acquiring new high-level skills which were not required to care for low acuity patients [48].

Identifying barriers that resulted in program discontinuation, and facilitators of long-term program sustainability

Eight of the 124 studies (6.5%) explicitly referred to the discontinuation of programs [24, 29, 35, 71,72,73,74,75]. Table 3 summarises factors associated with discontinuation, including lack of financial viability; workforce issues (lack of trained workforce, strict role boundaries, competing demands on staff time, poor preparation, training and planning); and lack of engagement and misunderstandings between implementers and staff expected to deliver the program. Misalignment with existing policies and workflows, lack of ongoing support from the implementation team and multiple changes being implemented at the same time also contributed to program discontinuation (Table 3).

Table 3 Summary of studies reporting on programs that were discontinued, illustrating common barriers of sustainability (n = 8)
Full size table

A total of 29 studies (23.4%) reported that the programs had been sustained for 5 years or longer (Table S5). Five example studies reporting on programs that were sustained for 5 years or more are summarised in Table 4. These five example programs were selected because they demonstrate a wide variety of factors that supported program sustainability. Eight of the 29 studies (27.6%) reported that program adaptability and/or adaptation were the key facilitators of long-term program sustainability. For example, most health facilities implementing a multi-site anti-retroviral therapy (ART) scale-up program in Uganda modified and tailored the intervention in order to improve fit with their resource-constrained conditions thereby fostering long-term sustainability between 2004 and 2014 [57]. Another study by Oliveira et al. [76] highlighted that ongoing monitoring and adaptation of the Family Health Program (FHP) in response to critical events were deemed as strategic facilitating factors for the sustainability of the program for 12 years.

Table 4 Five examples describing programs sustained for 5 years or more and illustrating common facilitators of sustainabilitya
Full size table

Multi-site studies demonstrated the importance of understanding the local contexts and several studies reported that the programs were sustained in one context but not in another. Vidgen et al. [24] demonstrated such contextual differences by highlighting that the decision to outsource a program to an external provider under a limited time contract was a barrier to sustainability. On the other hand, Zakumumpa et al. [25, 75] aptly demonstrated both sustainability and discontinuation in different sites to show significant barriers (Table 3) and facilitators (Table 4) related to local context.

Discussion

Our systematic integrative review demonstrated that the literature on the sustainability of innovations or improvement programs in healthcare is developing. Barriers and facilitators of healthcare program sustainability were identified and mapped to the ISF using our working definitions, with the most prevalent barriers/facilitators relating to inner setting (79.8%) and processes (79.8%). The review identified important gaps including limited long-term program evaluations. Studies often claimed program sustainability even at 1 or 2 years after implementation. Longer-term evaluations are needed to confirm such claims as few studies (n = 29; 23.4%) reported on program sustainability 5 years or more after implementation. Short-term evaluations were common which is not surprising given the approach often taken by health organisations and governments when implementing improvement programs in the healthcare system [79]. Short-term funding limits capacity to rigorously evaluate, adapt, sustain and scale programs over the longer-term.

Leadership and support emerged as a key influencer in program sustainability. Consistent with the findings from the systematic reviews by Cowie et al. [80] and Penno et al. [81] using Lennox’s consolidated framework [82], our review suggests that the support of leaders plays a critical role in achieving sustained programs. Enthusiasm and support of leaders, however, is not enough to effectively support and sustain healthcare programs without considerable skill, expertise, and capacity of these leaders. Ambitious leadership without sufficient managerial skills and technical experience can negatively impact sustainability of healthcare programs due to a loss of focus on the program after implementation [83].

Workforce issues, such as high staff turnover, were identified as common barriers to sustainability. This finding is consistent with other reviews [21, 80]. Our review identified that program discontinuation could be attributed to staff turnover associated with lack of adequate training and trained staff, lack of incentives and recognition, and competing priorities [29, 75]. In contrast, Shelton et al. [12] reported that the nature and influence of processes, including staff training, were barriers/facilitators less often reported to be associated with sustainability. Ninety-nine studies (79.8%) included in our review reported at least one factor under the processes category, with more than half mentioning training/support/supervision. The role of training was essential to equip staff with skills and knowledge required to deliver program interventions and to maintain fidelity [34, 62]. Ensuring adequate time and resources to train staff as required, and not only at the beginning of implementation, should be considered in planning new programs that are intended to last. New programs require new roles and new role descriptions, which should be developed, maintained, and updated to ensure role clarity, responsibility, and scope within the program and the context within which the program is being sustained.

Much of the literature about healthcare system sustainability is focussed on factors that make programs and systems last over time. This, of course, is sensible, however it provides a one-sided view. One of the unique aspects of our review is the analysis of factors that led to the discontinuation of programs. Understanding why programs cease is a critical complement to our understanding of factors that make programs sustainable. Increasingly, it is being recognised that many programs may continue despite becoming ineffective, inefficient, or no longer needed [84]. Therefore, the strident quest for sustainability of all implemented programs may be inappropriate and may in itself contribute to the wider system unsustainability, as maintaining ineffective, inefficient or defunct health programs can contribute to waste or low-value care [84, 85].

A greater emphasis in the literature on reporting on ineffective or unsustainable programs would enrich our understanding of factors associated with program sustainability and may also prevent others from wasting efforts and investment. However, there were only eight studies out of the 124 included in our review (6.5%) that reported on discontinuation. Publication bias, where negative results are less likely to be published, may be a factor skewing the literature towards successful ongoing programs [86]. This limited literature restricts our understanding of the factors that lead to discontinuation, or how the decision to discontinue was influenced, made, or planned. All eight studies describing discontinued programs, or parts of programs, argued that their programs should have been sustained; however, scaling-up healthcare programs must be accompanied by appropriate long-term monitoring and ongoing evaluation to ensure that decisions to sustain, adapt or discontinue are evidence-based [84].

In line with our review, two other systematic reviews applied the ISF to assess barriers/facilitators related to program sustainability [20, 21]. One of these, by Shoesmith et al. [21] provided specific factor definitions as applied in the context of schools and/or childcare services. In line with the results of Shoesmith et al. [21] and Braithwaite et al. [1] our review also found that factors predominantly related to the inner setting were reported to facilitate intervention sustainment, including leadership support. Another review by Hall et al. [20] applied the factor definitions developed by Shoesmith et al. [21] in a review of clinical, public health, and community health services. Hall et al. [20] focused on the measurement of sustainability and determinants of sustainability, and found that 28 individual measures were used among 223 articles, but only 2 of these measures specifically assessed sustainability as an outcome and one assessed both sustainability outcomes and determinants. This is an important gap also illustrated in our previous review [1], where only 6 out of 92 included studies (6.5%) reported using measures of program sustainability in healthcare delivery settings. This demonstrates a need for clear and consistent definitions and measures of sustainability that are relevant and applicable to the field being studied, in our case, healthcare. Whilst undertaking this review, it became necessary to develop working definitions guided by the literature that could be applied by the review team to operationalise the ISF and ensure consistency of interpretation of emerging factors associated with program sustainability. These new definitions should be considered and improved upon as new research emerges, to improve the consistency of assessments using the ISF.

Strengths and limitations

Our review builds on our previous work [1] and extends the work of others [2, 11, 12] by describing in detail the factors contributing to program sustainability in healthcare settings, whilst guided by a published framework, the ISF. This review enhances evidence about the sustainability of healthcare programs by identifying and mapping barriers to and/or facilitators of a sustainability framework that support sustainability or contribute to program discontinuation. The analysis of factors that led to the discontinuation of programs is also a strength and provides important learnings to guide future healthcare program planning to avoid known barriers. Few reviews have specifically addressed program discontinuation, possibly because there are fewer publications about discontinued programs, potentially due to the known publication bias to publish positive results [86].

The application of the ISF as an underpinning theory to map barriers and facilitators of sustainability is another strength of our review, which builds on this framework by providing a working definition of each emerging factor. Importantly, we conceptualised the factors on a continuum rather than binary facilitators or barriers, for example policy and legislation may be a barrier in some settings and a facilitator in others and this may change over time. Although working definitions of emerging factors were developed for this review, they have not been applied by others to determine their validity, consistency, and applicability to the study of sustainability of healthcare programs. Further work is needed to ensure the longer-term usability or adaptation of these definitions.

Double-blinded abstract and full-text reviews were conducted in addition to many team meetings to ensure consistency of study screening, inclusion and interpretation. Interpretation of how studies conceptualised sustainability required significant discussion by the whole team especially in studies that reported on sustainability at 12 months or less after implementation—where implementation and sustainability may have been conflated. Therefore, the decision to concentrate on programs that were sustained for 5 years or more adds further validity to our findings.

The large variety of definitions of sustainability, and the failure to provide definitions of sustainability for studies that report on sustainability, has been continuously highlighted as a major limitation in previous reviews on sustainability [12, 21, 80], including in our previous reviews [1, 2]. Reviewing the literature on healthcare program sustainability was made more challenging by the large body of literature on environmental sustainability, limiting the pace of screening for inclusion and exclusion. Moreover, the heterogeneous nature of the current included studies, including lack of clear definitions or inconsistent definitions, made synthesis of the literature challenging. The exclusion of grey literature, studies published in languages other than English may mean that other relevant studies could have been missed, limiting the comprehensiveness of the evidence synthesis.

Implication for practice and research

Our findings add to the understanding of which factors hampered or facilitated the sustainment of healthcare programs and complements previous reviews on program sustainability. The sustainability of healthcare system improvements in our review mapped to the inner setting and processes category of the ISF with leadership/support, training/support/supervision, and staffing/turnover being the most frequently reported barriers/facilitators. In line with other reviews [80, 81], the results suggest that these barriers/facilitators should be prioritised in the sustainability phase of programs and considered in light of organisational readiness and ongoing resources for program delivery. However, it should be noted that this suggestion is based on the findings from the 124 studies in our review, which were mainly located in high-income countries.

Aligned with the literature on program implementation, factors related to the inner setting and processes were commonly reported in studies included in our review. However, the outer contextual factors such as the socio-political context, funding environment, external leadership, and values, needs and priorities of stakeholders and populations were addressed in over a third of the selected studies. Furthermore, the importance of process factors that to some extent overlap with those related to the outer setting, such as partnerships and engagement with stakeholders, effective communication with stakeholders and evaluations and data were discussed, especially in studies reporting on programs that were sustained for more than 5 years. As implementation and sustainability are on a continuum, groups designing and planning health programs should consider these outer setting and process factors that have been reported to impact health program sustainability.

Despite rapidly growing literature about healthcare program sustainability, there remains a lack of conceptual clarity in defining and assessing sustainability. Furthermore, frameworks such as the Consolidated Framework for Implementation Research [18] and RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) [87] have been applied predominantly in the context of program implementation rather than sustainability. Applying these frameworks can be challenging when discerning the barriers/facilitators contributing to implementation as opposed to sustainability. Hall et al. [20] recommended the careful consideration of measures of determinants of sustainability that align with the construct of interest, such as objective and settings, to ensure robustness and relevancy of the program evaluation to sustainability. Moreover, it is important for future studies evaluating program sustainability to provide operational definitions of sustainability and clear evaluation timeframes as well as being explicit about theoretical frameworks to underpin their work.

Conclusions

Strong leadership and stakeholder engagement, supportive organisational culture/climate, intervention fit with context and policy, intervention simplicity, adaptability, and fit with need and context, were all important factors in program sustainability. Adequate resourcing including ongoing availability of funding, training for staff, and low staff turnover, as well as constrained intervention costs, and alignment with organisational or broader policy or strategy were also strongly associated with program sustainability. Our review identifies the need for greater use of clear definitions of program sustainability and the application of validated frameworks in future research in this field. To that end, this review provided a working definition of each factor in the ISF to ensure consistency in defining barriers/facilitators associated with sustainability. Furthermore, a greater understanding of the factors associated with discontinuation of healthcare programs is needed, and this can only occur if negative outcomes are published to address the likely publication bias towards positive findings.