Background

Over the past few decades, the field of public health implementation research (IR) has grown as a means by which the real-world conditions affecting health improvement efforts can be better understood. Peters et al. put forward the following broad definition of IR for health: “IR is the scientific inquiry into questions concerning implementation – the act of carrying an intention into effect, which in health research can be policies, programmes, or individual practices (collectively called interventions)” [1].

As IR emphasizes real-world circumstances, the context within which a health intervention is delivered is a core consideration. However, much IR implemented to date has focused on higher-resource settings, with many proposed frameworks developed with particular utility for a higher-income setting [2]. In recognition of IR’s potential to increase evidence across a range of settings, there have been numerous reviews of the use of IR in lower-resource settings as well as calls for broader use [3, 4]. There have also been more focused efforts to modify various approaches and frameworks to strengthen the relevance of IR to low- and middle-income country settings (LMICs), such as the work by Means et al. to adapt a specific IR framework for increased utility in LMICs [2].

Within LMIC settings, the centrality of context to a health intervention’s impact is of particular relevance in humanitarian settings, which present a set of distinct implementation challenges [5]. Humanitarian responses to crisis situations operate with limited resources, under potential security concerns, and often under pressure to relieve acute suffering and need [6]. Given these factors, successful implementation of a particular health intervention may require different qualities than those that optimize intervention impact under more stable circumstances [7]. Despite increasing recognition of the need for expanded evidence of health interventions in humanitarian settings, the evidence base remains limited [8]. Furthermore, despite its potential utility, there is not standardized guidance on IR in humanitarian settings, nor are there widely endorsed recommendations for the frameworks best suited to analyze implementation in these settings.

Sexual and reproductive health (SRH) is a core aspect of the health sector response in humanitarian settings [9]. Yet, progress in addressing SRH needs has lagged far behind other services because of challenges related to culture and ideology, financing constraints, lack of data and competing priorities [10]. The Minimum Initial Service Package (MISP) for SRH in Crisis Situations is the international standard for the minimum set of SRH services that should be implemented in all crisis situations [11]. However, as in other areas of health, there is need for expanded evidence for planning and implementation of SRH interventions in humanitarian settings. Recent systematic reviews of SRH in humanitarian settings have focused on the effectiveness of interventions and service delivery strategies, as well as factors affecting utilization, but have not detailed whether IR frameworks were used [12,13,14,15]. There have also been recent reviews examining IR frameworks used in various settings and research areas, but none have explicitly focused on humanitarian settings [2, 16].

Given the need for an expanded evidence base for SRH interventions in humanitarian settings and the potential for IR to be used to expand the available evidence, a scoping review was undertaken. This scoping review sought to identify IR approaches that have been used in the last ten years to evaluate SRH interventions in humanitarian settings.

This review also sought to shed light on whether there is a need for a common framework to guide research design, analysis, and reporting for SRH interventions in humanitarian settings and if so, if there are any established frameworks already in use that would be fit-for-purpose or could be tailored to meet this need.

Methods

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping reviews was utilized to guide the elements of this review [17]. The review protocol was retrospectively registered with the Open Science Framework (https://osf.io/b5qtz).

Search strategy

A two-fold search strategy was undertaken for this review, which covered the last 10 years (2013–2022). First, recent systematic reviews pertaining to research or evaluation of SRH interventions in humanitarian settings were identified through keyword searches on PubMed and Google Scholar. Four relevant systematic reviews were identified [12,13,14,15] Table 1.

Table 1 Relevant systematic reviews
Full size table

Second, a literature search mirroring these reviews was conducted to identify relevant papers published since the completion of searches for the most recent review (April 2017). Additional file 1 includes the search terms that were used in the literature search [see Additional file 1].

The literature search was conducted for papers published from April 2017 to December 2022 in the databases that were searched in one or more of the systematic reviews: PubMed, Embase, PsycInfo, CINAHL and Global Health. Searches were completed in January 2023 Table 2.

Table 2 Inclusion and exclusion criteria
Full size table

Screening

Two reviewers screened each identified study for alignment with inclusion criteria. Studies in the four systematic reviews identified were considered potentially eligible if published during the last 10 years. These papers then underwent full-text review to confirm satisfaction of all inclusion criteria, as inclusion criteria were similar but not fully aligned across the four reviews.

Literature search results were exported into a citation manager (Covidence), duplicates were removed, and a step-wise screening process for inclusion was applied. First, all papers underwent title and abstract screening. The remaining papers after abstract screening then underwent full-text review to confirm satisfaction of all inclusion criteria. Title and abstract screening as well as full-text review was conducted independently by both authors; disagreements after full-text review were resolved by consensus.

Data extraction and synthesis

The following content areas were summarized in Microsoft Excel for each paper that met inclusion criteria: publication details including author, year, country, setting [rural, urban, camp, settlement], population [refugees, internally displaced persons, general crisis-affected], crisis type [armed conflict, natural disaster], crisis stage [acute, chronic], study design, research methods, SRH intervention, and intervention target population [specific beneficiaries of the intervention within the broader population]; the use of an IR framework; details regarding the IR framework, how it was used, and any rationale given for the framework used; factors cited as impacting SRH interventions, either positively or negatively; and other key findings deemed relevant to this review.

As the focus of this review was on the approach taken for SRH intervention research and evaluation, the quality of the studies themselves was not assessed.

Results

Twenty papers underwent full-text review due to their inclusion in one or more of the four systematic reviews and meeting publication date inclusion criteria. The literature search identified 7,016 unique papers. After full-text screening, 69 met all inclusion criteria and were included in the review. Figure 1 illustrates the search strategy and screening process.

Fig. 1
figure 1

Flow chart of paper identification

Full size image

Papers published in each of the 10 years of the review timeframe (2013–2022) were included. 29% of the papers originated from the first five years of the time frame considered for this review, with the remaining 71% papers coming from the second half. Characteristics of included publications, including geographic location, type of humanitarian crisis, and type of SRH intervention, are presented in Table 3.

Table 3 Characteristics of included papers (n = 69)
Full size table

A wide range of study designs and methods were used across the papers, with both qualitative and quantitative studies well represented. Twenty-six papers were quantitative evaluations [18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43], 17 were qualitative [44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60], and 26 used mixed methods [61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86]. Within the quantitative evaluations, 15 were observational, while five were quasi-experimental, five were randomized controlled trials, and one was an economic evaluation. Study designs as classified by the authors of this review are summarized in Table 4.

Table 4 Study design and methods
Full size table

Six papers (9%) explicitly cited use of an IR framework. Three of these papers utilized the Consolidated Framework for Implementation Research (CFIR) [51, 65, 70]. The CFIR is a commonly used determinant framework that—in its originally proposed form in 2009—is comprised of five domains, each of which has constructs to further categorize factors that impact implementation. The CFIR domains were identified as core content areas influencing the effectiveness of implementation, and the constructs within each domain are intended to provide a range of options for researchers to select from to “guide diagnostic assessments of implementation context, evaluate implementation progress, and help explain findings.” [87] To allow for consistent terminology throughout this review, the original 2009 CFIR domains and constructs are used.

Guan et al. conducted a mixed methods study to assess the feasibility and effectiveness of a neonatal hepatitis B immunization program in a conflict-affected rural region of Myanmar. Guan et al. report mapping data onto the CFIR as a secondary analysis step. They describe that “CFIR was used as a comprehensive meta-theoretical framework to examine the implementation of the Hepatitis B Virus vaccination program,” and implementation themes from multiple study data sources (interviews, observations, examination of monitoring materials) were mapped onto CFIR constructs. They report their results in two phases – Pre-implementation training and community education, and Implementation – with both anchored in themes that they had mapped onto CFIR domains and constructs. All but six constructs were included in their analysis, with a majority summarized in a table and key themes explored further in the narrative text. They specify that most concerns were identified within the Outer Setting and Process domains, while elements identified within the Inner Setting domain provided strength to the intervention and helped mitigate against barriers [70].

Sarker et al. conducted a qualitative study to assess provision of maternal, newborn and child health services to Rohingya refugees residing in camps in Cox’s Bazar, Bangladesh. They cite using CFIR as a guide for thematic analysis, applying it after a process of inductive and deductive coding to index these codes into the CFIR domains. They utilized three of the five CFIR domains (Outer Setting, Inner Setting, and Process), stating that the remaining two domains (Intervention Characteristics and Characteristics of Individuals) were not relevant to their analysis. They then proposed two additional CFIR domains, Context and Security, for use in humanitarian contexts. In contrast to Guan et al., CFIR constructs are not used nor mentioned by Sarker et al., with content under each domain instead synthesized as challenges and potential solutions. Regarding the CFIR, Sarker et al. write, “The CFIR guided us for interpretative coding and creating the challenges and possible solutions into groups for further clarification of the issues related to program delivery in a humanitarian crisis setting.” [51]

Sami et al. conducted a mixed methods case study to assess the implementation of a package of neonatal interventions at health facilities within refugee and internally displaced persons camps in South Sudan. They reference use of the CFIR earlier in the study than Sarker et al., basing their guides for semi-structured focus group discussions on the CFIR framework. They similarly reference a general use of the CFIR framework as they conducted thematic analysis. Constructs are referenced once, but they do not specify whether their application of the CFIR framework included use of domains, constructs, or both. This may be in part because they then applied an additional framework, the World Health Organization (WHO) Health System Framework, to present their findings. They describe a nested approach to their use of these frameworks: “Exploring these [CFIR] constructs within the WHO Health Systems Framework can identify specific entry points to improve the implementation of newborn interventions at critical health system building blocks.” [65]

Three papers cite use of different IR frameworks. Bolan et al. utilized the Theoretical Domains Framework in their mixed methods feasibility study and pilot cluster randomized trial evaluating pilot use of the Safe Delivery App by maternal and newborn health workers providing basic emergency obstetric and newborn care in facilities in the conflict-affected Maniema province of the Democratic Republic of the Congo (DRC). They used the Theroetical Domains Framework in designing interview questions, and further used it as the coding framework for their analysis. Similar to the CFIR, the Theoretical Domains Framework is a determinant framework that consists of domains, each of which then includes constructs. Bolan et al. utilized the Theoretical Domains Framework at the construct level in interview question development and at the domain level in their analysis, mapping interview responses to eight of the 14 domains [83]. Berg et al. report using an “exploratory design guided by the principles of an evaluation framework” developed by the Medical Research Council to analyze the implementation process, mechanisms of impact, and outcomes of a three-pillar training intervention to improve maternal and neonatal healthcare in the conflict-affected South Kivu province of the DRC [67, 88]. Select components of this evaluation framework were used to guide deductive analysis of focus group discussions and in-depth interviews [67]. In their study of health workers’ knowledge and attitudes toward newborn health interventions in South Sudan, before and after training and supply provision, Sami et al. report use of the Conceptual Framework of the Role of Attitudes in Evidence-Based Practice Implementation in their analysis process. The framework was used to group codes following initial inductive coding analysis of in-depth interviews [72].

Three other papers cite use of specific frameworks in their intervention evaluation [19, 44, 76]. As a characteristic of IR is the use of an explicit framework to guide the research, the use of the frameworks in these three papers meets the intention of IR and serves the purpose that an IR framework would have in strengthening the analytical rigor. Castle et al. cite use of their program’s theory of change as a framework for a mixed methods evaluation of the provision of family planning services and more specifically uptake of long-acting reversible contraception use in the DRC. They describe use of the theory of change to “enhance effectiveness of [long-acting reversible contraception] access and uptake.” [76] Thommesen et al. cite use of the AAAQ (Availability, Accessibility, Acceptability and Quality) framework in their qualitative study assessing midwifery services provided to pregnant women in Afghanistan. This framework is focused on the “underlying elements needed for attainment of optimum standard of health care,” but the authors used it in this paper to evaluate facilitators and barriers to women accessing midwifery services [44]. Jarrett et al. cite use of the Centers for Disease Control and Prevention’s (CDC) Guidelines for Evaluating Public Health Surveillance Systems to explore the characteristics of a population mobility, mortality and birth surveillance system in South Kivu, DRC. Use of these CDC guidelines is cited as one of four study objectives, and commentary is included in the Results section pertaining to each criteria within these guidelines, although more detail regarding use of these guidelines or the authors’ experience with their use in the study is not provided [19].

Overall, 22 of the 69 papers either explicitly or implicitly identified IR as relevant to their work. Nineteen papers include a focus on feasibility (seven of which did not otherwise identify the importance of exploring questions concerning implementation), touching on a common outcome of interest in implementation research [89].

While a majority of papers did not explicitly or implicitly use an IR framework to evaluate their SRH intervention of focus, most identified factors that facilitated implementation when they were present or served as a barrier when absent. Sixty cite factors that served as facilitators and 49 cite factors that served as barriers, with just three not citing either. Fifty-nine distinct factors were identified across the papers.

Three of the six studies that explicitly used an IR framework used the CFIR, and the CFIR is the only IR framework that was used by multiple studies. As previously mentioned, Means et al. put forth an adaptation of the CFIR to increase its relevance in LMIC settings, proposing a sixth domain (Characteristics of Systems) and 11 additional constructs [2]. Using the expanded domains and constructs as proposed by Means et al., the 59 factors cited by papers in this review were thematically grouped into the six domains: Characteristics of Systems, Outer Setting, Inner Setting, Characteristics of Individuals, Intervention Characteristics, and Process. Within each domain, alignment with CFIR constructs was assessed for, and alignment was found with 29 constructs: eight of Means et al.’s 11 constructs, and 21 of the 39 standard CFIR constructs. Three factors did not align with any construct (all fitting within the Outer Setting domain), and 14 aligned with a construct label but not the associated definition. Table 5 synthesizes the mapping of factors affecting SRH intervention implementation to CFIR domains and constructs, with the construct appearing in italics if it is considered to align with that factor by label but not by definition.

Table 5 Factors cited as impacting SRH interventionsa
Full size table

Table 6 lists the CFIR constructs that were not found to have alignment with any factor cited by the papers in this review.

Table 6 CFIR Constructs without factor alignment
Full size table

Discussion

This scoping review sought to assess how IR frameworks have been used to bolster the evidence base for SRH interventions in humanitarian settings, and it revealed that IR frameworks, or an explicit IR approach, are rarely used. All four of the systematic reviews identified with a focus on SRH in humanitarian settings articulate the need for more research examining the effectiveness of SRH interventions in humanitarian settings, with two specifically citing a need for implementation research/science [12, 13]. The distribution of papers across the timeframe included in this review does suggest that more research on SRH interventions for crisis-affected populations is taking place, as a majority of relevant papers were published in the second half of the review period. The papers included a wide range of methodologies, which reflect the differing research questions and contexts being evaluated. However, it also invites the question of whether there should be more standardization of outcomes measured or frameworks used to guide analysis and to facilitate increased comparison, synthesis and application across settings.

Three of the six papers that used an IR framework utilized the CFIR. Guan et al. used the CFIR at both a domain and construct level, Sarker et al. used the CFIR at the domain level, and Sami et al. did not specify which CFIR elements were used in informing the focus group discussion guide [51, 65, 70]. It is challenging to draw strong conclusions about the applicability of CFIR in humanitarian settings based on the minimal use of CFIR and IR frameworks within the papers reviewed, although Guan et al. provides a helpful model for how analysis can be structured around CFIR domains and constructs. It is worth considering that the minimal use of IR frameworks, and more specifically CFIR constructs, could be in part because that level of prescriptive categorization does not allow for enough fluidity in humanitarian settings. It also raises questions about the appropriate degree of standardization to pursue for research done in these settings.

The mapping of factors affecting SRH intervention implementation provides an example of how a modified CFIR framework could be used for IR in humanitarian contexts. This mapping exercise found factors that mapped to all five of the original CFIR domains as well as the sixth domain proposed by Means et al. All factors fit well within the definition for the selected domain, indicating an appropriate degree of fit between these existing domains and the factors identified as impacting SRH interventions in humanitarian settings. On a construct level, however, the findings were more variable, with one-quarter of factors not fully aligning with any construct. Furthermore, over 40% of the CFIR constructs (including the additional constructs from Means et al.) were not found to align with any factors cited by the papers in this review, also demonstrating some disconnect between the parameters posed by the CFIR constructs and the factors cited as relevant in a humanitarian context.

It is worth noting that while the CFIR as proposed in 2009 was used in this assessment, as well as in the included papers which used the CFIR, an update was published in 2022. Following a review of CFIR use since its publication, the authors provide updates to construct names and definitions to “make the framework more applicable across a range of innovations and settings.” New constructs and subconstructs were also added, for a total of 48 constructs and 19 subconstructs across the five domains [90]. A CFIR Outcomes Addendum was also published in 2022, based on recommendations for the CFIR to add outcomes and intended to be used as a complement to the CFIR determinants framework [91]. These expansions to the CFIR framework may improve applicability of the CFIR in humanitarian settings. Several constructs added to the Outer Setting domain could be of particular utility – critical incidents, local attitudes, and local conditions, each of which could help account for unique challenges faced in contexts of crisis. Sub-constructs added within the Inner Setting domain that seek to clarify structural characteristics and available resources would also be of high utility based on mapping of the factors identified in this review to the original CFIR constructs. As outcomes were not formally included in the CFIR until the 2022 addendum, a separate assessment of implementation outcomes was not undertaken in this review. However, analysis of the factors cited by papers in this review as affecting implementation was derived from the full text of the papers and thus captures content relevant to implementation determinants that is contained within the outcomes.

Given the demonstrated need for additional flexibility within an IR framework for humanitarian contexts, while not a focus of this review, it is worth considering whether a different framework could provide a better fit than the CFIR. Other frameworks have differing points of emphasis that would create different opportunities for flexibility but that do not seem to resolve the challenges experienced in applying the CFIR to a humanitarian context. As one example, the EPIS (Exploration, Preparation, Implementation, Sustainment) Framework considers the impact of inner and outer context on each of four implementation phases; while the constructs within this framework are broader than the CFIR, an emphasis on the intervention characteristics is missing, a domain where stronger alignment within the CFIR is also needed [92]. Alternatively, the PRISM (Practical, Robust Implementation and Sustainability Model) framework is a determinant and evaluation framework that adds consideration of context factors to the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) outcomes framework. It has a stronger emphasis on intervention aspects, with sub-domains to account for both organization and patient perspectives within the intervention. While PRISM does include aspects of context, external environment considerations are less robust and intentionally less comprehensive in scope, which would not provide the degree of alignment possible between the Characteristics of Systems and Outer Setting CFIR domains for the considerations unique to humanitarian environments [93].

Reflecting on their experience with the CFIR, Sarker et al. indicate that it can be a “great asset” in both evaluating current work and developing future interventions. They also encourage future research of humanitarian health interventions to utilize the CFIR [51]. The other papers that used the CFIR do not specifically reflect on their experience utilizing it, referring more generally to having felt that it was a useful tool [65, 70]. On their use of an evaluation framework, Berg et al. reflected that it lent useful structure and helped to identify aspects affecting implementation that otherwise would have gone un-noticed [67]. The remaining studies that utilized an IR framework did not specifically comment on their experience with its use [72, 83]. While a formal IR framework was not engaged by other studies, a number cite a desire for IR to contribute further detail to their findings [21, 37].

In their recommendations for strengthening the evidence base for humanitarian health interventions, Ager et al. speak to the need for “methodologic innovation” to develop methodologies with particular applicability in humanitarian settings [7]. As IR is not yet routinized for SRH interventions, this could be opportune timing for the use of a standardized IR framework to gauge its utility. Using an IR framework to assess factors influencing implementation of the MISP in initial stages of a humanitarian response, and interventions to support more comprehensive SRH service delivery in protracted crises, could lend further rigor and standardization to SRH evaluations, as well as inform strategies to improve MISP implementation over time. Based on categorizing factors identified by these papers as relevant for intervention evaluation, there does seem to be utility to a modified CFIR approach. Given the paucity of formal IR framework use within SRH literature, it would be worth conducting similar scoping exercises to assess for explicit use of IR frameworks within the evidence base for other health service delivery areas in humanitarian settings. In the interim, the recommended approach from this review for future IR on humanitarian health interventions would be a modified CFIR approach with domain-level standardization and flexibility for constructs that may standardize over time with more use. This would enable use of a common analytical framework and vocabulary at the domain level for stakeholders to describe interventions and the factors influencing the effectiveness of implementation, with constructs available to use and customize as most appropriate for specific contexts and interventions.

This review had a number of limitations. As this was a scoping review and a two-part search strategy was used, the papers summarized here may not be comprehensive of those written pertaining to SRH interventions over the past 10 years. Papers from 2013 to 2017 that would have met this scoping review’s inclusion criteria may have been omitted due to being excluded from the systematic reviews. The review was limited to papers available in English. Furthermore, this review did not assess the quality of the papers included or seek to assess the methodology used beyond examination of the use of an IR framework. It does, however, serve as a first step in assessing the extent to which calls for implementation research have been addressed, and identify entry points for strengthening the science and practice of SRH research in humanitarian settings.

With one in 23 people worldwide in need of humanitarian assistance, and financing required for response plans at an all-time high, the need for evidence to guide resource allocation and programming for SRH in humanitarian settings is as important as ever [94]. Recent research agenda setting initiatives and strategies to advance health in humanitarian settings call for increased investment in implementation research—with priorities ranging from research on effective strategies for expanding access to a full range of contraceptive options to integrating mental health and psychosocial support into SRH programming to capturing accurate and actionable data on maternal and perinatal mortality in a wide range of acute and protracted emergency contexts [95, 96]. To truly advance guidance in these areas, implementation research will need to be conducted across diverse humanitarian settings, with clear and consistent documentation of both intervention characteristics and outcomes, as well as contextual and programmatic factors affecting implementation.

Conclusions

Implementation research has potential to increase impact of health interventions particularly in crisis-affected settings where flexibility, adaptability and context-responsive approaches are highlighted as cornerstones of effective programming. There remains significant opportunity for standardization of research in the humanitarian space, with one such opportunity occurring through increased utilization of IR frameworks such as a modified CFIR approach. Investing in more robust sexual and reproductive health research in humanitarian contexts can enrich insights available to guide programming and increase transferability of learning across settings.