Background

Implementation science has been defined as the transfer of clinical research findings and evidence-based results into the real world and hence how a study can affect or hinder its uptake in the routine practice [1,2,3,4]. Thus, implementation science is set to observe and study the gap between, on one side, a solution developed in a controlled environment and, on the other, the specific context where the intervention is applied by looking at contextual factors that may act as barriers or facilitators.

However, interventions and solutions built in a controlled setting and transferred to specific context, have been argued to obtain limited success, mostly in the long term. For instance, the misconsideration for complex systems and factors related to settings and the targeted population have been said to influence the lack of effectiveness [5, 6]

Taking into account the relevance and inclusion of stakeholders’ knowledge in research production as been put forward as a possible way to address the research-practice gap [2, 3]. For this reason, more recently, implementation science has been experiencing a shifts from this type of linear and controlled production models to more iterative participatory and complex models [7,8,9] with the design and creation of solutions and interventions directly in the real world.

Involving relevant stakeholders from the earliest stage of intervention design and/or implementation has been considered a way to increase uptake and positively affect not only patient satisfaction but also the quality of the service [10,11,12,13,14,15]. In line with this considerations, co-creation has been brought to the forefront as an opportunity to increase the successful uptake of evidence-based interventions and practices through meanginful and deep engagement of key stakeholders [16,17,18,19].

Co-creation is a collaborative approach of creative problem solving engaging diverse stakeholders at all project stages, from determining and defining the problem through to the final stages of a project [20]. By facilitating collaboration among key stakeholders, co-creation aims to taking into account social determinants and contextual factors that may influence the intervention’s feasibility and acceptability from the earliest stage of intervention design.

Considering co-creation’s intention to work within real-life settings and conditions, systems thinking seems to be a valuable approach to explore and potentially adopt when designing and evaluating co-creation. Adopting a system thinking approach would allow assessing contextual elements from an early stage of the intervention and gathering considerations around systemic factors that may influence the public health issue [21].

The need for formative evaluation in co-creation has been argued to be crucial to co-creation processes. An evaluation is intended to be formative when the implementation team and/or staff use data to improve or adapt the process of implementation [1]. Van Dijk-de Vries [22] argues that researchers, when co-creating, should assess the stakeholders’ engagement to ensure their perceptions are captured, suggesting this happens throughout the implementation. Formative evaluation would enable, if needed, to adapt and adjust the intervention.

Despite research advancement in the field, implementation guidance and recommendations for the planning and implementation of co-creation processes are lacking as existing implementation and evaluation frameworks have not been designed specifically for such approaches. The need to develop dedicated implementation and evaluation guidelines for co-creation lies in the distinctive nature of co-creation approaches, involving collaborative efforts with diverse stakeholders, emphasizing shared decision-making, innovation and creativity.

Implementation and evaluation guidance needs to be further developed to address the dynamic and participatory nature of these processes and the unique challenges of fostering meaningful partnerships, navigating diverse perspectives and harnessing collective creativity. Closing this gap is essential not only for the successful implementation of co-creation initiatives but also for unlocking the full potential of these collaborative efforts within the broader landscape of public health interventions. This study aims to address this gap by reviewing existing frameworks and offering an overview of recommendations that may guide the design and implementation of co-created interventions.

Methods

Eligibility criteria

To be included, studies had to describe or introduce an implementation or evaluation framework, therefore describing or introducing a framework representing key stages, factors, constructs or variables that explain or influence the implementation and/or evaluation of programs/interventions. Frameworks had to be generalizable, and therefore designed to be applicable for all public health topics. For the scope of this review, an intervention was defined as a set of actions with a coherent objective to bring about change or produce identifiable outcomes [26]. We identified the Promoting Action on Research Implementation in Health Services (PARIHS), designed in 1998, as our start date search, as one of the first frameworks to make explicit the multi-dimensional, complex nature of implementation and highlight the central importance of context [27]. Table 1 includes further details on applied criteria.

Table 1 Eligibility criteria
Full size table

Search strategy

This scoping review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines for producing the PRISMA flow diagram (see Additional file 1). A specialist librarian was involved in developing the search strategy on PubMed and CINAHL. These were equal search strategies adapted for each database. The detailed search is available in Additional file 2. In addition, we applied a snowballing technique of pursuing backwards references cited in selected publications.

Process of selection

At least two authors were independently involved in all review studies through each stage [23]. D.A., K.M., J.Z., M.B., M.G. and G.L. were involved in the title and abstract screening. All the latter co-authors, with the addition of K.G., conducted the full-text screening. D.A., K.M., J.Z., M.B., M.G., G.L. and K.G. took part in the data extraction, and K.G. and G.L. conducted the data analysis. Authors conducted individual and independent reviews through the software Rayyan.Footnote 1 Discrepancies were resolved by consensus and, if unresolved, by the involvement of a third reviewer (M.G., G.L., G.C.).

Included frameworks are reported in Fig. 1. From each framework, we extracted prominent constructs (Additional file 3) and steps (Additional file 4). For extraction purposes, we defined constructs as a fundamental unit of thought, smaller than a judgment or theory but integral to them [24]. Identified steps were presented by categorizing them into the Leask’s et al. [25] framework included stages of planning, conducting, evaluating and reporting [25].

Fig. 1
figure 1

Classification of extracted frameworks

Full size image

Data extraction

An Excel template was developed to extract information relate to the framework’s nature, type and scope, positioning of implementation and evaluation considerations within the frameworks, and type of evaluation covered and other elements (see Table 2 for full list of data extracted). For each framework, if applicable, we extracted the main constructs (Additional file 3) and main phases (Additional file 4). The data extraction sheet was developed and piloted by two reviewers (G.L., M.G.) on two eligible papers and reviewed accordingly. D.A., K.M., J.Z., M.B., M.G., K.G. and G.L. extracted all data independently and blind-folded.

Table 2 Data extracted
Full size table

Data analysis

With the scope of aligning findings and come to a final set, extracted data was then cross-checked between G.L. and K.G. G.L. and K.G. sought consensus and, if in disagreement, they involved MG as a third reviewer. Final data was then plotted by G.L. in Fig. 1, which includes the framework’s classification data and steps and constructs reported in Additional files 2 and 3. K.G. and G.L. further conducted thematic analysis of included frameworks, as described below.

Thematic analysis

Through thematic analysis, we aimed to identify from included frameworks recommendations that would be relevant to the context of co-creation. This process followed the six stages as outlined by Braun and Clarke [58], including the following steps: (1) G.L. and K.G. familiarized with the data and wrote familiarization notes; (2) G.L. and K.G. developed a coding; (3) G.L. and K.G. independently generated initial themes from coded and collated data; (4) finally met to develop and reviewing themes; as well as to (5) refine, define and name themes; and (6) they applied the thematic framework to the remaining frameworks. [24]. The two researchers (G.L. and K.G.) independently coded four frameworks and then met to develop the thematic framework, which was then applied to the remaining frameworks. The themes and related recommendations reported in the results section emerged as the result of the data coding and iterative theme development [58].

Coding themes included the following: (a) early implementation considerations—exploring how the frameworks were including early implementation consideration; (b) system thinking—understand in what way frameworks were framed within a systems thinking paradigm [21, 29, 30]; (c) partnering with stakeholders—retrieving information on how the frameworks partnered with the stakeholders; (d) experience—how frameworks were pointing towards an assessment of the users’ experience throughout the process; or, finally, (e) iterative and cyclical evaluation—we explored how frameworks were accounting for this aspect.

Results

Summary of identified frameworks

From a total of 9061, after removing duplicates, 5284 papers were screened at title and abstract and 425 retrieved for full-text screening. We identified 30 articles and related implementation and evaluation frameworks (Table 3).

Table 3 List of frameworks
Full size table

As shown in Fig. 1, among frameworks, we identified 18 process models, 11 determinant frameworks and 16 evaluation frameworks. Most frameworks (16) are implementation frameworks, including evaluation elements, while 9 frameworks exclusively look at implementation or evaluation. There are 13 descriptive and 18 prescriptive frameworks. A total of 12 frameworks include implementation considerations from the stage of intervention development. Frameworks focus on various evaluation elements, with most frameworks (21) including effectiveness and 16 looking at impact evaluation, 15 at efficacy, 17 frameworks concern process evaluation and 11 related to an evaluation of elements related to the user experience. A total of 3 evaluation frameworks were concerned with evaluation of the planning phase, 8 were concerned with the conducting phase, 11 were concerned with post-execution and 12 were used throughout all stages.

Prominent constructs and steps were extracted from selected frameworks to give an overview of constructs (Additional file 3) and steps (Additional file 4) included in and steps that made part of the frameworks. Constructs that have been mentioned in frameworks include evidence, feasibility, acceptability, etc. and steps reported were moments of the process that the framework suggested carrying out.

Thematic analysis results

Below, we share findings derived from thematic analysis. Our intention is for these insights to represent recommendations that might be relevant for future research and the implementation of co-creation in practice. Figure 2 represents key themes and sub-themes identified through the thematic analysis of the identified frameworks.

Fig. 2
figure 2

Recommendations

Full size image

Early implementation considerations in extracted frameworks

A total of 12 frameworks include reference to early implementation considerations. Eslava-Schmalback et al. [37], recommend identifying critical factors for implementing equity focus recommendations and exploring barriers and facilitators of the intervention from the design phase. Kitson et al. [40] pay attention to preparing the intervention for context application, and Wimbush and Watson [44] call out the possibility of significant inconsistency between an intervention developed in experimental conditions and implementation in the real world.

Among frameworks, Cambon and Alla [33] focus on the context in which the intervention is implemented and argue that this should be viewed as an “interventional system”. In most frameworks, taking into account potential barriers to implementation takes the form of attention to the acceptability of its final users. It is claimed that by assessing its acceptance rate with users, the intervention might address potential barriers to its real-world application. In its description of the intervention cycle, Campbell et al. [31], for instance, advises adopting an iterative process through which the potential recipients’ acceptability of the intervention is assessed and re-examined if needed.

Similarly, Gonot-Schoupinsky and Garip [46] framework dedicates special attention to appropriateness and morality and how the user feels about the intervention as it may impact the intervention’s scalability potential. Assessing acceptability to both end-users and to stakeholders early in the process may be a crucial consideration for large-scale intervention implementation because of its potential to identify potential contextual barrier, enablers and motivations to participation in interventions [46].

Systems thinking in frameworks

Among the frameworks identified, seven explicitly reference a systems thinking perspective. Best et al. [43] advocate for a systems-grounded frame to be built with key stakeholders. Lo and Karn [41] view complex health programme interventions as systems composed of interdependent features and factors. The latter includes interdependent features and characteristics, such as human behaviours/perceptions, skills and capacity, and governmental and physical structures.

Similarly, Titler [51] recommends finding ways to account for systemic factors in the study design and randomized controlled designs. Zucca et al. [52] place intervention within a systems approach, making a distinction between a complex systems approach, in which variables are so intertwined that the cause and effect relation is uncertain, versus a complicated system, where numerous elements and relationships exist but their relationship can be unveiled and understood. Within the same perspective, Eslava-Schmalback et al. [37] stress the importance of understanding complex systems to advance and enhance implementation.

Partnerning with stakeholders

Co-creation is considered an approach that promotes engagement in partnership with stakeholders throughout the intervention. We reported on the level of engagement by identifying frameworks that had involved their stakeholders in a partnership, meant, according to Arnstein’s ladder, as the commitment to share planning and decision-making responsibilities through a set structure with its key stakeholders [53].

Six frameworks include elements of partnership with stakeholders. Racher and Annis [48] and Leask et al.’s [25] framework define partnership as the instance in which the stakeholders experience ownership while also (b) providing directional guidance and (c) being invested with responsibilities for activities and outcomes [48].

Partnerships are to be established according to frameworks, with different groups, including with (a) people involved in programme operations, (b) those served or affected by the programme, and (c) primary users of the evaluation [54] or, when community partnerning, between (a) multidisciplinary researchers, (b) the health researchers and community practitioners, and (c) and community health organizations at an international, national and local level [43].

Stakeholders’ views and experience are considered equal to other types of knowledge by Kitson et al. [40] Chen [38], which include patient preferences, views and experience as equally valuable and crucial to evaluate whether an intervention is practical, affordable, suitable, evaluable, and helpful in the real world.

Evaluating the experience of the co-creators

Benefits of joining the co-creation process might include cognitive, social and personal benefits [55]. To maintain and assert the value of co-creation to the co-creators involved, assessing their experience seems essential.

We reviewed the extent to which frameworks were evaluating the experience of the stakeholder’s involvement in the intervention. Gonot-Schoupinsky and Garip [46] include the assessment of acceptability to include reflections on appropriateness or morality, and how the user might have felt about the intervention, while Jolley et al. [50] suggest investigating barriers to participation and state the importance of ensuring the process is inclusive and values diversity.

Hennesy Lavery et al. [37] and Masso et al. [47], when evaluating the intervention, assess whether the level of agency of participants has increased and regard it as crucial to achieving the sustainability of the intervention. Marckmann et al. [35] stress the importance of evaluating the impact on autonomy while including the elements of health-related empowerment, such as health literacy, respect for individual autonomous choice and protection of privacy and confidentiality.

Iterative and cyclical evaluation

To comprehensively account for influential implementation elements, iterative evaluation at the planning and conducting phases allows researchers to address and prevent implementation obstacles by assessing the stakeholders’ perceptions and views and adjusting the intervention as needed.

Among frameworks, seven studies recognize the need to perform a more cyclical and iterative evaluation to allow for an intervention to be sustainable within its actual context and replicability to others [36, 44, 47, 50, 54]. Wimbush and Watson, for instance, suggest iterative evaluation as a way to review the intervention's feasibility, practicability, acceptability and for adjusting the programme’s initial design.

Discussion

This review identified 30 implementation and evaluation frameworks, classified according to their types and according to the categories specified in the data extraction. By analysis the frameworks through thematic analysis, it also offered insights into considerations for when implementing and evaluating future research and practice of co-creation.

Recommendations included accounting for early implementation considerations. Anticipating implementation questions has in fact been argued to be a way to increase the sustainability and maintenance of the intervention in the real-world setting [42] and considered by Moore et al. as crucial for future intervention development and evaluation [28]. Considering the adoption of a systems thinking approach was included as a key facet. Interventions, it is argued, need to be contextualized and understood in, rather than isolated from, the systems they operate within and co-creating interventions with its relevant stakeholders and intended target population, who hold deep knowledge of the systems they are situated within, can ensure a closer tie between theory and context.

By working with multiple levels of influence and with related contextual settings and systems, systems thinking [56]  seems to fit well the scope and intention of co-creation. With its intention to map the larger environment and to identify obstacles and challenges impacting and affecting the public health matter in question, systems thinking enables co-creation to address beyond the isolated causal effect but rather to explore and identify the multiplicity of real-world systematic factors that collate and contribute to the complex problem.

Evaluation is essential and crucial to co-creation and key is formative and cyclical evaluation, as suggested by Anneke van Dijk-de Vries et al. [22]. This, in fact, allows researchers to address and prevent implementation obstacles by assessing the stakeholders’ perceptions and views and adjusting the intervention as needed throughout.

By fostering reflection moments to ensure that end users’ perceptions are continuously captured [25], iterative process evaluation can represent a powerful tool in placing the voice and perception of the co-creators at the core of the intervention cycle. Doing so is particularly relevant when co-creating, as the process and how the co-creators are involved throughout, become part of a co-created intervention’s major outcomes and value in itself  [22].

Partnering with stakeholders and evaluating the co-creators experience is key as the co-created solution is expected to be developed jointly and provide benefits to the co-creators. Valuing the co-creators’ perceived level of co-ownership has been previously regarded essential to the co-creation process [25] and a way to ensure the co-created solution is developed through meaningful engagement.

This review identifies 30 implementation and evaluation frameworks for co-creation and offers recommendations for the planning and evaluating of co-creation for public health. Recommendations emphasize the importance of early implementation considerations, adopting a systems thinking approach, and prioritizing formative and cyclical evaluation. Iterative process evaluation is suggested as a powerful tool to centre co-creators’ voices in intervention cycles, posing and recognizing the value of the co-creation process itself, and not only on the implementation of the co-created solution. To underscore the significance of meaningful engagement and co-ownership when developing co-created solutions, the review highlights attention on the partnering with stakeholders and on an evaluation of the co-creators’ experiences.

This scoping review is conducted as part of the Health CASCADE study and findings will be used to inform the development of further guidance on planning and evaluating co-creation for public health. Authors will dive deeper into the framework by Leask et al. 2019 [25] identified through this review to identify strengths and weakness and expand on the available guidance, through a scoping review, and qualitative interviews with key stakeholders. Authors will also conduct a scoping review on process evaluation studies for co-creation and qualitative interviews with key stakeholders to develop an evaluation framework for co-creation.

Limitations of the study

Firstly, as we intended to explore the broader implementation field, we included several types of implementation frameworks within our definitions. This meant we captured several non-primary studies, presenting frameworks that had been developed for and/or applied to specific settings and contexts. These studies were later excluded at full-text screening. This might have caused the loss of frameworks that were specific to a context and setting but relevant to the scope of the study. We, however, as previously explained, performed snowballing on the identified frameworks to reduce this possible loss.

Secondly, while we set the search strategy with a specialist librarian, the review might have missed terms used for the same scope by other professionals (e.g. reporting guidelines, checklist or step-by-step how-to).

As part of our analysis, we scoped for implementation intervention development frameworks and did not include public health intervention development frameworks as we were interested in frameworks built to help guide the implementation of the intervention. Therefore this means the search lacks frameworks supporting the intervention development. We applied the search strategy to databases focussing on public health per the review’s scope. Therefore, this review might lack frameworks in databases from social sciences, although the snowballing exercise aimed to reduce the bias as it was performed with no limitations to the databases’ field. It is also worth acknowledging that the frameworks’ classification and data extraction were extracted independently by two reviewers and agreed upon by consensus to ensure the analysis was accurate. Nevertheless, interpretations made as part of the frameworks’ analysis were based on the reviewers’ subjective appraisal.

Conclusions

This review identified, classified and analysed 30 implementation and evaluation frameworks and offered recommendations for professional stakeholders and researchers wanting to adopt a co-creation approach.

The study recommends, when co-creating, to (a) include implementation considerations from an early stage and at the stage of intervention planning, (b) adopt a systems thinking approach when co-creating, and (c) form a partnership relationship with stakeholders to (d) plan for an iterative and cyclical evaluation and (e) focus on evaluating the co-creators’ experiences.

Contributions to literature

  • This scoping review identifies and classifies 30 implementation and evaluation frameworks for the development, implementation and evaluation of interventions in public health.

  • The analysis suggests positioning implementation considerations from an early start of the intervention and adopting a systems thinking approach to the implementation and evaluation of co-created interventions.

  • The authors highlight the importance of partnering with stakeholders and recommend carrying out an evaluation that is iterative and cyclical and focusses on the experience of the co-creators.