Background

By examining intervention mechanisms, implementation, and context, process evaluations aim to understand how complex interventions bring about outcomes, shed light on unanticipated effects, and inform optimal integration into existing practice [1]. They are often conducted alongside outcome/effectiveness evaluations of complex interventions, including trials, pilot and feasibility studies, and implementation studies [1]. As recognition has grown that outcome/effectiveness evaluations often provided insufficient understanding of increasingly complex interventions and their effects in different contexts, process evaluations have become increasingly common [1].

Health research funding and commissioning bodies in the UK, including the Medical Research Council [1], National Institute for Health and Care Research [2], and Public Health England (now the UK Health Security Agency) [3], highlight benefits of including process evaluations with evaluations of complex interventions. Their importance is also recognised internationally [4, 5], and in other fields such as education [6]. However, process evaluations have potential disadvantages, including Hawthorne effects [3] and participant burden [7]. There are also possible challenges to conducting process evaluations, including under-resourcing [1], and the complexity of interventions and contexts being evaluated [8].

Questions about how to do process evaluations have been substantially addressed in the literature [1, 9], however to our knowledge the concept of the ‘value’ of process evaluations has not been systematically critically examined. In scoping for this review, we noted that authors often used value-laden but ambiguous adjectives, such as ‘high-quality’, ‘useful’ or ‘necessary’ to describe aspects of process evaluation and process evaluation knowledge, without defining these terms. Some aspects of value have been considered, including whether process evaluations can satisfactorily meet the aim of explaining outcomes [10], the value of pragmatic formative process evaluation [11], and the reported value of process evaluations in pragmatic randomised controlled trials (RCTs) [12]. O’Cathain et al. [13] investigated the value of combining RCTs and qualitative research but did not specifically examine process evaluations.

Recommendations and assertions about value are likely to reflect authors’ ontological and epistemological standpoints [8], and accordingly there are a variety of interpretations of ‘optimal’ process evaluation design and conduct in the literature. For example, the MRC process evaluation guidance [1] outlines ontological and epistemological debates about how aspects of process such as fidelity and intervention mechanisms may be conceptualised and studied. There are also paradigmatic differences in how complex interventions are conceptualised [14], which impact perspectives on what a process evaluation should be and do.

The concept of “value” in research is multifaceted, with diverse definitions such as”why we do things, what is important, and to whom” [15]; “the established collective moral principles and accepted standards of persons or a social group; principles, standards or qualities considered worthwhile or desirable” [16]; and “contribution, impact and success” [13]. Research value is also commonly described in terms of impact, and various typologies and frameworks for categorising and assessing research impact have been proposed [17,18,19,20]. Value is also often discussed in terms of financial value and reducing waste brought about through inefficient research processes [21, 22].

In this paper we take a broad perspective on value, aiming to examine the different ways in which the ‘value’ of process evaluation is conceptualised and consider how and why perspectives may differ within the field. Essentially, we seek to establish what may be gained from process evaluation and for whom, potential negative consequences of process evaluations, and what is considered to make a ‘good’ or ‘useful’ process evaluation. In agreement with O’Cathain et al.’s [13] rationale for studying the value of qualitative research in RCTs, we believe taking stock of, and critically analysing the value of process evaluation in its broadest sense is important to advance the methodological knowledge base.

We also believe developing a planning framework of process evaluation value provides practical assistance to researchers designing process evaluations. By making explicit at the outset different expectations of value by different stakeholders, potential tensions may be addressed [16]. Given that process evaluation researchers likely need to prioritise which aspects of interventions to examine and may choose from a wide selection of methods and frameworks [1], we suggest it pertinent to address the question ‘what do we want to get out of this process evaluation?’ before addressing the question ‘how are we going to do this process evaluation?’.

Our aims were to identify and critically analyse 1) how process evaluations may create value and negative consequences, and 2) what kind of value process evaluations may create.

Methods

We conducted a critical interpretive synthesis, broadly following the approach outlined by Dixon-Woods et al. [23]. Accordingly, we aimed to synthesise a diverse body of literature to develop a conceptual framework of a concept (value) that has not been consistently defined and operationalised in this context (process evaluation). The critical interpretive synthesis approach is inductive and interpretive, with the body of literature itself used as an object of analysis as well as individual papers, for example by questioning the inherent assumptions behind what is said and not said [23]. Dixon-Woods et al. [23] describe critical interpretive synthesis as an approach to review and not exclusively a method of synthesis, and do not prescribe a step-by-step method of operationalising their approach. Accordingly, we adopted the basic principles of their approach and adapted it to suit this body of literature, the aims of this review, and our available resources.

Since there has been little previous research into the value of process evaluations, we based this review on literature including process evaluation guidance, opinions about process evaluations, and discussion of methodological and practical issues. Thus, we considered what authors were stating about process evaluations and their value in texts such introductions, discussions, opinion pieces, and editorials, as well as any research findings we did locate in the searches.

Search strategy

We searched for literature on process evaluation, including guidance, opinion pieces, primary research, reviews, and discussion of methodological and practical issues.

We searched the following sources:

  1. 1.

    Reference lists of four major process evaluation frameworks [1, 4, 9, 24]

  2. 2.

    Forward citation searches of the same four process evaluation frameworks using Web of Science and Google Scholar

  3. 3.

    Medline database search for articles with term “process evaluation*” in title; limited to English language

  4. 4.

    Scopus database search for articles with term “process evaluation*” in title; limited to English language; subjects limited to medicine, social sciences, nursing, psychology, health professions, pharmacology, dentistry

  5. 5.

    ETHOS database for PhD theses with term ‘process evaluation’ in the title (excluded in updated search)

  6. 6.

    Literature items not located by the searches but which we knew contained relevant information about process evaluation from our work in this field, such as broader guidance documents about evaluation methods containing sections on process evaluation.

CF originally conducted the search in September 2017 and updated it in January 2021. In the updated search we excluded the ETHOS database search due to time constraints.

Definition of process evaluation

We used the definition of process evaluation provided in the Medical Research Council’s process evaluation guidance [1] when selecting items for inclusion: ‘a study which aims to understand the functioning of an intervention, by examining implementation, mechanisms of impact, and contextual factors’. We chose this definition because the MRC’s process evaluation guidance is extensive and widely cited, and we considered its definition comprehensive.

Screening, inclusion, and exclusion criteria

We did not aim to include every item of relevant literature, rather to systematically search for and select literature most relevant to our aims. For example, literature on mixed-methods research and process evaluation concepts such as fidelity would have been relevant, however we only included those focusing on the overall concept of process evaluation. Although we only searched health-related sources, we did not limit inclusion to the field of health.

Inclusion criteria

We included published literature (including editorials, letters, commentaries, book chapters, research articles) that met all the following criteria:

  1. 1.

    Used the term ‘process evaluation’ in line with the above definition

  2. 2.

    Discussed process evaluation in any field, providing ‘process evaluation’ met the definition above

  3. 3.

    Discussed process evaluation accompanying any kind of outcome/effectiveness evaluation, intervention development work, or standalone process evaluation

Exclusion criteria

  1. 1.

    Items in which term ‘process evaluation’ is used to describe an evaluation not meeting the definition in our review

  2. 2.

    Items which only reported process evaluation protocols or findings – these were only included if they also discussed wider process evaluation issues (e.g. methodological, operational)

  3. 3.

    No full-text available online

  4. 4.

    Not in English language

Results screening

CF screened the titles and abstracts of all results, obtaining full texts where necessary to aid decisions.

Data analysis and synthesis

We did not conduct quality appraisal of the included literature as we selected diverse items such as editorials, and synthesised whole texts as qualitative data, rather than aggregated research findings.

This review was inductive and we did not start out with a priori concepts or categories about how process evaluations create value or the type of value they create. We kept in mind however the value system of ‘process’, ‘substantive’ and ‘normative’ values outlined by Gradinger et al. [16] to sensitise us to values possibly stemming from 1) the conduct of process evaluation; 2) the impact of process evaluation or 3) the perceived intrinsic worth of process evaluation, respectively. We considered ‘value’ in its broadest possible sense, and examined what authors stated, implied, and discussed about what may result from a process evaluation (both positive and negative), the purposes of process evaluation, and what makes a ‘good’ or ‘useful’ process evaluation.

Following the critical interpretive synthesis approach [23], we also aimed to be critical through questioning the nature of assumptions and proposed solutions relating to process evaluation issues discussed in the literature. This enabled us to examine how authors covering diverse fields and types of process evaluation variously perceived value in different contexts.

CF initially undertook this work as part of her PhD from the original search results in September 2017 with 109 included items (see Fig. 1). Following initial reading of all items to gain familiarity she began the detailed analysis of approximately one third of randomly selected papers (n = 40) by extracting sections of text relating to how process evaluations create value and types of value that may be created. She organised these into an initial coding framework, using NVivo to manage the data and noting impressions of the overall literature. She then used this framework to code the remaining items (n = 69), amending the framework as necessary. A further 38 literature items were identified following the updated search in January 2021 (see Fig. 1), which CF coded in the same way, further refining the framework.

Fig. 1
figure 1

PRISMA flow diagram

Full size image

Dixon-Woods et al. [23] describe the benefits of a multidisciplinary team approach to the whole review and synthesis process. As this paper reports work initiated through individual doctoral work we decided to strengthen and deepen the analysis by independently double coding a total of 36 of the total 147 items (approximately 25%). We used purposive sampling to select the 36 papers for double coding, selecting papers with varied characteristics (year of publication, country of lead author, field of practice, and focus of paper). Four authors coded nine papers each using the coding framework developed by CF, also noting any new themes, interpretations, and areas of disagreement. We brought these to a team discussion to refine the themes and develop the final analysis. We developed this double coding approach as a pragmatic solution to incorporating multiple perspectives into the synthesis, based on our experience of conducting similar narrative reviews and team qualitative data analysis.

From the resulting themes, notes on interpretations, and team discussions we created a narrative and conceptual framework of our analysis, along with a practical planning framework for researchers designing process evaluations.

Results

Search results

We included 147 literature items, and our search results are shown in Fig. 1.

Table 1 shows characteristics of the included literature items, with a detailed summary table in additional file 1.

Table 1 Characteristics of included literature items
Full size table

Critical interpretive synthesis overview

Figure 2 provides an overview of the findings of this synthesis.

Fig. 2
figure 2

Overview of synthesis findings

Full size image

As shown in Fig. 2, we identified three ways in which process evaluations may create value: 1) through the socio-technical processes of ‘doing’ the process evaluation, 2) through the features/qualities of process evaluation knowledge, and 3) through using process evaluation knowledge.

From these three ways in which process evaluations may create value we identified 15 value themes. Many of these 15 themes included both positive and potentially negative consequences of process evaluations. Value and negative consequences may be created for many different stakeholders, including research participants, researchers, students, funders, research commissioners, intervention staff, organisations, practice settings, research sites, interventions, practice outcomes, and outcome evaluations.

However, as shown in the box describing process evaluation characteristics in Fig. 2, process evaluations may vary widely in terms of 1) which processes are evaluated 2) how these processes are evaluated, 3) the practical conduct of the process evaluation, and 4) how process evaluation knowledge is disseminated. Value is therefore at least partially contingent on the characteristics of individual process evaluations.

Finally, process evaluations are designed, conducted, and their knowledge applied in many different contexts. We found different stakeholders in different contexts may have different perspectives on what is valuable, meaning the value created by process evaluations is subjective. We therefore noted potential tensions and payoffs between certain values.

Figure 3 provides an overview of the themes of value and shows how the themes relate to the three identified ways in which value may be created. We describe these findings in detail in Tables 2, 3, and 4, including subthemes and examples from the synthesised literature. We then end this results section with a discussion of tensions between values.

Fig. 3
figure 3

Overview of the themes of value

Full size image
Table 2 Process evaluation value created through the socio-technical processes of ‘doing’ the process evaluation
Full size table
Table 3 Value related to the features/qualities of process evaluation knowledge
Full size table
Table 4 Value created by using process evaluation knowledge
Full size table

Value created through the socio-technical processes of ‘doing’ the process evaluation

Many social and technical processes are involved in the design, conduct, and dissemination of process evaluation, and thus value and negative consequences may arise from the ‘doing’ of the process evaluation. Examples of socio-technical processes include collecting observational data at a research site, inviting a trial participant to participate in a process evaluation, and designing a questionnaire. These are all carried out by multiple human actors (for example researchers and research participants) using a variety of knowledge products (for example evaluation frameworks and research protocols). In Fig. 2, these processes and actors are summarised under the heading ‘process evaluation characteristics’. Taking a stance that value is situated and formed out of context, the way in which these processes evolve have a direct impact on the value that can be derived from a process evaluation. We identified six themes of value stemming from socio-technical processes:

  • Relationships

  • Giving people a voice

  • Education

  • Ethical issues

  • Financial

  • Impact on the outcome evaluation

Table 2 shows the themes, subthemes, and examples of how socio-technical processes may create value from process evaluations.

Value related to the features/qualities of process evaluation knowledge

The second way in which process evaluations may create value relates to the features and perceived qualities of the knowledge they produce. The process evaluation characteristics outlined in Fig. 2 clearly lead to different kinds of process evaluation knowledge being produced, for example qualitative or quantitative. We identified three themes of value which relate to the features and qualities of process evaluation knowledge:

  • Knowledge credibility

  • Knowledge accuracy

  • Knowledge completeness

Table 3 outlines how process evaluation variables may impact on the perceived value of the knowledge that is produced.

Inevitably, some of the ways in which process evaluation knowledge may be inaccurate or incomplete described in Table 3 may be unavoidable. For example, it is likely impossible for financial, practical, and ethical reasons for process evaluations to investigate every potentially important aspect of an intervention [1, 41]. Issues such as gatekeeping, self-selection bias, and social desirability bias are research challenges not unique to process evaluations. However, the literature suggests that process evaluation reporting is often suboptimal, with detail on methods lacking, choices about methodology and areas of enquiry not justified [9, 34, 40, 55, 63, 71, 97, 131], and limited discussion of quality, validity, and credibility [9, 40, 63, 90]. This suggests inaccuracy and incompleteness of process evaluation knowledge may not always be acknowledged.

Furthermore, some authors suggest that some process evaluation researchers do not recognise that their methods may be overly simplistic portrayals of reality, and therefore fail to consider important aspects of process [40, 59]. Some papers conceptualised process evaluation components as highly complex, suggesting that methods such as ethnography [34], realist evaluation [46], and the use of theoretical frameworks such as normalisation process theory [132] were necessary to fully capture what was going on. At the opposite end of the spectrum some papers conceptualised process evaluation components simplistically, for example equating whether or not intervention recipients enjoyed intervention components with their effectiveness [91]. A potential negative consequence of process evaluations therefore may be if knowledge is uncritically presented as providing explanations when researchers did not account for all factors or the true level of complexity. For example, assessing single dimensions of implementation may lead to ‘type III errors’ through incorrectly attributing a lack of intervention effect to a single implementation factor, when the actual cause was not investigated [40, 117].

Value created by using process evaluation knowledge

The third way in which value and negative consequences may be created is through using the knowledge produced by process evaluations. Process evaluation knowledge may be used and applied after the evaluation. It may also be used formatively to make changes to interventions, implementation, contexts, and evaluation processes during the evaluation. Some experimental outcome evaluation methods prevent formative use of knowledge to maintain internal and external validity. We identified six themes of value stemming from the use of process evaluation knowledge:

  • Supporting implementation of interventions into practice

  • Informing development of interventions

  • Improving practice and outcomes

  • Contribution to wider knowledge

  • Financial value of knowledge

  • Impact on the outcome evaluation

These are described along with sub-themes and examples in Table 4.

Tensions within and between values

As well as identifying how process evaluations may create value and themes of value, we found that the concept of value in process evaluations is subjective and context-dependent, and there are tensions within and between values.

The value of process evaluation is not pre-existing but enacted and created through ongoing negotiation between those with a stake in what is being evaluated. Through designing and conducting a process evaluation and disseminating and using its knowledge, process evaluation actors and knowledge products may directly or indirectly create value and negative consequences for many different stakeholders and bystanders in different contexts. These include people and organisations who participate in research, conduct research, use research findings, receive interventions, work in research and practice settings, fund research, regulate research, or are simply present where process evaluations are being conducted. These groups and organisations have different expectations, values, and needs; and there is also variability within groups and organisations. This creates the potential for tension between expectations, values, and needs of different stakeholders.

We identified two broad perspectives on value. In the first, process evaluations are primarily valued for supporting the scientific endeavour of outcome evaluations, particularly trials. Examples of this include process evaluations being conducted to minimally contaminate or threaten interventions and outcome evaluations, with the generated knowledge applied post-hoc and providing retrospective understanding [87, 118]. Formative monitoring and correction of implementation aims to ensure internal validity [24, 44, 48, 77, 93, 94]. Value is framed around meeting the needs of the outcome evaluation, such as through complementing trial findings [9], and the perceived utility of findings may be contingent on what happens in an outcome evaluation [133]. They are also framed around the needs of researchers and systematic reviewers. For example, calls for them to include set components to make them less daunting to conduct and enable easier cross-study comparison [1, 5, 24, 57, 58].

In the second perspective process evaluations are mostly valued for formatively contributing to intervention development, improving practice, and forging relationships with stakeholders. Evaluating implementation may allow for the adaptation and tailoring of interventions to local contexts [1], which may result in them being more patient-centred [126], with better fit and feasibility in local settings [55]. Process evaluations may be seen as opportunities to utilise methodologies with different ontological and epistemological assumptions to RCTs, with flexible designs that are tailored to the uniqueness of each intervention and setting [34, 67]. These process evaluations are more likely to find multiple nuanced answers, reflecting assumptions that reality is unpredictable and complex, and that interventions are most effective when adapted to different contexts. These seem more concerned with giving participants voices and uncovering messy realities, developing effective sustainable interventions, and through these, improving outcomes [33, 60].

Some authors give examples of process evaluation designs which may capitalise on both perspectives on value. In-depth realist formative process evaluations at the stage of piloting interventions incorporate the benefits of developing and theorising effective, sustainable, adaptable interventions that are tailored to local contexts, which can then be tested in a rigorous outcome evaluation [46]. Pragmatic formative process evaluations theorise interventions which are already in practice and optimise implementation in readiness for outcome evaluations [11, 35].

The literature also contains examples of tensions between these two perspectives. For example, process evaluation methods that enhance engagement with participants may increase the effect of the intervention, which may be seen as desirable [32] or a problematic Hawthorne effect [1]. If data from summative process evaluations reveal problems with interventions or implementation during the evaluation, this can raise ethical and methodological dilemmas about whether to intervene [42, 43]. Riley et al. suggested process data monitoring committees as forums for debating such contentious scenarios to address these issues [43]. Others highlighted the importance of stakeholders having clear expectations about the value that process evaluations may create and when, to avoid tensions stemming from unmet expectation. Examples include establishing clear mandates with intervention staff about when they will receive feedback on their delivery [31] and how their data will improve interventions [89].

Discussion

Summary of findings

Process evaluations do not have value a priori. Their value is contingent on the features and qualities of the knowledge they produce, and the socio-technical processes used to produce that knowledge. There is also potential to create consequences that may be perceived negatively. However, there are not simple definitive answers to the questions ‘what kind of value do/should process evaluations create?’ or ‘how do/should process evaluations create value?’. This is because:

  • The label ‘process evaluation’ may be applied to many different types of studies producing diverse kinds of knowledge and using diverse socio-technical processes.

  • Process evaluations are undertaken in different research and practice contexts in which different kinds of knowledge and socio-technical processes may be perceived as more or less valuable or desirable.

  • Process evaluations are undertaken by researchers with differing ontological and epistemological standpoints and research traditions, who have different views on what constitutes high-quality, useful, and valuable knowledge.

Theoretical considerations

Our analysis shows that part of the challenge of interpreting the value of process evaluation is that researchers and other stakeholders are debating value from different ontological and epistemological starting points. These tensions resonate with the wider literature on qualitative research with quantitative outcome evaluation [13, 45, 134, 135], and how complex interventions should be conceptualised and evaluated [136,137,138].

There are tensions between values, particularly payoffs between optimising value to outcome evaluations and triallists, and optimising value to intervention development and relationship-building. While the professed aims of both are to improve practice and outcomes for intervention recipients and to advance knowledge, the beliefs about how this is best achieved often differ. For example, process evaluation researchers with a more positivist stance likely believe a positive primary outcome result with high internal validity is most likely to ultimately improve practice and outcomes. They may therefore value process evaluations which minimally contaminate interventions and measure fidelity. Process evaluation researchers with a more interpretivist stance likely believe in-depth understanding of the experiences of intervention recipients is more likely to ultimately improve practice and outcomes. They could therefore value process evaluations which engage participants in more in-depth data collection methods.

While it is beyond the scope of this paper to debate the relative merits of these paradigmatic differences, ontological and epistemological perspectives appear to strongly influence perspectives on what kind of knowledge it is valuable for process evaluations to generate. This demonstrates the importance of making ontological and epistemological perspectives explicit when discussing how to design and conduct process evaluations, for example in process evaluation guidance and frameworks [8].

We also encourage researchers to take stock of these different perspectives on value and critically reflect on whether concentrating value on one perspective potentially misses the opportunities to create value offered by another. For example, through the aim of minimally contaminating interventions are opportunities missed to engage stakeholders who could assist with intervention improvement and post-evaluation implementation? Are there potential ways to combine both approaches to process evaluation? As highlighted in our analysis, in-depth formative process evaluations in the intervention development and feasibility testing stages offer this opportunity [46]. Furthermore, the newly updated Medical Research Council Framework for evaluating complex interventions [138] (published after we completed the searches for this review) states “A trade-off exists between precise unbiased answers to narrow questions and more uncertain answers to broader, more complex questions; researchers should answer the questions that are most useful to decision makers rather than those that can be answered with greater certainty”. This suggests pragmatic weighing-up of the overall value created by process evaluations will become increasingly significant.

Practical applications

Our findings have practical applications for researchers designing process evaluations to be intentional in creating value and avoiding negative consequences. We recommend that since process evaluations vary widely, before researchers ask: ‘how do we do this process evaluation? they ask: ‘what do we want to get out of this process evaluation?’. Process evaluations will create value, and potentially negative consequences regardless of whether it is planned, so we suggest purposefully and explicitly preparing to create value in conjunction with stakeholders.

Figure 4 shows a planning framework to be used in conjunction with Fig. 3 and the analysis in this paper to aid this process. As would be good practice in any research, we recommend these discussions include as many stakeholders as possible, including intended beneficiaries of research, also reflecting the possible diversity of research backgrounds and epistemological standpoints within research teams. This would help guide decisions around design, conduct, and dissemination by making expectations of value explicit from the outset, addressing potential tensions, and ensure contextual fit. While the nature of any accompanying outcome evaluation will influence expectations of value, it is useful for stakeholders to be aware of potential payoffs and ensure there is a shared vision for creating value. This will likely also aid researchers to narrow the focus of process evaluation to make it more feasible and best allocate resources, as well as highlighting its value to stakeholders without relevant knowledge and experience.

Fig. 4
figure 4

Process evaluation planning framework

Full size image

Strengths and limitations

We included a large number of literature items relating to process evaluations in diverse contexts, which enabled us to synthesise a broad range of perspectives on value and highlight how value may be context dependent. This will enable readers to apply findings to their own contexts. Nonetheless our review does not include all literature that could have been informative, and therefore the values and issues identified are unlikely to be exhaustive. Furthermore, author texts we extracted as data for our review may have been influenced by expectations and limitations of publishing journals. Exploring the concept of value by reviewing the literature only captures perspectives which authors have decided to publish, and other aspects of value are likely to be uncovered through empirical study of process evaluation practice.

Although we have outlined our review methods as explicitly as possible, in line with critical interpretive synthesis the review was by nature interpretive and creative, therefore full transparency about step-by-step methods is not possible [23]. We present our interpretation of this body of literature and acknowledge that this will have been influenced by our pre-existing opinions about process evaluation. Nonetheless our team included researchers from different backgrounds, and through a double-coding process and reflective team discussion ensured we did not unduly focus on one aspect of value or prioritise certain perspectives.

Conclusions

Process evaluations vary widely and different stakeholders in different contexts may have different expectations and needs. This critical interpretive synthesis has identified potential sources of and themes of value and negative consequences from process evaluations, and critically analysed potential tensions between values. Accommodating all needs and expectations of different stakeholders within a single process evaluation may not be possible, but this paper offers a framework to support an open transparent process to plan and create value and negotiate trade-offs. This supports the developments of joint solutions and, ultimately, generate more value from process evaluations to all.