Advancing complexity science in healthcare research: the logic of logic models
Logic models are commonly used in evaluations to represent the causal processes through which interventions produce outcomes, yet significant debate is currently taking place over whether they can describe complex interventions which adapt to context. This paper assesses the logic models used in healthcare research from a complexity perspective. A typology of existing logic models is proposed, as well as a formal methodology for deriving more flexible and dynamic logic models.
Various logic model types were tested as part of an evaluation of a complex Patient Experience Toolkit (PET) intervention, developed and implemented through action research across six hospital wards/departments in the English NHS. Three dominant types of logic model were identified, each with certain strengths but ultimately unable to accurately capture the dynamics of PET. Hence, a fourth logic model type was developed to express how success hinges on the adaption of PET to its delivery settings. Aspects of the Promoting Action on Research Implementation in Health Services (PARIHS) model were incorporated into a traditional logic model structure to create a dynamic “type 4” logic model that can accommodate complex interventions taking on a different form in different settings.
Logic models can be used to model complex interventions that adapt to context but more flexible and dynamic models are required. An implication of this is that how logic models are used in healthcare research may have to change. Using logic models to forge consensus among stakeholders and/or provide precise guidance across different settings will be inappropriate in the case of complex interventions that adapt to context. Instead, logic models for complex interventions may be targeted at facilitators to enable them to prospectively assess the settings they will be working in and to develop context-sensitive facilitation strategies. Researchers should be clear as to why they are using a logic model and experiment with different models to ensure they have the correct type.
KeywordsLogic models Program theory Implementation models Complexity Complexity science Complex interventions Facilitation Context
Medical Research Council
Promoting Action on Research Implementation in Health Services
Patient Experience Toolkit
The case for process evaluations is now well-established in healthcare research following publication of the Medical Research Council (MRC) guidance in 2008 . The MRC guidance advocated for the greater use of qualitative, process evaluations to produce theory of how interventions work (sometimes referred to as “programme theory” or “theory of change”), said to be necessary to ensure their optimal development and use . Yet, questions are increasingly being asked of whether the MRC guidance does enough to address the challenges involved in evaluating complex interventions [2, 3, 4, 5, 6]. Scholars influenced by complexity science have argued that the MRC guidance is appropriate only for complicated interventions that work roughly the same way in different settings. Complex interventions, by contrast, seek to change social systems such that pre-existing contextual factors shape the form that they take [2, 3, 4, 5, 6]. Feedback loops provide the opportunity for those delivering and receiving the intervention to adapt it to context, potentially changing the activities to be delivered and the outcomes that are produced . An example of this dynamic can be found in public health in the case of school-based nutrition education interventions. A qualitative exploration of their work found that nutritionists’ practices varied according to their past experiences and each school setting and they strategically adapted interventions to keep people engaged, exhibiting an intuitive awareness of the needs and goals of students and teachers. This implies a blurring of the boundaries between interventions and context that is difficult to reconcile with traditional evaluation techniques .
Significant debate has taken place about the methods suitable for designing and evaluating these more complex interventions. Greater focus on the developmental stage of interventions is said to be necessary and formative methods that allow interventions to adapt on implementation are increasingly advocated [4, 5]. Yet, while the need for theoretical evaluation of complex interventions continues to be recognised, the role of logic models in this new research paradigm is unclear.
Logic models are assigned the role, in process evaluations, of representing the underlying theory of interventions in simple, diagrammatical form (see Additional file 1: Appendix 1 for a glossary of key terms related to logic models). For their advocates, they can be useful to help evaluators develop understanding of exactly how interventions produce outcomes [1, 3], to organise empirical data and specify process and outcome measures for the purposes of evaluation  and/or to provide a talking point for stakeholders to forge consensus on the need for change and how to go about it . Logic models can also be useful to demonstrate programme logics to funders and aid the process of knowledge transfer whereby research findings are applied outside of initial test sites . Yet, existing guidance on logic modelling in healthcare research pays very little attention to the interaction between interventions and context [2, 3, 4, 5, 6]. Some have concluded that logic models have reached the limits of their use [4, 8, 10, 11, 12, 13, 14, 15, 16].
The utility of logic models has been a frequent topic of BMC Medical Research Methodology [8, 11, 17, 18]. Addressing the aforementioned debate, Greenwood et al. question whether logic models can represent the dynamics of complex interventions that adapt to context, stating that “no matter how sophisticated, a logic model alone is not sufficient, as complexity cannot be understood purely through qualitative description” . We feel this is too quick a rejection of qualitative logic models. While logic model types that are currently dominant in healthcare research may be inadequate for describing complex, adaptive interventions, more flexible and dynamic types are possible. We demonstrate this with reference to our experience of developing and evaluating a Patient Experience Toolkit intervention. A typology of logic model types is proposed based on a scoping review of the literature, along with a formal methodology for developing dynamic models, referred to as “type 4” logic models. We hope this will help researchers to a) know which logic model type to use when evaluating interventions and b) overcome the challenges of modelling complex interventions.
Modelling a patient experience improvement toolkit intervention
Various logic models were tested as part of an evaluation of a Patient Experience Toolkit (PET), developed to guide healthcare professionals through a facilitated process of reflecting and acting on patient experience data1. This process includes stages for setting up a multidisciplinary team, reflecting on patient feedback and making changes using QI techniques. Six hospital wards across three NHS Trusts in the North of England were involved in the study, specifically chosen to present very different contexts for the PET intervention. Ward teams and patient representatives worked with researchers in an action research project to implement and refine PET over the course of a year. The task of the evaluation was to develop generalisable theory of how the intervention works as a whole, using a logic model approach.
Roles – The roles and responsibilities of ward team members differed in accordance with their willingness and capacity to engage, with the action researchers adapting their role to fit each team. They carried out some of the facilitation tasks for one team which a ward manager or patient representative had done for another team.
Interaction between the facilitation and moderators – The action researchers could also be seen responding to the presence of moderators existing in each ward setting. For example, coaching was particularly prominent when ward cultures were perceived to be unsupportive of improvement work, characterised by low staff engagement, wellbeing and self-efficacy. Low organisational support and a lack of escalation channels could also be overcome by the action researchers establishing relationships with corporate staff. The initial logic model does not model this dynamism between the facilitation and moderating factors, implying they were experienced only as enablers or barriers.
Irregular patterns of proximal outcomes – Some of the proximal outcomes identified in the logic model, such as the emergence of a shared agenda, action planning/implementation and meaningful involvement of patient representatives, were apparent on all wards and can therefore be considered core “mediators” of PET. Yet, other proximal outcomes were linked to the action researchers’ efforts to overcome moderators that were specific to particular ward settings, such as improved ward culture or connections between actors.
Proximal outcomes influencing later success – Finally, the initial logic model does not show how the emergence of the proximal outcomes could strengthen the work of the project. Initial improvements and the emergence of proximal outcomes could create a more receptive context for the intervention, making later improvement efforts easier to implement.
A typology of logic models used in healthcare research
Type 1 and type 2 logic models
It is common for researchers using type 2 logic models to recognise that their models poorly express intervention dynamics. Caveats can be included as to how logic models should be interpreted in the narrative that sits alongside any model (see Additional file 1: Appendix 1). For the PET intervention, the narrative would have to both explain the contents of the model and warn against a linear and rigid interpretation of it. Yet, this begs the question of whether alternative logic model types exist that could give a better elucidation of the PET’s dynamics. This would lessen reliance on the narrative and enable it to focus on explaining the core aspects of the intervention as captured in the model.
Type 3 logic models
These type 3 logic models can be useful to develop and test hypotheses related to the precise relationships between intervention components and outcomes. They are also often practitioner-oriented, used as part of consensus-building exercises about the requirement for change and how to go about it . However, the focus of type 3 logic models is interventions rather than intervention settings and they are unable to accommodate interventions taking on a different form. Some type 3 logic models do incorporate “alternative causal strands”, enabling them to convey how interventions work in different settings [3, 12, 13]. Yet, the level of variation they can accommodate is limited to the number of strands they include. The question remains whether logic models can describe interventions which potentially take on a different form every time they are delivered.
Type 4 logic models?
While the PARIHS model is not a logic model as such, the centrality it assigns to facilitation and context make it relevant to PET and indeed complex interventions in general which adapt on delivery through feedback loops . In addition, while PARIHS has been used retrospectively to explain project outcomes, it can be used prospectively to plan implementation strategies before projects commence . This is significant as it points to a potential new role for logic models of informing the development of context-sensitive facilitation strategies, as opposed to the traditional role of providing precise guidance as to how to act. In the next section, we incorporate aspects of the PARIHS model into a traditional logic model structure to create a type 4 logic model.
Using PARIHS to model the PET intervention (Fig. 6)
Discussion: Principles for advancing the field of logic models
Create logic models through robust qualitative research
To create a robust logic model, we recommend that researchers adopt a framework approach to qualitative data analysis  to manage and analyse data across multiple intervention sites. Logic model categories (intervention mechanisms, moderators and outcomes) can inform the columns of the framework matrix and each intervention site can be assigned a separate row, enabling potentially vast data to be organised and analysed so that model contents can be tested and refined in light of emergent categories and themes. This approach can be entirely inductive or combine deductive elements with prior theory informing the initial contents of the model. Testing against empirical data is crucial to ensure the robustness of the model.
In the case of interventions that are already known to be complex and adaptive, researchers can adopt an outline of our type 4 model and develop its contents in relation to the data contained within the framework matrix. Yet, it is likely that the level of complexity of an intervention will be unclear before it is tested, in which case researchers can experiment with different logic types as they are analysing their data. In our case, the PET intervention initially seemed complicated, with multiple component parts interacting in roughly similar ways . Only by creating a type 2 logic model and testing and refining its contents did the full complexity of PET become apparent. We found that the type 2 logic model failed to convey 1) differences in the roles of facilitators and intervention users/recipients across settings 2) how the facilitators’ response to contextual moderators changed the shape of the intervention 3) irregular patterns of outcomes across different settings and 4) the influence of early proximal outcomes on the intervention’s later success. If interventions are found to share these characteristics, then a type 4 logic model will be necessary.
Use narrative to describe intervention logics
Narrative will always play a fundamental role describing the theoretical basis of interventions and explaining the content of logic models. If the narrative surrounding a model has to explain the inadequacy of a type 1, 2 or 3 type logic model to describe an intervention’s dynamics, this is a further sign that a type 4 logic model is necessary. In our case, we also listed the core intervention mechanisms in the model instead of a precise list of activities and resources to allow for greater variation in how interventions play out across different settings. This is consistent with a view of interventions as constituted by underlying mechanisms that are sensitive to context  or functions as opposed to precise forms, allowing for variation across different settings [2, 3]. The underpinning narrative should describe and reference the evidence-base for the mechanisms/functions, as is common in all logic model types .
Use diverse shapes and arrows to model dynamic relationships and contingencies
While one type 4 logic model shape has been proposed here, we encourage researchers to experiment with it to ensure a fit with their interventions. Wider policy analysis literatures highlight the potential of different types of lines and arrows to express dynamic relationships and contingencies in logic models, while it may be possible to use diverse shapes such as triangles and circles instead of a Venn diagram [3, 12, 13]. The key issue to remember, however, is that type 4 logic models must convey a dynamic relationship between the facilitation of an intervention, the users/recipients of the intervention, contextual moderators and outcomes. It is this level of dynamism which demarcates the approach from other logic model types. Revisions to our proposed type 4 logic model shape must therefore replicate how it displays the influence of context on intervention delivery and the functions of the Venn diagram and the dotted, double headed arrows in some form. The use of the Venn meant it was possible to model variation in terms of the roles and relationships of project facilitators and intervention recipients; the dotted, double headed arrows conveyed how certain proximal outcomes were contingent on the form the intervention took on and how they could improve the intervention’s functioning at a later stage.
Include the full spectrum of contextual moderators
Type 4 logic models are as much about context as they are interventions, consistent with the view of interventions as “events in systems” . In our study, six diverse hospital wards/departments were involved, providing insight into the effects of context on the PET intervention. We drew upon frameworks of context to differentiate between moderators exerting influence from the “inner” and “outer” ward contexts while outcomes were categorised as “core” or “contextual”. An alternative would have been to use the micro/meso/macro distinction . Either way, displaying the full spectrum of contextual moderators is vital to inform conversations about how interventions may be adapted to context or how and at what level the receptiveness of context may be improved.
Target logic models at facilitators
Because type 4 logic models are designed for complex, adaptive interventions which change shape across different settings, the traditional uses of logic models to forge consensus among stakeholders or provide precise guidance as to how to act to produce positive outcomes are increasingly irrelevant. However, because complex interventions adapt to contexts through a flexible facilitation function, making them “inextricably linked” to implementation and context , a new role for type 4 logic models emerges: to guide how future users of complex interventions adapt them to context. While all logic models are accompanied with a narrative of some sort, in the case of type 4 logic models this can be tailored to inform facilitators’ assessments of context and to enable them to develop context-sensitive facilitation strategies. This may enhance the scale-up of complex interventions.
Incorporate differences of opinion
Finally, while we recognise that the type 4 logic models we propose will be less suitable for forging agreement among stakeholders than traditional logic models, accommodating differences of opinion may be more suitable for complex interventions given that the potential for disagreement increases with more complex problems . Here, it is interesting to note that some report logic models to have caused unnecessary friction when used to forge consensus over a proposed change  while others have warned they supress marginalised voices [12, 16]. In our case, stakeholders had different views on the PET document, with some viewing it as central to the intervention and others peripheral. Stakeholders also disagreed on the order of significance of the moderating factors: some downplayed the significance of staffing pressures while others argued that improvement work was not possible without addressing these first. Rather than resolve these differences or prioritise one over the other, our model allows for the possibility that both are right in different settings.
In this paper, we have proposed a typology of logic models, including strengths and weaknesses, to help researchers select between different logic model types in intervention research. In addition, we have outlined a formal methodology for developing more dynamic logic models than those which currently exist, incorporating aspects of the PARIHS model into a traditional logic model structure. These “type 4” logic models are capable of expressing interaction between interventions and context but some change to how logic models are used is required. Because type 4 logic models are designed for complex interventions which change shape across different settings, the traditional uses of logic models of forging consensus among diverse stakeholders and/or providing precise guidance as to how to act to produce positive outcomes are increasingly irrelevant. We propose that type 4 logic models should be developed and refined through rigorous qualitative research rather than consensus-building exercises. In addition, they should seek to guide future users of complex interventions to help them develop context-sensitive facilitation strategies. A benefit of this approach is that it may enhance the scale-up of complex interventions.
HS&DR commissioned the Yorkshire Quality and Safety Group of the Bradford Institute of Health Research to carry out the study
Thanks to the participants who generously gave their time to take part in the study. Thanks also to Claire Marsh and Rosemary Peacock of Bradford Institute of Health Research for contributing as members of the research team and to Beverley Slater of the Improvement Academy (IA) for granting permission to use an IA logic model.
This research was funded by the NIHR Health Services and Delivery programme (Ref: 14/156/32). The research was supported by the NIHR CLAHRC Yorkshire and Humber (www.clahrc-yh.nihr.ac.uk). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.
Availability of data and materials
Any requests for data should be directed to the corresponding author.
LS and RL designed the study as a whole. TM conceived the idea for the paper, collected and analysed the data and wrote the first draft. LS and RL commented on the first draft of the paper and TM actioned their comments. All authors read and approved the final manuscript.
Ethics approval and consent to participate
Ethics approval was required for the study and this was granted by the Yorkshire & Humber – Bradford Leeds Research Ethics Committee on 04/11/2016. The Health Research Authority granted approval on 25/11/2016. All participants gave informed, written consent to take part in this study.
Consent for publication
No individualised data is presented in this paper.
The authors declare they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 7.Bisset S, Potvin L, Daniel M. The adaptive nature of implementation practice: case study of a school-based nutrition education intervention. Eval Program Planning. 2010;12:004.Google Scholar
- 8.Lalor JG, Casey D, Elliott N, Coyne I, Comiskey C, Higgins A, Murphy K, Devane D, Begley C. Using case study within a sequential explanatory design to evaluate the impact of specialist and advanced practice roles on clinical outcomes: the SCAPE study. BMC Med Res Methodol. 2013;13:55.CrossRefGoogle Scholar
- 13.Funnell S, Rogers P. Purposeful program theory: effective use of theories of change and logic models. San Francisco: Jossey-Bass/Wiley; 2011.Google Scholar
- 15.Garrod B, Exley J, Harte E, Miani C, Newbould J, Saunders C, Ling T. An evaluation of the first phase of Q: engaging the founding cohort in a co-designed approach to healthcare improvement. 2016. https://www.rand.org/pubs/research_reports/RR1517.html. Accessed 08 March 2019.
- 16.Bakewell O, Garbutt A. The use and abuse of the logical framework approach: a review of international development NGO’s experiences. 2015. https://www.alnap.org/help-library/the-use-and-abuse-of-the-logical-framework-approach. Accessed 08 March 2019.
- 21.Krenn H. The step-by-step guide to evaluation: how to become savvy evaluation consumers. 2017. https://www.wkkf.org/resource-directory/resource/2017/11/wk-kellogg-foundation-step-by-step-guide-to-evaluation. Accessed 08 March 2019.
- 22.Wahlster P, Brereton L, Burns J, Hofmann B, Mozygemba K, Oortwijn W, Pfadenhauer L, Stephanie P, Rehfuess E, Schilling I, van Hoorn R, van der Wilt GJ, Baltussen R, Gerhardus A, Guidance on the integrated assessment of complex health technologies – The INTEGRATE-HTA Model. 2016. https://www.integrate-hta.eu/wp-content/uploads/2016/02/Guidance-on-the-integrated-assessment-of-complex-health-technologies-the-INTEGRATE-HTA-model.pdf. Accessed 08 March 2019.
- 24.Bennett B, Provost L. What’s your theory? Driver diagram services as tool for building and testing theories for improvement. Qual Prog. 2015:36–43.Google Scholar
- 25.Rehfuess EA, Booth A, Brereton L, Burns J, Gerhardus A, Mozygemba K, Oortwijn W, Pfadenhauer LM, Tummers M, van der Wilt GJ, Rohwer A. Towards a taxonomy of logic models in systematic reviews and health technology assessments: a priori, staged, and iterative approaches. Res Synth Methods. 2018;9:13–24.CrossRefGoogle Scholar
- 26.Wildschut, L. Theory-based evaluation, logic modelling and the experience of SA non-governmental organisations, PhD thesis, Stellenbosch University, 2014. http://scholar.sun.ac.za/handle/10019.1/86604. Accessed 30 Nov 2018.
- 27.Cochrane. Developing Logic Models, Cochrane Infectious Diseases, Effective Healthcare Research Consortium, 2016Google Scholar
- 28.Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10.Google Scholar
- 29.Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, Wahlster P, Polus S, Burns J, Brereton L, Rehfuess E. Making sense of complexity in context and implementation: the context and implementation of complex interventions (CICI) framework. Implement Sci. 2016;12:21.5.Google Scholar
- 32.Bate P, Robert G, Fulop N, Ovretveit J, Dixon-Woods M. Perspectives on context: a selection of essays considering the role of context in successful quality improvement. 2014. https://www.health.org.uk/publications/perspectives-on-context. Accessed 08 March 2019.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.