Background

In healthcare, scenario-based simulation increasingly involves multi-professional teams with the aim of enhancing the application of knowledge, embedding evidence-based practice and improving the performance of the team as a whole [1, 2]. In-situ simulation (ISS) enables teams to practice in the clinical environment where genuine care takes place, and has principally focused on low probability, high consequence events [3]. ISS may offer additional benefits to traditional ‘education-centre’ based simulation, enabling participants to problem solve within their own dynamic setting and facilitating the contextualised implementation of learning into practice [4, 5]. ISS can be a mechanism to explore the interplay between and within micro systems (individual, team and task factors) and macro level phenomena (hospital departments, facilities and systems; [6]), enabling latent safety threat (LST) detection leading to improvements in safety and performance [1, 7]. ISS models are relevant to various healthcare settings as they are flexible to different contexts [8]. There may also be economic benefits associated with ISS when compared to simulation training undertaken in dedicated simulation suites [4, 9]. During the COVID-19 pandemic, ISS interventions have been used to test the ability of healthcare teams to effectively implement use of personal protective equipment (PPE), test infection control guidelines and operational readiness of intensive care units and operating rooms [10,11,12,13,14].

Haji et al. [15] developed a theory-based, iterative, programmatic framework for simulation interventions adapted from the Medical Research Council (MRC) framework for complex interventions [16]. Multiple theories were identified which are applicable to stages of simulation development: learning or instructional design theories for intervention development and modelling; cognitive and behavioural science where participant behaviour change or transfer of knowledge and skills is required; socio-cognitive theories for team based training; and implementation and complexity science to explore integration and contextual factors [15]. Although there is a paucity of conceptual, planning and evaluation frameworks which are solely focused on ISS, components of existing conceptual frameworks for general simulation training and existing literature reporting and evaluating ISS interventions may provide a useful starting point.

The National Simulation Development Project Report (The Higher Education Academy, Association for Simulated Practice in Healthcare (ASPiH) & Health Education England, 2014) identified that the use of ISS is increasing within the United Kingdom (UK) but requires more comprehensive and cohesive strategic and operational support to achieve the potential benefits offered. Efforts to standardise ISS terms and concepts and develop an integrated model of learning is required [2, 4, 17]. Previous reviews have focused on ISS for education and training of healthcare professionals [18], ISS in operating rooms [19], effect on patient outcomes [20] or have included ISS within a broader simulation approach within acute care settings [21], for caesarean section training [22] and obstetric emergency teams [23]. Reviews have highlighted a limited but promising evidence base for ISS and reported high variability of ISS approaches to design, delivery, and evaluation [18,19,20,21,22,23].

The aim of this study was to explore the current evidence, theories and concepts associated with ISS across all areas of healthcare and develop a conceptual model to inform future ISS research and best practice guidance.

Methods

This study involved three components:

  1. 1.

    A scoping review of the current literature relating to ISS

  2. 2.

    Consultation with stakeholders from clinical and health education organisations

  3. 3.

    Development of conceptual and logic models of ISS interventions

Component one: scoping review

We selected a scoping review as they are designed to explore the extent, range, and nature of the emerging evidence [24,25,26]. A scoping review would enable us to categorise the concepts and theoretical approaches for ISS in healthcare and develop theories about how distinctive mechanisms of ISS (‘natural teams in natural settings’) have the potential for addressing specific learning and clinical needs for the individual, team and various organisation levels. We conducted a scoping review of the published ISS literature following the framework by Levac et al. [27].

The scoping review aimed to address the following questions:

  • What types of ISS healthcare interventions have been evaluated and reported?

  • What were the reported objectives, design and outcomes reported in ISS studies?

Studies were included in they: 1) included any type of healthcare professionals and/or healthcare support workers as participants; 2) reported ISS interventions as part or whole of an intervention; 3) were conducted in any healthcare setting including primary or secondary care. The scoping search was limited to OECD countries. Studies which conducted simulation in laboratory, off site or training facilities, were solely focused on pre-registration or undergraduate participants were excluded.

A two-step search strategy was used, the initial search was conducted in September 2018 and updated on 10th December 2020. The search included papers published in the English language from inception to December 2020. Due to time restrictions, we limited the search to three database: Medline, OpenGrey (now archived in the DANS EASY data archive) and Web of Science.

Broad search terms were developed and refined by the study team [27] which included ‘simulation; drills; simulation training; patient simulation’ AND ‘in situ; clinical care; practice; real world; point of care; workplace’ (Additional file 1). Reference lists from all identified studies were reviewed for additional citations to enhance rigour of the approach. Searches and screening were completed by one reviewer (KE). Full text review was completed by KE and members of the study team. Agreement was reached through group discussion. The process of charting the data was conducted by two reviewers (KE, LB) from included papers using a standardised data extraction form to record characteristics of the included studies and the key information relevant to the review question [28].

Data were analysed using a descriptive approach, summarising the data and study characteristics. Data were then organised into categories based around the different motivations, concepts and theoretical approaches for ISS as identified in previous reviews [27, 29]. This was an iterative process completed through discussion with the project team and further refined through the stakeholder engagement.

Guidelines: The scoping review was conducted following the methods described by Levac et al. [27]. The protocol development and scoping review reporting was guided by the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist [29].

Component two stakeholder engagement

Levac et al. [27] recommend incorporating consultation with stakeholders as a required knowledge translation component of scoping study methodology. We sought expert feedback on the preliminary scoping review findings to build on the evidence and offer a higher level of meaning, content expertise, and perspective to inform the conceptual model [27, 30]. An ISS workshop was held at the Association of Simulated Practice in Healthcare (ASPIH) national conference (2018). As part of the workshop, attendees were presented with the preliminary study findings. A ‘World Café’ method [31, 32] with four × 10 min small group discussions which focused on the ISS categories identified from the preliminary findings. Participants were presented with a series of questions What are the benefits (if any) of ISS over other methods?

  • What type of enquiry is more suited to ISS and why?

  • What type of ISS design should be used for different types of enquiry?

  • How can ISS support psychological safety of participants?

Data from flipcharts, posters and facilitators field notes, were transcribed and summarised.

Attendees from higher education and healthcare provider institutions including multi-professional clinical and managerial staff groups attended the workshop.

Component three: developing a conceptual model of ISS ion healthcare settings.

Data from the scoping search findings, stakeholder discussion and debate were synthesised and developed into a conceptual model of the concepts and theories associated with ISS in healthcare settings to address specific contextual needs. Logic models were then developed to identify short, medium and long-term outcomes that are linked to the key activities of ISS mechanisms [33].

Results

The results of the three components will now be presented.

Component one: scoping review

The search identified 4237 papers which were assessed for eligibility via title and abstract. Eighty papers were retrieved for full text assessment, following discussion and agreement with the review team, 3 reviews and 35 studies were selected for inclusion (Fig. 1, Additional file 2). Papers were from Australia, Canada, Denmark, France, Hong Kong, Israel, Japan, Netherlands, Norway, Sweden, Switzerland, UK and the US and publication dates ranged from 2008 to 2018.

Fig. 1
figure 1

PRISMA diagram – Papers describing ISS interventions in healthcare settings

Full size image

Systematic reviews of ISS interventions

An umbrella review of simulation-based training for nursing education and practice [34] included one paper focused on ISS [35]. ISS was found to foster improved competencies related to patient safety and collaborative practice such as interdisciplinary communication and teamwork, as well as identifying and correcting actual clinical safety issues. Rosen et al. [18] conducted a systematic review of ISS interventions that included 29 papers which were focused mainly on surgical or maternity care. Most studies were rated as low quality. Approaches to design, delivery, and evaluation of ISS were highly variable across studies. Formal needs analysis was rarely used to develop simulations, there was little evidence of formal training or performance management for facilitators and few programmes reported meaningful evaluations of programme effectiveness. However, a positive impact of ISS on learning and organisational performance was demonstrated in a small number of studies. Owei et al. [19] conducted a systematic review of operating room ISS which included 19 papers, describing its application for a variety of purposes and in a variety of settings, all premised on the potential to offer unique advantages over other types of simulation. One randomized controlled trial (RCT) comparing ISS to off-site simulation found few significant differences. One large-scale outcome study showed improved perinatal outcomes in obstetric care. The authors concluded that although ISS theoretically offers certain advantages over other types of simulation, especially in addressing system-wide or environmental threats, its efficacy has yet to be reliably demonstrated.

Characteristics of studies reporting ISS interventions

The remaining papers reported ISS interventions conducted in maternity care settings, paediatrics, neonatal, trauma and emergency departments, resuscitation response teams, nursing, mental health and primary care settings. Papers reported various methods for evaluating ISS interventions including: RCTs, surveys, focus groups, cohort studies, observation (pre and post intervention studies) and clinical audit. Papers reported ISS interventions lasting from 15 min to 12 h, describing exercises that were announced and unannounced, and providing single or repetitive ISS sessions. Most papers reported that ISS had been developed by clinical educators and senior clinical staff. Various frameworks for designing, conducting and evaluating ISS were utilised, including systems science models, such as the Systems Engineering Initiative for Patient Safety (SEIPS) model [36], and improvement science models such as the ‘plan, do, study, act’ (PDSA) cycle [37].

Studies aimed to evaluate ISS on various outcomes (Table 1), including:

Table 1 Objectives and outcome measures of the studies in the scoping review
Full size table

Component two: stakeholder workshops

The stakeholder group indicated formative rather than summative ISS assessments have the potential to improve learning and encourage Healthcare Professionals (HCPs) to identity their own training needs. Formal teaching and traditional simulation assessment programmes often fail to prepare staff for the ‘real world’ and ISS may help apply learning and reduce the theory-to-practice gap. However there are no robust strategies or non-technical skills standards to assess behaviours, attitudes and communication in changing complex and dynamic settings. It would be very difficult if not impossible to present all participants with equal opportunities to demonstrate their skills during ISS. It was suggested that ISS assessment interventions should:

  • aim for realism rather than hyper realism; attempts to exaggerate or exacerbate scenarios should be avoided.

  • have clear learning objectives; the object of assessment (i.e. individual, team and/or system) should be defined as each requires a different approach and assessment technique.

  • have objectives informed by training needs analyses which form part of an integrated curricula employing numerous learning and assessment approaches. The ‘gaps’ in learning which ISS can address should be clearly identified.

  • ensure information including learning resources is standardised and available for staff to access prior to ISS implementation.

  • ensure ISS is conducted within supportive learning cultures which have well defined supportive and training packages.

  • provide facilitators with specialist training including peer assessment.

  • ensure ISS interventions involve creative planning to avoid being continually cancelled in busy departments; consider alternative setting such as staff areas and social spaces.

Stakeholders suggested that ISS has potential to complement traditional investigation approaches but that not all clinical incidents were thought to be appropriate for ISS exploration. Comments provided during the workshop were formed into a series of questions to help HCPs and healthcare educators decide if ISS is an appropriate intervention to aid in the investigation of clinical incidents with a view to foster deeper learning of the factors involved and how these might be mitigated in future (Additional file 3).

Component three: developing a conceptual model of ISS in healthcare settings.

The scoping review findings and stakeholder feedback identified various principles, theories and approaches for ISS in healthcare settings (Fig. 2).

Fig. 2
figure 2

In-situ simulation principles, theories and approaches used in healthcare settings

Full size image

Four distinct concepts were identified (Table 2, Fig. 3):

  • ISS to understand why errors have occurred

  • ISS to design and testing new equipment, spaces, pathways, systems and procedures

  • ISS to practice skills and develop competency

  • ISS to assess, evaluate and improve performance

Table 2 Functions of in-situ simulation in healthcare
Full size table
Fig. 3
figure 3

Conceptual model of ISS for healthcare settings

Full size image

Discussion

The four concepts of ISS are presented alongside logic models (Figs. 4 and 5) with discussion of how each ISS approach may be developed to address specific contextual needs.

Fig. 4
figure 4

Logic model: Design ISS

Full size image
Fig. 5
figure 5

Logic model: Practice ISS

Full size image

In-situ simulation: understand why events have occurred

Learning from adverse incidents should move beyond attributing cause wholly to human failings and more towards investigating the role of the system in which humans operate [69,70,71]. Root cause analysis (RCA) investigations promote a systematic approach to investigating serious incidents, although RCA may impede organisational learning as it can restrict explanation to single causes and is often over reliant on individual testimonies or medical notes [72]. Alternative ways of capturing less reductive incident ‘stories’ can encourage reflection and wider organisational learning, considering the interaction between a range of systemic factors which contribute to incidents. ISS may provide a useful mechanism for such inquiry, assisting clinicians to discover potential interacting components and identify additional LSTs [65, 73,74,75].

As part of a wider approach which aims to encourage healthcare organisations to progress from a ‘blame culture’ to a just, learning culture [71], embedding simulation activities underpinned by Human Factors principles can help to focus on the organisational, procedural and contextual influences on clinical reasoning and actions. Participants of ISS interventions should be encouraged to observe, reflect, ask questions, raise concerns, challenge practices and clarify actions while taking context and complexity into account.

Patient involvement in the design and delivery of simulation training enables the patients’ experience to be expressed and considered. Many ISS scenarios have been developed to reflect ‘real life’ risks drawn from patterns in care, sentinel events, or concern from advocacy groups. However it maybe be possible to work in collaboration families involved in a serious incident to help facilitate learning opportunities and disseminate the findings [73]. This would involve careful attention to local sensitivity; where harm has occurred this of course may be distressing for staff, patients and families involved (particularly where organisations have a culture of appropriating blame and censure). The acceptability of co-designed ‘reconstruction’ via ISS scenarios from care histories should always assess the potential effect on patients and staff involved.

In-situ simulation: design and testing

Tools of safety and complexity science such as Discrete Event Simulation [76], Cognitive Task Analysis [77] and general system thinking models have been used to design ISS interventions to enhance patient flow, improve the design of clinical spaces, and identify LSTs within new emergency and obstetric departments. For example, Bender [6] scripted commonly encountered clinical scenarios to explore the functioning of a new perinatal facility. Scenarios challenged participants to test the new facilities under stressful conditions, focusing on identifying LSTs and making improvements which were more readily adopted into practice as they were driven the clinical teams. Medwid et al. [64] used ISS to test a new emergency department; numerous LSTs were identified and addressed prior to the department opening. ISS facilitated deliberate practice within the new space and helped orientate staff to the new facility. ISS has also been reported to assist the development and testing of new clinical services and procedures for cardiac services [66] and stroke thrombolysis pathways [78]. The MHRA guidance for medical devices [79] highlights that usability testing with representative users in a simulated environment or the actual environment of use can help identify which device features people find easy to use and which cause problems, thus determining whether the device is susceptible to user errors that could cause harm.

Design-driven improvement is a core Human Factors principle which can benefit healthcare organisations [80, 81]. The performance of a process is the emergent property of the whole interacting system which is complex, dynamic and situation specific [82, 83]. The ability to experiment and see what occurs through interactions, attunement and disturbances enables participants to question how things might be done differently, try out various options and consider possible unintended outcomes [84]. ISS can be designed to test the synergy or dissonance between micro and macro factors: task factors, organisational factors, internal environments and external environments [81]. The logic model proposed in Fig. 4 has been developed with reference to the current literature and approaches to ISS design and testing interventions. The model highlights the mechanisms of ISS which would be potentially amenable to standardised approach.

In-situ simulation: practice, developing capability and resilience

Organisational resilience is focused on understanding how healthcare organisations can deliver standardised, replicable and predictable services while embracing inherent variations, disruptions and unexpected events [85]. This involves designing, testing and improving the organisational systems that support rapid and adaptive responses to emergency situations. Deficiencies and LSTs in current systems can be identified and addressed through ISS activities as they enable a systematic examination to provide a realistic picture of work, contextualised in real time and place [7, 55, 62, 86]. ISS scenarios can also support HCPs to develop non-technical skills: task management; teamwork; situation awareness; problem-solving; and decision-making, while testing and probing real-world organisational systems [4, 43, 85, 87,88,89]. ISS to assist teams prepare, rehearse and practice for low frequency, high impact events was the most reported ISS activity included in the review and was often underpinned by a combination of established learning theories (Fig. 5).

Behavioural learning

Skills are developed through repetition. Learning and behaviour change occurs through feedback from the simulation activity, interaction between the task, environment, and the team. ISS provides opportunity for teams to identify solutions and take action to introduce and reinforce changes [90]. Video playback can support behavioural learning alongside a discussion of strategies, alternative approaches, personal experiences and emotional aspects. Teams and individuals can be supported to identify gaps in knowledge, challenge current procedures and identify and address system weaknesses.

Cognitive learning

Preconceptions are explored, and new or unexpected events are presented via the simulation activity to challenge precognitions [91]. Questioning and reflection help participants acquire deeper understanding, develop problem solving skills and new insights [92]. ISS enables learning from making mistakes in a way that would be inconceivable with actual patients; ‘thinking aloud’ helps participants reflect on why they took a particular course of action and present an opportunity to correct their actions [84, 93, 94]. ISS activities should provide a concrete experience, debrief with reflexive observation and conceptualisation, ideally followed by a second scenario for active experimentation [95]. Knowledge transfer is optimal when the learning environment matches the environment in which it will be applied [93]. ISS can help develop and maintain situational awareness [96], for example, awareness of vital signs, medication requirements, actions of other team members and equipment function. Comprehension of a current situation based on these elements, understanding their significance within the environment, and forming a holistic picture helps anticipate future actions and consequences [97].

Social learning

Learning through observing others and developing a shared understanding of roles and processes within a team which result in the desired outcomes. ISS enables interdisciplinary team practice while maintaining interaction with the environmental and system factors present during actual patient care events [39]. Individuals and teams explore the social processes involved in building shared expectations, establishing patterns of collective working and building trust between multidisciplinary teams [85]. An open and safe context encourages professionals from different backgrounds to acknowledge their strengths and address their weaknesses in a respectful and trustful manner [8, 85].

In-situ simulation: assess, evaluate and improve performance

Assessing competency through simulation is well established in the military, nuclear and aviation industries and is used in healthcare to assess clinical competency via objective structured clinical examinations (OSCE). Brunette and Thibodeau-Jarry [93] suggest that simulation, through the application of mastery learning theory can be used to formatively assess competency in clinical environments. Learners are required to achieve a level of mastery in particular essential skills and knowledge before progressing onto new or more advanced practice. Learners progress at their own pace and are provided with opportunities for deliberate practice via simulation. Again, ISS enables participants to operate and be assessed in a familiar work environment [59].

Miller’s pyramid for assessing clinical competence distinguishes between four different levels of competence: knows; knows how; shows how; and does. The level ‘does’ is described as the most accurate way to assess competence in actual clinical practice [98]. However, assessment in actual clinical practice has the potential to distress patients and can be problematic in terms of controlling variables such as task difficulty [59]. Sørensen et al. [99] stress the difference between simulation-based training and simulation-based assessment (SBA). In SBA, participants need to be well informed about the proposed activity and know what will be expected of them [87]. Careful attention needs to be paid to creating a safe learning environment. Validated metrics and standards need to be developed for individuals and teams. Strategies to assess non-technical skills during ISS have been developed for medical students, obstetrics and anaesthesiology and could be refined and adopted for other clinical specialties [100,101,102].

Griswold et al. [103] identify that for clinical procedures with clear chains of action and well-defined processes and standards, summative assessment via ISS is much simpler than in more “dynamic, multifactorial practices in which cognitive, procedural, and communication skills are simultaneously applied in a team environment” (Griswold et al. 2018, page 170). Measurement methods for assessing competencies involved in complex care processes are less well-defined, and further complicated when individual performance needs to be isolated from the wider team. Concepts such as ‘effective communication’ are subject to interpretation, and clinical outcomes may be attributed to concepts such as teamwork and coordination in addition to individual clinical skills and knowledge [103]. Criterion standards and benchmarks of quality performance need to be defined to reliably and accurately capture the individual performance which is linked to relevant outcomes. Strategies to define and control for difference in skill mix, staffing and capacity pressures are also required.

Simulation for training has been reported as effective when instructional features, such as the event-based approach to training (EBAT) are embedded within the simulation [104]. EBAT has been used in aviation and military environments and relies on the “a priori” embedding of multiple events into the scenario at different time intervals. These events are designed to enable participants to exhibit competencies and measure performance. Scenario development can be facilitated by performing a Cognitive Task Analysis to identify the cues expected to be used to perform complex tasks [104]. Data collection should capture ‘moment-to moment’ actions and behaviours to identify how performance can be improved.

Limitations

The scoping review sought to identify and categorise ISS approaches within a wide range of healthcare settings, however only three databases were searched. Selection and screening were completed by a single reviewer (KE) and the review protocol was not registered. The included studies were conducted across thirteen countries and the cultural context in which ISS was conducted was not explored within the review. Stakeholder feedback which were used to further inform the development of the conceptual model may not represent the whole picture of the concepts and mechanisms of ISS being conducted in healthcare and health education settings. However, we consider that the scoping review and conceptual model have highlighted key characteristics related to ISS and thus provided a useful starting point to develop more specific questions to addressed by a systematic review or primary research.

Conclusions

This paper provides an overview of the application of the use of ISS in healthcare settings. The recent literature highlights the heterogeneity in ISS objectives and the complex delivery landscape which has resulted in a lack of an integrated ISS approach across healthcare organisations. Most reported ISS interventions provide little evidence of formal development processes and lack validated outcome measures. ISS in healthcare is often underpinned by Human Factors (HF) principles which overlap and synergise with other approaches, methods and theories including non-technical skills development, quality improvement (QI) methods, and systems modelling. Many existing models and frameworks exist within a wide ISS curriculum as part of a general approach to simulation training. This can result in a lack of clear guidance to inform ISS designs. A conceptual model has been provided to inform discussion and debate about the objectives, feasibility and usefulness of ISS interventions to guide clinicians and educators. We have set out the learning mechanisms intrinsic within ISS and suggest the context in which these mechanisms can be actualised. We have highlighted the potential for ISS to improve the design of clinical spaces and equipment, develop team performance and healthcare systems resilience, and support clinical investigation and competency assessment. An ISS approach presents distinctive advantages to explore and improve clinical team and organisational functioning. In addition, the ability to address the practice-theory gap makes ISS an attractive approach for educators, managers and policymakers as part of quality and safety improvement strategies. However there are limitations associated with ISS, such as the impact on the clinical setting, the provision of an open learning culture and availability of relevant expertise. We strongly recommend that ISS is not introduced without due consideration of the specific objectives and learning needs it is proposed to address. Effectiveness of ISS has not yet been established and further research is required to assess the specific effect of particular ISS designs on clinical outcomes, learning outcomes, team performance, non-technical skill development, acceptability and perceived benefit. We encourage researchers, clinicians and educators to work collaboratively to rigorously design, develop, evaluate ISS interventions and disseminate the findings to further inform the evidence base. We recommend future systematic reviews are conducted to assess ISS intervention effectiveness to guide clinicians, researchers and educators to develop effective ISS interventions and provide useful guidance as they continue to address various clinical concerns by ISS interventions in dynamic settings.