Background

Improving patient safety is imperative for every health care organization [1]. Inter-professional teamwork is key for patient safety and team training is an effective strategy to improve patient outcome [2–4]. Nurses and doctors working in acute care experience critical clinical events of rapidly deteriorating patients in need of cardiopulmonary support. Mismanagement of deteriorating patients is the most common failure reported in patient-safety-related Hospital deaths in England [5]. Recent reports suggest that hospital staff members feel inadequately prepared, perceive deficits and a high level of anxiety when managing cardiopulmonary arrest situations [6–8]. Optimal management of these events requires knowledge, technical skills, teamwork and can be enhanced by inter-professional training of hospital staff. Recently published studies show a good impact of simulation training for technical as well as non-technical skills [9–17]. In addition to teamwork and individual skills, patient safety can be improved through the identification and correction of latent safety threats [18–20].

In-situ simulation is a relatively new strategy with emerging data to support its efficacy. Recently published reports have discussed how to implement efficient simulated team training, [15, 21] but best practices for the design, delivery and implementation of an in-situ program have yet to be established [21–26]. A programmatic approach to training and assessment based on system thinking is required for a sustained improvement of team performance and patient safety [21, 27]. We developed an inter-professional in-situ simulated team and resuscitation training program (iSTaRT) for patient safety according to the framework for curriculum development by Kern [28]. The purpose of this prospective study is to describe the development, implementation and impact of an inter-professional in-situ simulated team and resuscitation training program designed with a programmatic approach based on Kern's six steps.

Methods

Setting

The Children's Hospital of Lucerne is a tertiary teaching hospital in the heart of Switzerland with general paediatric and paediatric surgery departments. Before implementation of iSTaRT, regular continuing education for doctors and nurses included lectures, journal club, small learning groups and low-fidelity simulation (paediatric and neonatal resuscitation). Driven by a small highly motivated group of staff members, an inter-professional project group was launched to design and implement a simulation-based team and resuscitation training program to improve patient safety during future critical events of rapid deteriorating patients in need of cardiopulmonary support at the Children’s Hospital of Lucerne. The group was led by 3 staff members with experience in simulation training (KZ, IBH, MS), whereas other group members did not have knowledge regarding simulation-based training. Two co-authors of the study with expert knowledge in designing and implementation of an in-situ simulated team training program were invited to be external consultants (MA, MB) [22]. The aim was to select a programmatic approach for curriculum development to improve the chance for successful design, implementation and acceptance of the training program and to prospectively evaluate the impact [21, 27]. According to published evidence there is a 6- to 12-month learning curve in the implementation of a simulated training program and therefore we selected a 2-year study period [22].

Programmatic approach for curriculum development

Kern’s framework for curriculum development for medical education was selected due to the short, practical and general approach [28]. Grounded in systems thinking it assumes curriculum development is a continuous, dynamic and interactive process that may start at different steps, is not always linear and the process never really ends [28]. Systems thinking states that changes in a part of a system will always cause the whole system to change, and that change developed through a programmatic approach is more likely to be sustainable [29, 30]. The six-step approach of Kern includes: i) general needs assessment to identify the problem, ii) focused needs assessment of targeted learners, iii) goals and specific measurable objectives, iv) educational strategies, v) implementation, vi) evaluation and feedback to stakeholders (Fig. 1).

Fig. 1
figure 1

Programmatic approach. Programmatic approach using the six-step approach of Kern for curriculum development [28]

Full size image
Fig. 2
figure 2

Close the loop between training and reality. Close the loop between simulated training sessions, experienced real critical events and system-based factors (latent safety threats)

Full size image

Kern’s step 1 and 2: General and specific needs assessments were conducted as independent cross-sectional surveys. The purpose of the general needs assessment was to determine self-evaluated preparedness, comfort and anxiety regarding performance in a possible future critical event among inter-professional staff members at the Children's Hospital Lucerne (5-item questionnaire). Classification of answers was done with a 5-point Likert scale. The specific needs assessment was a questionnaire including pre-defined topics and an option for free-text responses asking regarding training needs of technical skills, non-technical skills (teamwork) and specific medical conditions.

Kern’s step 3: According to Kern's six-step approach for curriculum development we used the results of the needs assessments to define the goals and objectives of the training program [28]. Technical skills as correct bag and mask ventilation, cardiac compressions and knowledge of emergency medication; non-technical skills as role allocation, leadership and communication (close-loop communication) and attenuation of anxiety regarding participation in future critical events were defined as learning objectives. In addition, system-based factors are well known as latent safety threats in every health care organization and there is evidence that system redesign may help to reduce errors in hospitals [31]. Therefore, detection of latent safety threats - defined as factors that make errors more likely or more dangerous [32] - was selected as a specific measurable outcome of our training program. In order to assess the impact of the program on the detection of latent safety threats all debriefing sessions included the question “Do you feel that specific system-based factors had an impact on your (or the teams) performance?” Feedback forms after the training session and after real events also prompted participants to consider latent safety threats and document possible solutions. Latent threats identified were shared with department heads. All implemented changes made due to identification of system-based factors during an iSTaRT session were prospectively recorded.

Kern’s step 4: We developed iSTaRT based on the concept of in-situ simulation-based crisis resource management (CRM) training as educational strategy [21–25]. We used a number of established learning theories to optimize our conceptual framework [24]. “Challenge participants to the edge”, “facilitate critical reflection” and “motivate with reality and context” were guiding statements for the facilitators. Each training session consisted of an introduction to CRM principles, a simulated scenario and a structured debrief over 2 h. Simulated scenarios were derived from real critical events and according to the collected needs assessment data. To provide care as realistically as possible participants were asked to perform as in reality including airway management, cardiopulmonary resuscitation, defibrillation, drawing-up and administration of medications. Initially, we used a low-fidelity mannequin (Neonatal Resuscitation Baby, Laerdal®) and, after 6 sessions a high-fidelity mannequin (Pediatric HAL 1 year, Gaumard®) aiming to further improve the realistic context of the training. The replacement was mainly in response to the good experience of some of the facilitators with high fidelity simulation in past [22]. Purchase of the high-fidelity mannequin was possible thanks to managerial buy-in with the project. The structured debriefing lasted around 1 h and was mainly based on the 3D model of debriefing and debriefing with good judgment [23, 24, 33, 34]. Facilitators of the debriefing guided and empowered participants to reflect openly on their own behaviors, to challenge own beliefs and frames and to conceptualize new principles and actions. Facilitators of the debriefing were trained within the PAEDSIM train-the-trainer programme (www.paedsim.org).

Kern’s step 5: Monthly training sessions were embedded within the work day of participants. Usually, the session was realized before or at the end of participant’s shift facilitating participation and raising acceptance. Due to restricted resources, we commenced the iSTaRT program just in our highest acuity wards with the highest risk of occurrence of critical clinical events: the emergency department, combined neonatal and paediatric intensive care unit, intermediate care unit. According to the principles of in-situ simulation all training sessions were completed on the regular unit with real inter-professional team members

Kern’s step 6: Assessment of training sessions (Fig. 2): After scenario and structured debriefing, all participants received an anonymous feedback form to assess the training session and its impact on their clinical practice. The feedback form contained six questions answered on a 5-point Likert scale [22]. Van der Vleuten’s framework of a programmatic approach to evaluate multiple sets of assessment longitudinally guided our assessment strategy of real events during the second half of the reported time of the programme [35]. TeamMonitor, a modified version of the Mayo High Performance Teamwork Scale published with a good reliability (kappa = 0.86) and content validity for CRM training, was used as the assessment tool [36]. TeamMonitor was selected due to the reported feasibility and low use of resource requirements compared to video analysis or expert observation [36]. As a 9-item team-based self-assessment tool capturing aspects of leadership, role clarity, communication, resource utilization and situational awareness, every item is scaled as 0 (never/rarely), 1 (inconsistently), 2 (consistently) or not applicable. The definition of training needs was selected after discussion within the facilitator group. We determined that when a desired behavior was experienced as rare or inconsistent by one third or more of participants, then further improvement was required. Therefore, items scored ≤1 in more than 33 % of response were considered to be targets for training, and in more than 50 % to be require urgent attention in further training sessions.

A combination of quantitative and qualitative methodology was utilised. Descriptive statistics were used for overall responses. General needs assessments and feedback forms (5-point Likert scales) were consistent with independent ordered categories. Scores ≤2 were defined as very low (score 1) or low (score 2), scores ≥4 as high (score 4) or very high (score 5) competence. Scores ≥4 were designated as desirable and/or good impact, scores ≤2 were problematic and/or showing no impact. Physicians and nurses responses were compared using chi-square tests with two degrees of freedom. Items scoring ≤1 in the TeamMonitor assessment tool were compared for single as well as for all events together using chi-square test with two degree of freedom. Pearson’s p value <0.05 was considered as statistically significant with a confidence interval at 95 %. All authors had full access to all data in the study and take responsibility for the integrity and accuracy of the analysis. Anonymous questionnaires, as a standard part of an educational program, did not require ethical approval according to the institute’s ethical committee, as well as according to the ethical guidelines of the British Educational Research Association (BERA) [37].

Results

Needs assessment

In August 2011, 121 staff members (48 % of all staff members) participated in the general needs assessment. 82 (41 % of nursing staff) of the 121 respondents were staff nurses and 39 (71 % of physicians) were paediatricians including subspecialists and paediatric surgeons. Results of the general needs assessment are shown in Table 1. Whilst the majority of both professions evaluated their competence and preparedness to detect a deteriorating patient highly, less than 50 % of surveyed staff felt confident to manage a critical deteriorating patient in need of cardiopulmonary support with slightly more nurses feeling confident compared to medical staff.

Table 1 Results general needs assessment. Self-evaluated preparedness, comfort and anxiety regarding performance in a possible future critical event among inter-professional staff members at the Children’s Hospital Lucerne
Full size table

In October 2011, 124 staff members participated in the specific needs assessment (49 % of all staff members). 32 were physicians (58 % of all staff physicians: paediatricians including subspecialists and paediatric surgeons of all level of expertise), 92 nurses (46 % of nursing staff). The result of the specific needs assessment regarding knowledge and technical skills, specific clinical situations and management of critical events are shown in Table 2. In contrast to the results of the general needs assessment, nurses perceived higher needs for education, reaching statistical significance for cardiac arrest/defibrillation, emergency medications/algorithms, multiple trauma and team concept/organisation/crisis resource management (CRM) principles (Table 2).

Table 2 Results specific needs assessment. Questionnaire including pre-defined topics and an option for free-text responses asking regarding training needs of technical skills, non-technical skills (teamwork) and specific medical conditions among inter-professional staff members at the Children’s Hospital Lucerne. Only topics reported by more than 50 % of participants (nurses and physicians) are shown
Full size table

Feedback responses

95 staff members participated in 20 simulation scenarios which took place on the 3 units between January 2012 and February 2014. 50 of the 95 participants (53 %) were staff nurses and 45/95 (47 %) were physicians (paediatricians including subspecialists, paediatric surgeons and anaesthetists). Two to three physicians and two to four nurses participated in each scenario. 86/95 (91 %) participants returned their questionnaires. Results of the feedback questionnaire regarding self-perceived impact on improvement on teamwork, technical skills, knowledge and anxiety for future critical events are shown in Table 3. There was no statistical significant difference between nurses and physicians.

Table 3 Participants self-perceived impact of an iSTaRT-session
Full size table

Between January 2013 and February 2014, 8 real critical events occurred in the emergency department. 35 involved team members (nurses and physicians including paediatricians, paediatric surgeons and paediatric intensivists) answered the TeamMonitor questionnaire regarding team performance (mean 4.4 members/event; minimal 2, maximal 6 members). Understanding of his/her role and communication (loud verbalizing of activities involving the patient and closed loop communication) were assessed as urgent gaps with >50 % scores ≤1 (Table 4). Leadership (recognition of the leader and balance between command authority and team member participation) and response to potential errors or complications were assessed as gaps with >33 % scores ≤1, whereas the other 3 areas were perceived as having been performed well. The difference between items assessed as urgent gaps compared to items with good performance was statistically significant.

Table 4 Teamwork assessment in real critical events (TeamMonitor) [36]
Full size table

Latent safety threats and system changes

A total of 23 different latent safety threats were detected during both the iSTaRT sessions (1.1 per session) and following real events (1.4 per event). Identification of these latent threats resulted in implementation of new guidelines, workshops and equipment changes (selected examples in Table 5). Changes were possible within a short period of time due high motivation of study participants and support from heads of department/senior management team.

Table 5 Latent safety threats. Selected sample of identified latent safety threats and implemented changes
Full size table

Discussion

We designed our in-situ simulated inter-professional team training using a programmatic approach according to the six steps of Kern's curriculum development. This approach helped to overcome important barriers to curriculum design. Our general needs assessment revealed the need for change and played a key role in getting senior management and stakeholder engagement. The specific needs analysis helped to align objectives to the learners needs and may have played a role achieving a good impact. Using in-situ simulation as an educational strategy revealed latent safety threats and helped to change system-based risk factors improving patient safety. The identification and correction of latent threats played a key role in maintaining senior management support for the program. TeamMonitor as a team-based self-assessment tool was successfully applied in real critical events, reduced the gap between training session and reality and identified specific areas to further target training.

A recently published review provides insight regarding how in-situ simulation is currently being used, implemented, and evaluated with the following key findings: i) Formal needs analysis methods are rarely used, ii) in-situ simulation trainers are rarely trained for assessment and feedback, iii) programmes typically address multiple levels of performance, iv) performance measurement practices are rarely formal or rigorous and v) evaluation practices are currently poor [21]. The conclusions are that evidence regarding in-situ simulation efficacy is still emerging and that practical programme planning strategies are needed [21].

We conducted a formal general needs assessment. Our general needs assessment showed that a substantial number of staff members at the Children's Hospital Lucerne felt inadequately trained regarding management of a future critical clinical event and reported to have a high level of anxiety. Lack of adequate training can be responsible for feelings of being unprepared, overwhelmed, and a heightened anxiety about committing errors during the critical clinical event. These findings are consistent with the literature and a strong argument for revision of existing training strategy [6–8]. This information played a key role in getting senior management and stakeholder engagement. Emerging evidence indicates that leadership support at all levels is mandatory for implementation and sustained success of new training methods [38–41]. Engagement of the leadership team also facilitated the purchase of a high-fidelity mannequin to improve context and environment of the training session. A high realism simulated environment may increase functional fidelity creating a powerful learning experience [42]. The specific needs analysis defined the program goals and objectives by allowing the targeted learners to order the set the possible subject matter to include in the curriculum [28]. Participants scoring of perceived benefits of the program may be high because the curriculum was well aligned to learner needs. The results of participants’ self-assessed confidence and self-efficacy is in accordance with other recent publications using in-situ simulated team training [22, 26, 43].

Using in-situ simulation, different levels of learning can be addressed: individual, team-based, unit-based and organizational learning [15, 21]. The possibility to address organizational learning and system-based issues is one of the key benefits of in-situ simulation compared to simulation-center based learning. Observations during and reflected topics by participants and facilitators within the facilitated debriefing after simulated events revealed latent safety threats and had an impact on many system-based factors. This is in line with the literature using in-situ simulation as a tool to reveal latent safety threats or to design new clinical procedures or environments [44–47]. Interestingly, even more latent safety threats (1.4 per event) were detected through facilitated reflection by self assessment after experienced real critical events. The identification and correction of latent threats played a key role in maintaining senior management ongoing support for the program. Recording of latent safety threats and implementation of changes of system-based factors are at the second highest level of measurement effectiveness adapted from Kirkpatrick’s levels of effectiveness [48]. Applying Kern’s six steps approach for curriculum design may have helped to identify important objectives for improvement and facilitated sustained change through multiple impacts at different levels as personal, process and data generation. There are some paediatric studies reporting an impact on effective health care outcomes after simulated team training. Andreatta reported that a simulation-based mock code program significantly improved paediatric cardiopulmonary arrest survival over a period of 3 consecutive years [49]. Theilen reported an improved recognition and management of deteriorating patients coincided with significantly reduced hospitality mortality with implementation of an in-situ simulated team training comparable to our program [50]. Similar, implementation of an interdisciplinary, post-event debriefing program was associated with improved survival with favourable neurologic outcome in a study at the Children’s Hospital of Philadelphia [51].

The evaluation of team performance during real critical events with team-based self-monitoring using TeamMonitor gave additional information regarding behaviours of inter-professional teams in reality [36]. This approach helps to close the gap between training session and reality. Video analysis and observation by third party are alternative approaches with the benefit of more objective data, but resource intensive and difficult ethically [52]. The analysis of our results of TeamMonitor shows that the use of TeamMonitor after real critical events is feasible. Interestingly, despite the small number of scores, there was a significant difference between items of good and low performance. Communication, role allocation and leadership were perceived as on-going training needs. Consistent with a programmatic design, this information further directed our simulated team training iSTaRT curriculum. Importantly, communication and leadership are essential for efficient and successful teamwork [53, 54].

There are several limitations of our study: First, due to the single centre, observational study setting it is not possible to inform regarding cause and effect of our programmatic approach. Second, due to the short study period, the low occurrence of critical events with mortality and the limited introduction of the program to only our highest acuity wards we were not able to assess the impact of the program in regard to patient outcome. Third, we were not able to analyze the direct impact of iSTaRT on the performance in real critical events due to the anonymous team-based assessment.

Conclusions

A programmatic approach of curriculum development may help to improve the implementation, effectiveness and impact of a new educational program. The application of Kern’s framework for curriculum development was feasible, helped achieve stakeholder buy-in and guided the successful implementation of the program to achieve a good impact. TeamMonitor as team-based self-assessment tool was successfully applied in real critical events and identified specific areas to further target training. Detection of latent safety threats during in-situ simulated training sessions and during real critical events helped to change system-based risk factors improving patient safety.