Abstract
Checklists can improve adherence to standardized procedures and minimize human error. We aimed to test if implementation of a checklist was feasible and effective in enhancing patient care in an emergency department handling internal medicine cases. We developed four critical event checklists and confronted volunteer teams with a series of four simulated emergency scenarios. In two scenarios, the teams were provided access to the crisis checklists in a randomized cross-over design. Simulated patient outcome plus statement of the underlying diagnosis defined the primary endpoint and adherence to key processes such as time to commence CPR represented the secondary endpoints. A questionnaire was used to capture participants’ perception of clinical relevance and manageability of the checklists. Six teams of four volunteers completed a total of 24 crisis sequences. The primary endpoint was reached in 8 out of 12 sequences with and in 2 out of 12 sequences without a checklist (Odds ratio, 10; CI 1.11, 123.43; p = 0.03607, Fisher’s exact test). Adherence to critical steps was significantly higher in all scenarios for which a checklist was available (performance score of 56.3% without checklist, 81.9% with checklist, p = 0.00284, linear regression model). All participants rated the checklist as useful and 22 of 24 participants would use the checklist in real life. Checklist use had no influence on CPR quality. The use of context-specific checklists showed a statistically significant influence on team performance and simulated patient outcome and contributed to adherence to standard clinical practices in emergency situations.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Checklists were first introduced in aviation decades ago as a cognitive aid to minimize human error during standardized procedures and since then, ever more industries, especially high reliability organizations, have benefited from the experience of checklist research [1, 2]. Recently checklists have also become the focus of medical studies [3, 4]. Placed in the right context, checklists have a huge potential for increasing patient security by reducing exposure to adverse events during hospitalization [5,6,7,8].
The most prominent example for checklist implementation in clinical use is the WHO Surgical Safety Checklist, which has shown to reduce postoperative complications and mortality significantly [9]. Since this publication in 2009, the number of studies describing checklist development in various medical fields has steadily increased [10,11,12,13,14,15,16].
Besides checklists for daily routines, there is strong interest in developing emergency checklists for critical and high hazard situations given the prevalent exposure to human error [17,18,19,20]. Even trained personnel are susceptible to cognitive impairment caused by stress during medical emergency situations resulting in higher rates of patient exposure to adverse events [21,22,23]. When set in the right context, professionally developed, and carefully implemented, checklists can be an effective method to countering these effects [3, 4, 24]. Hence, in the past decade, there have been several publications of pilot trialling checklists for different kinds of medical crisis situations [24,25,26,27,28,29,30].
Yet, emergency checklists for internal medicine are scarce, possibly because emergencies occur in diverse kinds of settings in comparison to, e.g., an operating theater with fixed procedures [12, 14] or the typical sequence of endotracheal intubation [25].
Therefore, we aimed to develop four crisis checklists for use in an emergency department focused on handling internal medicine cases and to evaluate the checklists’ effect on crisis intervention during four simulation scenarios with structured observation.
Our hypothesis is that our crisis checklists improve adherence to best practices and evidence-based medicine in high hazardous emergency situations, hence reduce adverse events and, in consequence, lead to an improved patient outcome in acute care medicine.
Materials and methods
Study design
We defined the following topics for our checklists: Cardiac arrest with cardiopulmonary resuscitation and approach to hemodynamic unstable cardiac arrythmias. Content was determined by extensive literature research to display up-to-date and evidence-based methods and procedures supported by careful review of our medical experts.
An open-source template by the American think tank AriadneLabs (Boston, Massachusetts, USA) was used for fundamental design [31] and was edited using the graphics program Adobe Illustrator CS5® (Adobe Systems Incorporated, San José, California, USA). Subsequently, we designed four checklists in German with adaption to local conditions following expert methodology [1, 32, 33]. Two checklists comprised the topic Cardiac arrest displaying the approach to shockable and non-shockable rhythms, respectively. A third checklist covered the topic Haemodynamic unstable Tachyarrhythmia, a fourth the topic Hemodynamic unstable Bradyarrhythmia. The checklists as used in the trial are provided in the Supplementary Material with an English translation.
Corresponding to the checklists, we developed four crisis scenarios and protocols which are also available in the Supplementary Material.
Teams of four participants spent a session of three hours in a simulated emergency room with a simulator manikin (ResuscieAnne® QCPR, Laerdal Medical GmbH Deutschland, Puchheim, Germany) being confronted with these crisis scenarios. In half the scenarios, the teams had access to a binder format booklet with the four checklists. All teams were confronted with the same scenarios and had access to the same checklists. The order of the scenarios was arranged randomly and in which scenario a checklist could be used was determined by the study committee beforehand. No indication as to which of the 4 available checklists was relevant to the current scenario was given.
Participants entered as single individuals and were allocated to their teams by fixed dates determined by the study committee. Characteristics did not account for team allocation. Each team appointed a leader and a checklist reader prior to starting the session. The team leader stayed in his or her role throughout the session; whereas, the checklist reader became a normal participant in the scenarios without a checklist.
Every scenario was preceded by a patient history. After this, teams were informed if a checklist was available in this scenario. They had 8 min per scenario to reach the primary endpoint. Access to diagnostics and the patient history was granted and vital parameters were constantly displayed on a patient monitor.
A questionnaire with 43 topics presented in a 4-point Likert scale was handed to each participant after each scenario to collect their experience regarding the checklists and the scenarios. Each scenario was video-recorded for later review (Camcorder HDR-CX240E, Sony Corporation, Tokyo, Japan).
The study was carried out over a period of 24 months.
Study participants
All study participants were medical students at a German university and had to have a minimum medical education of 2 years which includes training in advanced life support courses. Field training in emergency medicine was not a necessary requirement. Participation was voluntary, took place outside of the regular medical curriculum and was not compensated. Each study participant attended only one study session and gave prior written informed consent including consent to publish. Participants had no prior access to or knowledge of the checklists, the content of the study, the crisis scenarios, or the scenario order. An example checklist was presented in an introductory presentation to familiarize the participants with the general structure of the checklists.
Data acquisition
Data were collected and analyzed using Microsoft Excel® (Microsoft Corporation, Redmond, Washington, USA). “Primary endpoint” was defined as the combination of establishing hemodynamic stability and declaration of the underlying diagnosis by one of the team members. The scenario was ended as soon as the primary endpoint was reached or after 8 min by us.
We determined secondary endpoints for each scenario which represent key process steps that significantly influence patient outcome according to current literature. 108 key processes were defined by the study committee, their merit displayed by a point system in which every key process holds a point value corresponding to its individual significance, resulting in a total score of 704 points to be acquired by each team in one session (a list of all key processes per scenario, their assigned point values and corresponding literature are provided in the Supplementary Material). Points were assigned during review of video recordings.
Statistical analysis
For the primary endpoint, the odds ratio was calculated based on contingency tables (Supplementary Material) and Fisher’s exact test. To account for the fact that the observations are not independent since each team passed every scenario, we performed additional sensitivity analysis (Supplementary Material), i.e., a model fitted by generalized estimation equations (gee) with a logistic model to determine the effect of the checklist on the primary outcome. This approach is suitable to account for dependencies between observations by weighting these equations with a Covariance Matrix. This model is used for sensitivity reasons due to the small sample size. The secondary endpoint was tested by a multivariable linear regression, which was adjusted by Scenarios. This analysis was justified by additional sensitivity analysis: accounting for interactive effects between the scenario and the checklist and a random-effect model to quantify a possible effect of the team itself on the outcome (Supplementary Appendix). The odds ratio for the primary endpoint and regression parameters for the secondary endpoint were calculated each with a 95% confidence interval. Analyses and figures were generated using R (Version 3.6.1 (2019-07-05)).
Results
Participants’ characteristics
Six teams with 24 participants (four participants per team) completed 24 crisis sequences, 12 sequences with and 12 sequences without a checklist. Participants’ characteristics are displayed in Table 1. All participants were medical students in their 3rd year or above and presented a wide range of medical experience levels and knowledge regarding medical crisis situations. 42% of the participants had worked as paramedics for a year or more prior to or while studying; whereas, 46% had never been confronted with a real-life CPR situation before.
Primary and secondary endpoints
The combined primary endpoint of this study was reached in 8 of the 12 sequences with a checklist and in 2 of the non-checklist sequences resulting in an odds ratio of 10 (95% CI 1.11, 123.43; p = 0.03607, Fisher’s exact test) for reaching the primary endpoint when using the checklist in our crisis scenarios.
Adding together the point values of all key processes, a total of 704 points could be reached by each team throughout the session. In the 12 sequences with a checklist, participants reached an average of 573 (± 42) points or a performance score of 81.9% vs. an average of 376 (± 40) points or a performance score of 56.3% in the 12 sequences without a checklist (Fig. 1). Linear regression analysis identified a highly significant increase of 26% in the average performance score that could be attributed to use of the checklist (p = 0.00284).
In all four scenarios, more critical steps were taken, and higher point values reached using a checklist. However, the limited number of simulations tested does not allow for individual statistical analysis (Table 2). We note, that the most prominent effect was observed in the Haemodynamic unstable Tachyarrhythmia scenario. With the checklist, 22 of 27 (± 1) critical steps were taken and 162 of 186 point values reached, while only 8.3 (± 1.5) critical steps taken and 50 of 186 point values reached without a checklist present (performance score of 87.1% with checklist, vs. 26.9% without checklist) (Table 2).
To differentiate the effect of our checklist intervention from study participant’ pre-existing experience in handling medical crises, we integrated the average experience level of each team into the linear regression analysis model (see Supplementary Material). This did not change the outcome of the analysis, suggesting, that team experience was not a cofounding factor in the observed differences.
Simulator data
The manikin used in this study is designed for CPR training. Its internal computer records CPR quality factors such as CPR depths and rate, thorax recoil and hand position during CPR, no-flow-time, ventilation volume and rate. Each of these CPR quality factors was displayed on the Cardiac arrest checklists. Despite their distinct display with colorful emphasis, analysis did not show any difference in these factors for teams with vs. teams without a checklist (Table 3). Consequently, we investigated determinants of CPR quality beyond the checklist. Unsurprisingly, participants’ experience in managing crises turned out to be the most dominant factor in influencing CPR quality (see Supplementary Material).
Questionnaire analysis
The response rate was 100%. An excerpt of the questionnaire analysis is presented in Table 4 (for full analysis, see Supplementary Material). Data show strong participant approval for the checklists and the scenarios, e.g., all participants rated the checklists as useful (56.25% agree strongly, 43.75% agree) and 96% disagreed with the statement that the checklist impeded the team in solving the scenario (62.5% disagree strongly, 33.5% disagree). All but one participant stated they would prefer the crisis management team to use a checklist if they themselves were a patient in a critical situation (62.5% agree strongly, 35.4% agree).
Other results from video analysis
In five of six teams, the most experienced team member was elected team leader; whilst in one team, the most experience member was elected checklist reader.
In eleven of the twelve sequences, the checklist reader chose the checklist intended for the scenario. In one sequence, the “wrong” checklist was initially selected and the checklist reader changed to the correct checklist after 161 s. The checklist booklet was never given to another team member. In one sequence, the booklet was laid aside for 73 s.
Discussion
In this study, the use of a crisis checklist was strongly associated with an increased adherence to standard procedures and successful completion of scenarios simulating medical emergencies. The study aimed to establish and test the effect of using checklists in a site specific context, for which the content and layout of the checklists were customized. A randomized cross-over design was chosen to minimize learning effects of study participants.
Crisis checklists have been successfully developed and employed in many medical settings, including intraoperative pediatric emergencies or evacuations from the operating room [34, 35]. While most crisis checklists are tailored towards perioperative situations or anesthesiology procedures, this study aimed to evaluate their use specifically for complex situations in emergency medicine that frequently occur in internal medicine, such as arrhythmias or cardiac arrest. Besides the complexity, such situations are characterized by a competent immediate need of action, but minimal time to evaluate the situation.
For this reason, we designed the lists to be short and easy to read with the opportunity to transfer the complexity into a feasible operating procedure. The increased performance associated with the use of a crisis checklist occurred despite the lack of prior instructions in their use to the participants. This suggests that one advantage of checklists is to serve as a useful ad hoc cognitive aid. In contrast to guideline charts or quick reference instruction cards, the checklists used in our study were specifically customized to the local setting. This has been found to be critical measure for successful implementation of checklists into clinical practice and may have contributed to the strong effect on performance [36]. We observed the highest benefit from checklist use in the hemodynamically unstable tachycardia sequence, an event that requires a complex technical approach. Site and equipment specific information provided by the checklist may facilitate the management of such cases. While our study was tailored to mimic the management of severe cardiac emergencies in an emergency room, it is possible to customize the checklists and procedures towards a different setting, such as inpatient wards. Checklists may have a stronger impact on the performance of less experienced teams. Integrating checklists and simulations may facilitate training of junior team members, and reinforce adherence to best practice care in training sessions [37, 38].
A possible disadvantage of the use of checklists is the potential to divert attention away from patient management in critical care situations. In our study, we did not find that the checklist had a distractive effect, neither in the self-assessment of participants nor in objective parameters, such as CPR quality. On the other hand, we also did not find CPR quality to benefit from checklist usage, consistent with the finding that intensive training is the most beneficial factor in this regard [39,40,41]. Following a checklist, however, does require additional resources. In our simulations, one team member communicated the checklist items to the team. Thus, checklists will be most efficiently implemented when enough staff is available that can devote sufficient attention to a checklist.
It has been pointed out that the most crucial aspect of implementing checklists into clinical routines is to adapt them to the local requirements of multidisciplinary teams [32, 42]. Improving the design and layout of the checklists may increase usability and, thus, lower the implementation burden, but iterative testing and constant revision are essential, all of which are resource intensive and require a concerted team effort [43].
One strong limitation of this study is that participants were medical students, and it is unclear how well our findings will translate to well trained and experienced medical professionals. However, 42% of participants were trained and worked as paramedics (a separate educational track to medical school in Germany) and, thus, had above-average experience in handling critical care situations. The statistical analysis did not find an association of experience level with checklist-dependent performance enhancement. Still, checklists cannot compensate knowledge or experience and, thus, cannot replace clinical decision making. Further studies, preferably with more teams, additional scenarios, and more experienced health care provider are needed to evaluate the utility of crisis checklists in clinical settings.
Availability of data and material
Parts of the data underlying this study are available in aggregated and anonymized form in the supplementary material. Anonymized raw data can be made available from the authors upon request.
Code availability
Not applicable.
References
Degani A, Wiener E (1993) Cockpit checklists: concepts, design and use. Hum Factors 35:345–359
Gawande A (2009) The Checklist Manifesto: How to get things right. Metropolitan Books, New York
Hales B, Pronovost P (2006) The checklist: a tool for error management and performance improvement. J Crit Care 21:231–235
Just K, Hubrich S, Schmidtke D et al (2015) The effectiveness of an intensive care quick reference checklist manual - a randomized simulation-based trial. J Crit Care 30:255–260
Bradley E, Curry L, Webster T et al (2006) Achieving rapid door-to-balloon times: how top hospitals improve complex clinical systems. Circulation 113:1079–1085
Pronovost P, Needham D, Berenholtz S et al (2006) An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med 355:2725–2732
Byrnes M, Schuerer D, Schallom M et al (2009) Implementation of a mandatory checklist of protocols and objectives improves compliance with a wide range of evidence-based intensive care unit practices. Crit Care Med 37:2775–2781
Dhillon P, Murphy R, Ali H et al (2011) Development of an adhesive surgical ward round checklist: a technique to improve patient safety. Ir Med J 104:303–305
Haynes A, Weiser T, Berry W et al (2009) A surgical safety checklist to reduce morbidity and mortality in a global population, for the safe surgery saves lives study group. N Engl J Med 360:491–499
Resar R, Pronovost P, Haraden C et al (2005) Using a bundle approach to improve ventilator care processes and reduce ventilator-associated pneumonia. Jt Comm J Qual Patient Saf 31:243–248
Young G, Frewen T, Barr H et al (1991) Checklist for diagnosis of brain death. Can J Neurol Sci 18:104
El Boghdady M, Tang B, Taiti I, Alijani A (2017) The effect of a simple intraprocedural checklist on the task performance of laparoscopic novices. Am J Surg 214:373–377
Kok E, Abed A, Robben S (2017) Does the use of a checklist help medical students in the detection of abnormalities on a chest radiograph. J Digit Imaging 30:726–731
van Boxtel A, van Veghel D, Soliman Hamad M et al (2017) Use of an intraoperative checklist to decrease the incidence of re-exploration for postoperative bleeding after cardiac surgery. Interact Cardiovasc Thorac Surg 25:555–558
Laws E, Wong J, Smith T et al (2016) A checklist for endonasal transsphenoidal anterior skull base surgery. J Neurosurg 124:1634–1639
Hand W, Bridges K, Stiegler M et al (2014) Effect of a sts. Anesthesiology 120:1339–1353
Driskell J, Salas E, Johnston J (1999) Does stress lead to a loss of team perspective? Group Dyn 3:291–302
Tversky A, Kahneman D (1974) Judgement under uncertainty: heuristics and biases. Science 85:1124–1131
Kuhlmann S, Piel M, Wolf O (2005) Impaired memory retrieval after psychosocial stress in healthy young men. J Neurosci 25:2977–2982
Sexton J, Thomas E, Helmreich R (2000) Error, stress, and teamwork in medicine and aviation: cross sectional surveys. Br Med J 320:745–749
Dieckmann P, Reddersen S, Wehner T, Rall M (2006) Prospective memory failures as an unexplored threat to patient safety: results from a pilot study using patient simulators to investigate the missed execution of intentions. Ergonomics 49:526–543
Kurrek M, Devitt J, Cohen M (1998) Cardiac arrest in the OR: how are our ACLS skills? Can J Anaesth 45:130–132
Semeraro F, Signore L, Cerchiari E (2006) Retention of CPR performance in anaesthetists. Resuscitation 68:101–108
Goldhaber-Fiebert S, Howard S (2013) Implementing emergency manuals: can cognitive aids help translate best practices for patient care during acute events? Anesth Analg 117:1149–1161
Mayo P, Hegde A, Eisen L et al (2011) A program to improve the quality of emergency endotracheal intubation. J Intensive Care Med 26:50–56
Buznik S, van Lier L, de Hingh I, Jakimowicz J (2010) Risk-sensitive events during laparoscopic cholecystectomy: the influence of the integrated operating room and a preoperative checklist tool. Surg Endosc 24:1990–1995
Arriaga A, Bader A, Wong J et al (2013) Simulation-based trial of surgical-crisis checklists. N Engl J Med 368:246–253
Harrison T, Manser T, Howard S, Gaba D (2006) Use of cognitive aids in a simulated anesthetic crisis. Anesth Analg 103:551–556
Marshall S, Mehra R (2014) The effects of a displayed cognitive aid on non-technical skills in a simulated ‘can’t intubate, can’t oxygenate’ crisis. Anaesthesia 69:669–677
Parsons S, Carter E, Waterhouse L et al (2014) Improving ATLS performance in simulated pediatric trauma resuscitation using a checklist. Ann Surg 259:807–813
Operating Room Crisis Checklists (2013). https://www.ariadnelabs.org/areas-of-work/surgery-or-crisis-checklists/resources/#Downloads&%20Tools. Accessed 7 Jan 2019
Hales B, Terblanche M, Fowler R, Sibbald W (2008) Development of medical checklists for improved quality of patient care. Int J Qual Health Care 20:22–30
Weiser T, Haynes A, Lashoher A et al (2010) Perspectives in quality: designing the WHO Surgical Safety Checklist. Int J Qual Health Care 22:365–370
Clebone A, Watkins SC, Tung A (2020) The timing of cognitive aid access during simulated pediatric intraoperative critical events. Pediatr Anesth. https://doi.org/10.1111/pan.13868
Acar YA, Mehta N, Rich M-A et al (2019) Using standardized checklists increase the completion rate of critical actions in an evacuation from the operating room: a randomized controlled simulation study. Prehospital Disaster Med 34:393–400. https://doi.org/10.1017/S1049023X19004576
Urbach DR, Govindarajan A, Saskin R et al (2014) Introduction of surgical safety checklists in Ontario, Canada. N Engl J Med 370:1029–1038. https://doi.org/10.1056/NEJMsa1308261
Howard S, Gaba D, Fish K et al (1992) Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviat Space Environ Med 63:763–770
Moorthy K, Vincent C, Darzi A (2005) Simulation based training. Br Med J 330:493–494
Dane F, Russell-Lindgren K, Parish D et al (2000) In-hospital resuscitation: association between ACLS training and survival to discharge. Resuscitation 47:83–87
Moretti M, Cesar L, Nusbacher A et al (2007) Advanced cardiac life support training improves long-term survival from in-hospital cardiac arrest. Resuscitation 72:458–465
Wayne D, Didwania A, Feinglass J et al (2008) Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest 133:56–61
Russ S, Sevdalis N, Moorthy K et al (2015) A qualitative evaluation of the barriers and facilitators toward implementation of the WHO surgical safety checklist across hospitals in England: lessons from the “Surgical Checklist Implementation Project.” Ann Surg 261:81–91
Burian BK, Clebone A, Dismukes K, Ruskin KJ (2018) More than a tick box: medical checklist development, design, and use. Anesth Analg 126:223–232. https://doi.org/10.1213/ANE.0000000000002286
Acknowledgments
Acknowledgments go to Oliver Veits, Timo Danner, Dominik Gottlieb, and Katharina Nocon for their support of our study by providing the material used in the simulations. We thank Paulina Ogar for revising grammar and spelling. S.S.L. was supported by an Emmy-Noether grant from the German Research Foundation (DFG).
Funding
Open Access funding provided by Universität Zürich. This work was supported by a grant by the German Research foundation (DFG) to Soeren Lienkamp.
Author information
Authors and Affiliations
Contributions
We confirm that all authors have made substantial contributions to the conception and design of the study, data acquisition and/or analysis and interpretation of data, have participated in drafting the article, given final approval for this version to be submitted and have agreed to be held accountable for its accuracy and integrity.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
The study was conducted in full adherence to local laws and regulations and submitted to the local institutional review board, which determined it not be subject to authorization.
Human and animal rights
This study was performed in accordance with the institutional ethical standards and with the 1964 Helsinki declaration and its later amendments. This study did not involve animals.
Informed consent
The study was conducted within a voluntary emergency simulation exercise, was not compensated, and participants gave written informed consent. Personal data were anonymized and only accessible to the authors, ensuring the privacy of the study participants.
Consent for publication
Data were collected anonymized. All participants gave consent in these data being published.
Consent to participate
All participants gave written informed consent to participate in this study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Knoche, B.B., Busche, C., Grodd, M. et al. A simulation-based pilot study of crisis checklists in the emergency department. Intern Emerg Med 16, 2269–2276 (2021). https://doi.org/10.1007/s11739-021-02670-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11739-021-02670-7