Abstract
Early recognition and prevention comprise the first ring of the Chain of Survival for in-hospital cardiac arrest (IHCA). We previously developed and internally validated an emergency department (ED) triage tool, Emergency Department In-hospital Cardiac Arrest Score (EDICAS), for predicting ED-based IHCA. We aimed to externally validate this novel tool in another ED population. This retrospective cohort study used electronic clinical warehouse data from a tertiary medical center with approximately 130,000 ED visits per year. We retrieved data from 268,208 ED visits over a 2-year period. We selected one ED visit per person and excluded out-of-hospital cardiac arrest or children. Patient demographics and computerized triage information were retrieved, and the EDICAS was calculated to predict the ED-based IHCA. A total of 145,557 adult ED patients were included. Of them, 240 (0.16%) developed IHCA. The EDICAS showed excellent discrimination with an area under the receiver operating characteristic (AUROC) of 0.88. The AUROC of the EDICAS outperformed those of other early warning scores (0.80 for Modified Early Warning Score [MEWS] and 0.83 for Rapid Emergency Medicine Score [REMS]) in the same ED population. An EDICAS of 6 or above (i.e., high-risk patients) corresponded to a sensitivity of 33%, a specificity of 97%, and a positive likelihood ratio of 12.2. In conclusion, we externally validated a tool for predicting imminent IHCA in the ED and demonstrated its superior performance over other early warning scores. The real-world impact of the EDICAS warning system with appropriate interventions would require a future prospective study.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Introduction
In-hospital cardiac arrest (IHCA) remains a major public health problem worldwide, with an estimated 290,000 IHCA events per year in the United States1,2,3,4. About 10% of IHCAs occur in the emergency department (ED)2, where infrequent physiologic measurements, ED crowding, and unstable patient conditions all contribute to this devastating outcome2,5,6,7.
Early recognition and prevention comprise the first ring of the Chain of Survival for in-hospital cardiac arrest (IHCA)8. The use of early warning scores (EWSs) has the potential to achieve this goal and has been extensively studied in inpatient unitsx9,10,11,12. Few studies have attempted to validate ward-based IHCA prediction tools in the ED with variable performance13,14,15. A previous systemic review indicated that several EWSs had an excellent predictive ability for 2-day in-hospital mortality (with or without resuscitation) after admission from the ED, but none of the EWSs adequately predicted clinical deterioration or resuscitated cardiac arrest in the ED16. As such, we previously developed and internally validated an emergency department (ED) triage tool, Emergency Department In-hospital Cardiac Arrest Score (EDICAS), for predicting ED-based IHCA17,18.
The EDICAS comprises an 8-item composite score with a possible range of 0 to 13, including age, arrival mode, and categorized vital signs with simple cutoff points trained by the ED data. In our previous study, patients with EDICAS of 6 or above may have a higher risk of developing IHCA in the ED, and higher levels of monitoring are suggested if available. In the original study, that EDICAS served as an excellent discriminator of ED-based IHCA, with an area under the receiver operating characteristic (AUROC) of 0.86 in the internal validation set. However, the predictive performance of EDICAS in external validation remains unclear. Therefore, we aimed to externally validate this novel tool in another ED population of a tertiary medical center.
Methods
Study design and setting
We conducted a retrospective cohort study using data from the Medical Database of the Far Eastern Memorial Hospital (FEMH), a tertiary academic medical center with approximately 1,200 beds and 130,000 ED visits per year. This database serves as a central clinical data repository for all electronic medical records of the healthcare system, including inpatient, outpatient, and ED records. The electronic database houses various information of the patient, including demographics, diagnosis, treatment, imaging, laboratory, prescription, nursing, billing, and administrative data. In this investigation, we retrieved data from 268,208 ED visits in the FEMH from January 1 2016 to December 31 2017. For those with multiple visits, we selected the last visit per patient to maximize the statistical power for the cardiac arrest analysis. Cardiac arrest may result in death during an ED visit, and that visit became the last visit for the patient. Besides, patients younger than 18 years of age or those who presented with out-of-hospital cardiac arrest (OHCA) were excluded. The subject selection process is shown in Fig. 1. ED-based IHCA was defined as the cardiac arrest that occurred in the ED. This study is a retrospective study of medical records and all experimental protocols have been approved by Institutional Review Board (IRB) of FEMH (IRB number: 108026-E) and was conducted in compliance with the Declaration of Helsinki in 1964. The informed consent was waived by the IRB of FEMH.
Data collection
The patient’s demographics and time-stamped clinical information at triage were retrieved, including chief complaint on presentation, mode of arrival, transfer status, vital signs (temperature, heart rate, systolic and diastolic blood pressure, respiratory rate, oxygen saturation), and levels of consciousness coded as Glasgow Coma Scale (GCS). The data extractors were hospital information technology engineers who were blinded to the study hypothesis. After the investigators’ meetings, the data underwent electronic cleaning, and invalid data were designated as missing values. Pain scores were evaluated on a numeric rating scale (NRS) of 0 to 10, where 0 indicated no pain and 10 indicated the worst pain imaginable. We further categorized the NRS scores into no (0), mild (1–3), moderate (4–6), and severe (7–10) pain19. We also classified levels of consciousness as severe coma (GCS ≤ 8), moderate coma (9–12), and minor coma to normal status (GCS ≥ 13)20. Emergency department shifts were classified as day (07:00–14:59), evening (15:00–22:59), and night (23:00–06:59).
We retrieved the five-level computerized Taiwan triage and acuity scale (TTAS), which contains information on 179 structured chief complaints. The chief complaints included OHCA, which was used to identify and exclude the patients experiencing OHCA. The TTAS classifies patients based on computerized algorithms: level 1, resuscitation; level 2, emergent; level 3, urgent; level 4, less urgent; and level 5, non-urgent. This acuity scale has been validated against hospitalization and length of ED stay21.
Outcome measure
The primary outcome measure, ED-based IHCA, was identified via a cardiopulmonary resuscitation (CPR) code (i.e., treated cardiac arrest). ED-based IHCA was defined as the cardiac arrest that occurred while patients were still in the ED. Patients with do-not-resuscitate (DNR) status were not included in the treated cardiac arrest. According to consensus guidelines on reporting IHCA22, the incidence of ED-based IHCA was calculated as the number of treated arrests (numerator) divided by the ED study population (denominator).
Statistical analysis
Summary statistics are presented as proportions [with 95% confidence intervals (CIs)], means [with standard deviations (SDs)], or medians [with interquartile ranges (IQRs)]. Bivariate associations were examined using Student’s t-test, the Mann–Whitney test, and chi-squared test, as appropriate. We used complete-case analysis as the vast majority of variables in the analysis had few or no missing values.
The EDICAS, an 8-item composite score with a possible range of 0 to 13, included age, arrival mode, and categorized vital signs with simple cutoff points (supplemental eTable 1). We fit the eight variables comprising the EDICAS in a multivariable logistic regression model to examine the independent association between each variable and the ED-based IHCA. We also calculated each patient’s EDICAS and computed its sensitivity and specificity with varying cutoff points. The discriminatory ability of the EDICAS was evaluated by using the area under the receiver operating curve (AUROC). For comparison purposes, we also computed the AUROCs by using other EWSs, including the Modified Early Warning Score (MEWS)12 and Rapid Emergency Medicine Score (REMS)23. Finally, the test characteristics of the EDICAS were computed using varying cutoff points. We conducted two sensitivity analyses to test the robustness of our findings. First, we excluded patients with DNR status (n = 293) and repeated the AUROC analysis. Second, we tested the EDICAS performance separately in non-traumatic and trauma patients.
All odds ratios (ORs) and beta-coefficients are presented with 95% confidence intervals (CIs). All analyses were performed using Stata 16.0 software (StataCorp, College Station, TX). A 3-D figure for visualizing IHCA data was plotted using Python matplotlib v3.2.1. All P values are two-sided, and those less than 0.05 were considered to be statistically significant.
Results
Of the 268,208 ED visits over the 2-year study period, 184,184 unique patient visits were included in our study. After excluding those aged less than 18 years or with out-of-hospital cardiac arrest, 145,557 patient visits served as the study population, including 240 cases of IHCA and 145,317 cases of non-IHCA. The patient selection process is illustrated in Fig. 1. In our study population, the mean age of these patients was 47 years, and 52% were women. The overall incidence of ED-based IHCA was 0.16% (95% CI 0.14 to 0.19%). As shown in Table 1, patients with IHCA were much older, predominantly male, and were likely to present to the ED in the winter in comparison with non-IHCA patients. In terms of weekend or time of ED presentation, there were no significant differences between these two groups. Compared with non-IHCA patients, IHCA patients were more likely to arrive by ambulance, to be transferred from other facilities, and to present with dyspnea and chest pain. IHCA patients were less likely to present with injuries. IHCA patients were more likely to be categorized to higher triage levels, with altered mental status, but were less likely to express pain of any levels. Regarding vital signs at triage, IHCA patients were more likely to present with lower body temperature, lower heart and higher respiratory rates, and lower systolic blood pressure and oxygen saturation status. The median length of ED stay was about 4 h in the IHCA group, and was about 2 h in the non-IHCA group. Compared with non-IHCA patients, the admission and ED mortality rates were significantly higher among patients with IHCA.
Multivariable analysis showed that all components in the EDICAS remained statistically significant, including older age, arrival by ambulance, low systolic blood pressure (< 90 mmHg), brady- (< 60/min) and tachycardia (> 90/min), low oxygen saturation (< 95%), tachypnea (≥ 22/min), hypothermia (< 36 °C), and altered mental status (GCS < 15) (Table 2). The distribution of the EDICAS was shown in Fig. 2 with the proportion of IHCA in each EDICAS category.
The EDICAS, including age, arrival mode, and categorized vital signs with simple cutoffs, showed excellent discrimination with an AUROC of 0.88 when applied to the ED population (Fig. 3). The AUROC of the EDICAS outperformed those of other EWSs (0.80 for Modified Early Warning Score [MEWS] and 0.83 for Rapid Emergency Medicine Score [REMS]) in the same ED population (p < 0.001 for MEWS; p = 0.002 for REMS vs. EDICAS). A calibration plot (Fig. 4) showed a good agreement between the observed and predicted probability of IHCA within approximately 1% risk. A fair agreement was noted at a higher-risk range (> 1%) of IHCA.
The test characteristics of the EDICAS are shown in Table 3. An EDICAS of 6 or above (i.e., high-risk patients) corresponded to a sensitivity of 33%, a specificity of 97%, a positive likelihood ratio of 12.2, and overall accuracy of 97%.
In the sensitivity analyses, when we excluded the DNR patients, the AUROC did not materially change (0.88). Similarly, when we tested the EDICAS performance in medical and trauma patients, and its AUROC did not materially change (0.87 in medical and 0.88 in traumatic patients).
Discussion
In this ED-based study of 145,557 patients, we externally validated a triage tool, the EDICAS, for predicting imminent IHCA in the ED. Overall, this simple 8-item triage tool outperformed other previously established EWSs and remained to possess excellent discriminatory power in another ED population with a different case-mix. The logical next step would be to embed the EDICAS in the EMR system with appropriate interventions to evaluate its real-world impact in the ED.
Recent epidemiological evidence has suggested that the characteristics of ED-based IHCAs are quite different from those of ward-based IHCA1,5. Previous studies revealed that the median time to cardiac arrest was approximately two days in ward patients2,23. However, patients in the ED could develop IHCAs within hours after ED arrival. Such a short interval from ED arrival to cardiac arrest in these patients was commonly associated with the time-sensitive nature of certain emergent situations, such as acute coronary syndrome, acute respiratory distress, and so on. Indeed, chest pain and dyspnea were the most common presenting complaints in our IHCA population, a finding that is consistent with the original study. Hence, how to promptly recognize and appropriately manage these high-risk patients within hours is an important issue in the hectic ED environment. We externally validated the EDICAS in a different ED population, and its excellent performance was sustained, suggesting its clinical utility as a screening measure at ED triage for imminent IHCAs in the ED11,12,24.
In the current study, our estimate of IHCA incidence was about 1.6 per 1,000 ED visits, which was similar to that in the original study (1.9 per 1,000 visits). In addition to the similar prevalence of IHCA, the test characteristics and the model performance were also remarkably similar in the current study. For example, an EDICAS of 6 or above was used to define the high-risk group in the original study, corresponding to a specificity of 98% and a positive likelihood ratio of 12.7. Using the same cutoff, the high-risk EDICAS corresponded to a specificity of 97% and a positive likelihood ratio of 12.2 in the current study. The AUROC of the EDICAS was 0.86 in the interval validation of the original study, and this figure even raised a bit to 0.88 in the current study. Taken together, these robust results suggested that the EDICAS could be used to raise the possibility of ED-based IHCA and to prompt the second level of physician review. Moreover, the EDICAS outperformed the general EWSs (MEWS and REMS)25,26, supporting its role as an ED-specific predictive tool for ED-based IHCA. Alternatively, the EDICAS can be used particularly in patients already triaged to levels 1 or 2, further raising its positive predictive value of IHCA.
There were several potential limitations in the study. First, as this external validation was done in another tertiary medical center, a prospective multicenter study would be ideal to further validate the EDICAS in different hospital settings. For example, differences in triage methods, ED length of stay, and emergency medical services systems exist between hospitals. Second, the real-world clinical impact was not addressed in the current study. The EDICAS may be integrated with the EMR system as clinical decision support with sending alerts to ED staff. It would be of interest to see whether there would be an increase in lead time to identify and intervene high-risk patients, thereby potentially preventing ED-based IHCA and associated deaths. Finally, as our study was retrospective in nature, the results were susceptible to selection and information bias.
In summary, we externally validated EDICAS, a novel 8-item ED triage tool predicting IHCA with the excellent discriminatory ability and test characteristics. We also demonstrated the superior performance of EDICAS over other pre-existing EWSs. The logical next step would be to embed the EDICAS in the EMR system with appropriate interventions to evaluate its real-world impact in the ED.
References
Andersen, L. W., Holmberg, M. J., Berg, K. M., Donnino, M. W. & Granfeldt, A. In-hospital cardiac arrest: A review. JAMA 321, 1200–1210 (2019).
Kayser, R. G., Ornato, J. P. & Peberdy, M. A. Cardiac arrest in the emergency department: A report from the national registry of cardiopulmonary resuscitation. Resuscitation 78, 151–160 (2008).
Merchant, R. M. et al. Incidence of treated cardiac arrest in hospitalized patients in the United States. Crit. Care Med. 39, 2401–2406 (2011).
Holmberg, M. J. et al. Annual incidence of adult and pediatric in-hospital cardiac arrest in the United States. Circ. Cardiovasc. Qual. Outcomes 12, e005580 (2019).
Mitchell, O. J. L., Edelson, D. P. & Abella, B. S. Predicting cardiac arrest in the emergency department. J. Am. Coll. Emerg. Phys. Open 1, 321–326 (2020).
Camargo, C. A. Jr. et al. Safety climate and medical errors in 62 US emergency departments. Ann. Emerg. Med. 60(555–63), e20 (2012).
Tan, S. C. & Leong, B. S. Cardiac arrests within the emergency department: An Utstein style report, causation and survival factors. Eur. J. Emerg. Med. 25, 12–17 (2018).
Merchant, R. M. et al. Part 1: Executive summary: 2020 American Heart association guidelines for cardiopulmonary resuscitation and emergency cardiovascular care. Circulation 142, S337–S357 (2020).
Churpek, M. M. et al. Predicting cardiac arrest on the wards: A nested case-control study. Chest 141, 1170–1176 (2011).
Churpek, M. M. et al. Multicenter development and validation of a risk stratification tool for ward patients. Am. J. Respir. Crit. Care Med. 190, 649–655 (2014).
Prytherch, D. R., Smith, G. B., Schmidt, P. E. & Featherstone, P. I. ViEWS—Towards a national early warning score for detecting adult inpatient deterioration. Resuscitation 81, 932–937 (2010).
Subbe, C. P., Kruger, M., Rutherford, P. & Gemmel, L. Validation of a modified early warning score in medical admissions. QJM 94, 521–526 (2001).
Alam, N. et al. Exploring the performance of the national early warning score (NEWS) in a European emergency department. Resuscitation 90, 111–115 (2015).
Bilben, B., Grandal, L. & Søvik, S. National Early Warning Score (NEWS) as an emergency department predictor of disease severity and 90-day survival in the acutely dyspneic patient—a prospective observational study. Scand. J. Trauma Resusc. Emerg. Med. 24, 80 (2016).
Keep, J. W. et al. National early warning score at Emergency Department triage may allow earlier identification of patients with severe sepsis and septic shock: A retrospective observational study. Emerg. Med. J. 33, 37–41 (2016).
Spencer, W., Smith, J., Date, P., de Tonnerre, E. & Taylor, D. M. Determination of the best early warning scores to predict clinical outcomes of patients in the emergency department. Emerg. Med. J. 36, 716–721 (2019).
Lu, T. S., Tsai, C. L., Huang, C. H. & Wang, C. H. Development and validation of a triage tool in predicting cardiac arrest in the emergency department. Circulation 136, 19751 (2017).
Tsai, C. L. et al. Development and validation of a novel triage tool for predicting cardiac arrest in the emergency department. West J. Emerg. Med. 2, 2 (2022).
Boonstra, A. M. et al. Cut-off points for mild, moderate, and severe pain on the numeric rating scale for pain in patients with chronic musculoskeletal pain: Variability and influence of sex and catastrophizing. Front. Psychol. 7, 2 (2016).
Kerby, J. D., MacLennan, P. A., Burton, J. N., McGwin, G. Jr. & Rue, L. W. 3rd. Agreement between prehospital and emergency department glasgow coma scores. J. Trauma 63, 1026–1031 (2007).
Ng, C. J. et al. Validation of the Taiwan triage and acuity scale: A new computerised five-level triage system. Emerg. Med. J. 28, 1026–1031 (2010).
Morrison, L. J. et al. Strategies for improving survival after in-hospital cardiac arrest in the United States: 2013 consensus recommendations: A consensus statement from the American Heart Association. Circulation 127, 1538–1563 (2013).
Olsson, T., Terent, A. & Lind, L. Rapid emergency medicine score: A new prognostic tool for in-hospital mortality in nonsurgical emergency department patients. J. Intern. Med. 255, 579–587 (2004).
Lee, S. B. et al. Emergency department triage early warning score (TREWS) predicts in-hospital mortality in the emergency department. Am. J. Emerg. Med. 38, 203–210 (2020).
Yu, Z., Xu, F. & Chen, D. Predictive value of modified early warning score (MEWS) and revised trauma score (RTS) for the short-term prognosis of emergency trauma patients: A retrospective study. BMJ Open 11, e041882 (2021).
Imhoff, B. F. et al. Rapid emergency medicine score (REMS) in the trauma population: A retrospective study. BMJ Open 4, e004738 (2014).
Acknowledgements
This project was supported by grants from the National Taiwan University Hospital and Far Eastern Memorial Hospital (107-FTN18); and the Ministry of Science and Technology (MOST 109-2634-F002-041).
Author information
Authors and Affiliations
Contributions
Study concept and design: JTS, CLT; acquisition of data: JTS, CLT; analysis and interpretation of data: all authors; first drafting of the manuscript: JTS, CCC; critical revision of the manuscript for important intellectual content: all authors; statistical analysis: JCL, CLT; obtained funding: JTS, CCF, CHH, CLT; administrative, technical, and material support: JTS, TCL, CHH, CLT; study supervision: CLT.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Sun, JT., Chang, CC., Lu, TC. et al. External validation of a triage tool for predicting cardiac arrest in the emergency department. Sci Rep 12, 8779 (2022). https://doi.org/10.1038/s41598-022-12781-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-022-12781-6
- Springer Nature Limited
This article is cited by
-
Management and prevention of in-hospital cardiac arrest: present and future
npj Cardiovascular Health (2024)
-
Development of a machine learning-based clinical decision support system to predict clinical deterioration in patients visiting the emergency department
Scientific Reports (2023)