Background

Guidelines suggest that patients with pulmonary embolism (PE) who are identified to have a low-risk of early post-PE all-cause mortality may be candidates for abbreviated hospital admission or outpatient treatment if appropriate follow-up can be arranged [1, 2]. Data from randomized trials and observational studies suggest that early discharge or outpatient treatment of low-risk PE patients is feasible and safe [3, 4].

A prior meta-analysis suggested at least one-third of acute PE patients could be classified as low-risk for early mortality according to clinical prediction rules [5]. This same meta-analysis identified the Pulmonary Embolism Severity Index (PESI), simplified PESI (sPESI) and Hestia clinical prediction rules as having high sensitivities and negative predictive values (NPVs) for identifying low-risk PE patients. In order to use PESI [6], sPESI [7] and Hestia [8] in the risk stratification of a patient with PE, access to vital signs, laboratory values, comorbid conditions and a cognitive evaluation at presentation is necessary. While PESI, sPESI and Hestia can be helpful in clinical practice, the extensive clinical data required to score these rules are not commonly found in claims databases or easily accessible to individual hospitals/health-systems. As a result, the utility of PESI, sPESI or Hestia for retrospective identification of low-risk patients with PE is limited.

While not originally derived to aid in prognostication in a prospective clinical setting, the In-hospital Mortality for Pulmonary embolism using Claims daTa (IMPACT) multivariable prediction rule utilizes data accessible within claims databases and/or claims from individual hospitals to retrospectively risk stratify patients with PE for early mortality [9]. Prior validation studies suggest IMPACT can accurately identify pulmonary embolism (PE) patients at low-risk of early mortality [10, 11]. The ability of IMPACT to correctly identify patients at low- and higher-risk of early mortality has not previously been compared to analogous clinical prediction rules. Therefore, using data from a single center, this study sought to externally validate IMPACT, PESI, sPESI and Hestia for predicting in-hospital and 30 day post-PE mortality.

Methods

Preparation of this study report was in accordance with the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement [12]. For this retrospective cohort study, we identified consecutive patients diagnosed with acute PE between October 21, 2010 and May 12, 2015 using computerized claims records for admissions to Hartford Hospital (Hartford, Connecticut, USA). The hospital’s computerized claims system contains information on source of admission, International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes, admission and discharge dates and discharge status. To be eligible for inclusion into this study, patients ≥18 years of age presenting to our institution had to have a primary diagnosis of PE (ICD-9-CM code = 415.1x). Consistent with prior studies, we excluded patients lacking objective confirmation of acute PE according to clinical guidelines. The following were considered confirmatory studies for the diagnosis of acute PE: high probability perfusion-ventilation lung scan (V/Q scan), computed tomography pulmonary angiography (CTPA) or pulmonary angiography diagnostic for PE, or a non-diagnostic V/Q scan or CTPA in combination with an abnormal compression ultrasonography of the lower extremities. Consistent with many prior studies of PE clinical prediction rules [5], subjects that received thrombolysis and/or pulmonary embolectomy within the 48 h of presentation were excluded as clinical guidelines do not consider such patients low-risk [1, 2]. All patients included in this study were managed according to usual clinical practice for our institution.

Risk stratification of patients with acute PE using IMPACT, PESI, sPESI and Hestia was performed according to published methods (Additional file 1) [6–9]. Patients with an IMPACT predicted mortality risk ≤1.5 % [9], PESI score ≤85 [6] or a sPESI or Hestia scores =0 [7, 8] were classified as low-risk for early mortality. Estimated mortality risk according to the claims-based IMPACT prediction rule [estimated % absolute risk = 1/(1 + exp(-x); where x = −5.833 + (0.026*age) + (0.402*myocardial infarction) + (0.368*chronic lung disease) + (0.464*stroke) + (0.638*prior major bleeding) + (0.298*atrial fibrillation) + (1.061*cognitive impairment) + (0.554*heart failure) + (0.364*renal failure) + (0.484*liver disease) + (0.523*coagulopathy) + (1.068*cancer)] was determined using all available hospital claims data (i.e., ICD-9-CM diagnosis and procedural codes) for each patient’s index PE encounter along with their age at time of presentation. ICD-9-CM coding for relevant IMPACT co-morbidities were performed according to the original IMPACT derivation paper [9]. Data necessary to classify patients as low- or high-risk of early mortality according to the PESI, sPESI and Hestia clinical prediction rules [6–8] were obtained by linking all included patients identified through hospital claims to the hospital’s electronic health record (EHR). We used vital signs (heart rate, blood pressure, respiratory rate, oxygen [O2] saturation, body temperature), laboratory values (serum creatinine, platelet count, total bilirubin) and mental status assessments obtained as close to the time of presentation for the index PE encounter as possible to score each of the clinical prediction rules [6]. For all patients admitted through the emergency department, the first vital sign, laboratory value and/or mental status assessment upon presentation (but within 24 h) was utilized. For patients directly admitted to the hospital we used the first values recorded on the day of hospital admission. Consistent with previous studies of this type, missing vital, laboratory and mental status assessment data were assumed to be normal [6]. For PESI, sPESI and Hestia, the presence of cancer, heart failure, chronic lung disease, severe liver disease (defined as a total bilirubin ≥2.5 mg/dL), heparin-induced thrombocytopenia and recent clinical events (gastrointestinal bleeding within 14 days, stroke with 4 weeks, surgery with 2 weeks) were assessed at time of hospital admission for the index PE encounter. All required data was abstracted from the electronic health record (including vital signs, laboratory values and emergency department, admission and consult notes) by trained study personnel blinded to study outcome.

All-cause in-hospital and 30 day post-PE mortality served as a priori endpoints for this study. In-hospital mortality was determined using the discharge status field for the index admission within the hospital billing system and electronic health record. Thirty-day mortality was based upon searches of the Social Security Death Index (SSDI) [11] performed >6 months after the last day of eligible inclusion in this analysis. Computerized health-system encounter data from our hospital’s billing records were queried for subsequent emergency visit claims, clinic visits and/or hospital readmissions outside of 30 days. These were used as confirmatory markers of vital status at 30 days. Beginning in March 2014, rules regarding access to data within the SSDI changed; restricting the release of three most recent years of data [13, 14]. For this reason, our 30 day mortality endpoint was assessed only in the subset of consecutive patients presenting to the hospital prior to January 31, 2014.

Baseline characteristics are described with means ± SDs for continuous data and counts and proportions for categorical data. Sensitivity, specificity and negative and positive predictive values for predicting early mortality were calculated for IMPACT, PESI, sPESI and Hestia along with 95 % confidence intervals (CIs). C-statistics were computed to evaluate each rule’s overall discriminative power. All database management and statistical analyses were performed using IBM SPSS Statistics version 22.0 (IBM Corp., Armonk, NY, USA). The study was approved by the Hartford Hospital institutional review board.

Results

A total of 861 patients with a primary ICD-9-CM code for acute PE and objective confirmation of diagnosis were identified (Fig. 1). Of these, 54 received thrombolytic therapy/embolectomy within 48 h of presentation, leaving 807 for analysis. Baseline characteristics of the cohort, stratified by low- and high-risk for each rule are shown in Table 1. Two-hundred and thirty-four patients presented with PE after January 2014 and were excluded from the 30 day mortality endpoint analysis, as SSDI data are not available for patients past this time point. The baseline characteristics of the 30 day mortality analysis patient subset were similar to the overall population (Additional file 1: Table S1).

Fig. 1
figure 1

Identified PE patients and distribution of risk classes. IMPACT = hrs = hours; In-hospital Mortality for Pulmonary embolism using Claims daTa; PE = pulmonary embolism; PESI = Pulmonary Embolism Severity Index; SSDI = social security death index; sPESI = simplified Pulmonary Embolism Severity Index

Full size image
Table 1 Characteristics of pulmonary embolism patients
Full size table

The IMPACT, PESI, sPESI and Hestia scores (mean ± SD) for the complete cohort of 807 PE patients were 3.9 ± 4.3, 96.4 ± 33.3, 1.1 ± 0.9 and 1.3 ± 1.1, respectively. While mean age for patients was 64.1 ± 16.57; when dichotomized into risk groups, high-risk patients were considerably older, regardless of prediction rule used. At time of presentation, mean vital sign values were within normal ranges in the overall analysis population; but more than half of the subjects required O2 supplementation to maintain saturations >90 %. The use of thrombolysis and/or embolectomy after 48 h was infrequent, occurring in only 4 (0.5 %) of patients.

The overall incidence of all-cause in-hospital mortality was 2.6 % (21/807). IMPACT, PESI, sPESI and Hestia classified 26 % (Hestia)-38 % (PESI) of the cohort as low-risk for early post-PE mortality. Fatality among low-risk patients was low (0–0.6 %), corresponding to NPVs of 99.4 (PESI)-100 % (sPESI and Hestia) (Table 2). IMPACT’s sensitivity was 95.2 % (95 % CI = 74.1–99.8 %), and the sensitivities of clinical rules ranged from 91 (PESI)-100 % (sPESI and Hestia). Specificities of all rules ranged between 26.8 (Hestia)-39.1 % (PESI) and C-statistics from 0.76 (sPESI)-0.86 (Hestia). Additional file 1: Table S2 describes the characteristics of patients who died in-hospital and had discordant risk categorization across any of the four prediction rules.

Table 2 Prognostic test characteristics for in-hospital mortality
Full size table

Among the subset of 573 patients accessible within the SSDI, 33 (5.8 %) died of any cause within 30 days of presentation for PE. All rules classified 27.9 (Hestia)-38.0 % (PESI) of PE patients as low-risk, and fatality occurred in 0.0 % (Hestia)-1.4 % (PESI) of low-risk patients (NPVs = 98.6–100 %) (Table 3). IMPACT’s sensitivity for predicting 30 day mortality was 97.0 % (95%CI = 82.5–99.8 %), while sensitivities for clinical rules ranged from 91 (PESI)-100 % (Hestia). Specificities of rules ranged between 29.6 % (Hestia)-39.8 % (PESI) and their C statistics ranged from 0.73 (sPESI)-0.81 (PESI). Additional file 1: Table S3 describes the characteristics of patients who died within 30 days of presentation and had discordant risk categorization across any of the 4 prediction rules.

Table 3 Prognostic test characteristics for 30 day mortality
Full size table

Discussion

In this analysis, the claims-based IMPACT prediction rule displayed prognostic accuracy similar to that of commonly used clinical risk stratification rules, including PESI and sPESI (which have been prospectively validated for identification of low-risk PE patients) and the Hestia criteria (which was prospectively designed to identify patients whom could be treated as outpatients). The 4 rules evaluated in this study classified between 1/4th and 2/5ths of patients as low-risk. Each exhibited sensitivities >90 %, NPVs >98.6 % and specificities <40 % for predicting in-hospital or 30 day all-cause mortality, and these findings are consistent with prior derivation and validation studies. To our knowledge, this is the first external validation study of Hestia [5, 9–11]. Taken together, our results suggests each of the four rules can identify a cohort of low-risk PE patients whom are unlikely to die within the first 30 days of presentation. However, because a minority of patients with PE (<6 % in our study) die within 30 days, these prognostic rules classify a substantial number of patients who ultimately survive into higher-risk groups (hence their lower specificities). Prognostic tests seldom have both high sensitivity and specificity. Therefore, when using a prognostic test to decide upon implementing a less conservative treatment strategy (e.g., discharging a patient with acute PE directly from the emergency department) higher sensitivity and NPV values are preferable.

The American College of Chest Physicians and the European Society of Cardiology guidelines support early discharge and/or home treatment of PE patients at low-risk for early mortality as long as they have adequate home circumstances [1, 2]. These guidelines suggest that clinicians use validated clinical prediction rules to assist in identification and selection of low-risk patients. Although in this analysis IMPACT displayed a similar ability to identify low-risk PE patients as PESI, sPESI and Hestia; IMPACT was not originally derived or validated to assist in prospective clinical decision-making (and is relatively more computationally complex compared to clinical prediction rules), and it should not be used for individual patient decision-making [9]. However, IMPACT appears valid for retrospective identification of low-risk PE patients and therefore could be used to aid in the performance of real-world outcomes studies and to enable payer/institution benchmarking of rates of low-risk PE patients treated at home or following an abbreviated admission. Using claim s data as described by IMPACT may have advantages over obtaining highly granular clinical information from the EHR (including reduced time and effort requirements).

Our study has limitations that require consideration. First, this validation study was performed retrospectively and therefore may be subject to biases, particularly due to missing data. Nonetheless, our study had similar rates of missing data than reported in prior prospective derivation/external validation papers of clinical prediction rules [5–7]. Second, this was a single-center study limiting its generalizability and sample size. However, baseline characteristics and mortality rates were consistent with national estimates [10, 15] and our sample size (573–807 patients) was large relative to many previously published external validation studies of PE clinical prediction rules [5]. Third, we could not assess 30 day mortality in our entire study cohort due to restrictions on the availability of SSDI data [14]. Despite this, the sample size of patients with objectively confirmed PE available for final analysis was robust. Moreover, we are unaware of any programmatic changes in evaluation and treatment of patients with PE at our institution since February 2014. Consequently, the likelihood of selection bias resulting from the unavoidable exclusion of patients after this date is low. Next, the more subjective nature of certain criteria in Hestia (i.e., medical or social reason for hospital admission and the need for intravenous pain medication for >24 h), make retrospective scoring challenging. This being said, the proportion of patients in our study classified as higher-risk because of these “subjective” criteria was not inconsistent with the Hestia derivation study [8]. Lastly, the 48 h cut-off used to exclude patients undergoing thrombolysis and/or embolectomy is somewhat arbitrary. We excluded patients receiving thrombolysis and/or embolectomy in less than 48 h because such patients likely had hemodynamic instability at presentation and would not be considered low-risk per guidelines [1, 2]. Of note, numerous studies evaluating the prognostic accuracy of clinical prediction rules have similarly excluded patients undergoing early thrombolysis and/or embolectomy [5]. However, when these procedures are performed later in a hospital stay (day 3 onwards), they are more likely an indicator of a therapeutic failure resulting in a poor clinical course (i.e., respiratory failure or cardiac arrest). In addition, the need for and timing of thrombolysis and/or embolectomy can easily be detected in a claims database and a clinical setting, allowing it to be implemented in our evaluation of IMPACT and clinical prediction rules. The 48 h cut-off was chosen a priori to approximate the likely timing used for assessing the similar Hestia criterion (i.e., the Hestia study required discharge within 24 h of PE diagnosis, likely resulting in the assessment of the 11 Hestia criteria within 48 h of initial PE presentation) [8].

Conclusion

IMPACT identified low-risk PE patients with similar accuracy as PESI, sPESI and Hestia. While not designed for prospective clinical decision-making, IMPACT appears useful for identification of low-risk PE patient in retrospective claims-based studies.

Ethics approval and consent to participate

This study was approved by the Hartford Hospital institutional review board with a waiver of informed consent.

Consent for publication

Consent for publication: not applicable.