Background

Site-of-care decision according to disease severity is one of the most critical steps in the management of pneumonia [1]. Several established prediction models have been developed to identify patients with community-acquired pneumonia (CAP) who are at low risk for death and can be treated in an outpatient setting. The representative models include the Pneumonia Severity Index (PSI) and the CURB-65 (confusion, blood urea nitrogen > 7 mmol/L [20 mg/dL], respiratory rate ≥ 30/min, low blood pressure [diastolic blood pressure ≤ 60 mm Hg or systolic blood pressure < 90 mm Hg], and age ≥ 65 years) [2, 3].

Identification of patients at high risk for severe adverse events (SAEs) at the time of pneumonia diagnosis is also crucial. These risk assessments aid physicians in determining patients who require higher-intensity treatment [4]. Some of the prediction models proposed for the identification of these patients include the 2007 Infectious Diseases Society of America (IDSA)/American Thoracic Society (ATS) criteria [1], the España SCAP rule [5], and the SMART-COP [6]. However, approaches to identify and assess patients who might need more intense treatment compared to those at low risk for SAEs remain a debatable issue [7,8,9]. Furthermore, although the management of patients with CAP and those with healthcare-associated pneumonia (HCAP) have been considered as the same entity in the 2019 ATS/IDSA CAP guidelines [4, 10], evidence is limited regarding prediction models that can be applicable to all patients with CAP including those with HCAP to identify those who require higher-intensity treatment at the time of pneumonia diagnosis.

Therefore, we aimed to develop and validate a prediction tool for the identification of all patients with CAP who are at high risk for SAEs using two different cohort datasets that were prospectively collected.

Methods

Study design and setting

This study was performed using datasets of two different prospective observational studies [11, 12]. A prediction model was developed using a larger dataset from an observational study (derivation cohort), which was performed at ten medical institutions in Japan (one 1000-bed university hospital and nine major community hospitals, each equipped with more than 500 beds) between March and December 2010 [11, 13, 14]. The prediction model was validated using a dataset from another observational study (validation cohort), which was performed at four medical institutions between April 2013 and March 2014. These four institutions participated in the first study for the derivation cohort as well [12]. There were no overlapping cases between the derivation and validation cohorts.

The study protocol adhered to the Declaration of Helsinki and the Japanese Ethics Guidelines for Epidemiological Studies. The study was approved by the ethics committee of Nagoya University and respective institutional review boards of the participating institutions. Informed consent of the participants was waived, but the opt-out method was adopted according to the ethics guidelines.

Patients

Details on pneumonia definitions and categories, inclusion and exclusion criteria, definitions of variables, procedures, and data collection have been described elsewhere [11, 12] and can be found in the Additional file 1. Briefly, patients aged 20 years or older who developed pneumonia outside of a hospital and needed inpatient treatment were enrolled. Definitions of pneumonia categories were in accordance with international guidelines [1, 15]. The study included adult patients with CAP including those with HCAP. Outpatients and patients with hospital-acquired and ventilator-associated pneumonia were excluded.

Endpoints

The primary study endpoint was SAEs, which were composite and included all-cause death and requirement of mechanical ventilation (MV) or vasopressor support (VS) within 30 days after pneumonia diagnosis. MV included invasive and noninvasive MV. Secondary endpoints were individual components of the primary endpoint and intensive care unit (ICU) admission within 30 days after the pneumonia diagnosis.

Model development

Patient data without missing values were used for model development (complete-case analysis). A total of 33 candidate variables potentially associated with SAEs, except type of pneumonia, were identified from the literature review and are shown in Additional file 2 and Table 1 [1,2,3, 5, 6, 16,17,18]. Categorization of continuous variables and cutoff values were determined with reference to previous studies [1,2,3, 5, 6, 16,17,18], clinical significance, and data distribution of each variable. First, univariate logistic regression analysis was performed to assess the effects of candidate variables on SAEs. Next, multivariate logistic regression analysis was performed using all 33 candidate variables. A forward stepwise selection procedure was used with the inclusion of variables with a p value of less than 0.1. β-coefficients, odds ratios (ORs), and the corresponding 95% confidence intervals (CIs) were calculated. A simple prediction scoring system was developed based on the multivariate logistic regression model by rounding β-coefficients. The frequencies of SAEs were calculated by points. Receiver operating characteristic (ROC) curves of the multivariate logistic regression model and the simple prediction scoring system were assessed. The point with higher sensitivity was preferred when determining the threshold of the simple prediction scoring system. We also considered that the possible acceptable specificity was at least 60% (between 60 and 70%) according to previous studies on the España SCAP rule and SMART-COP [4,5,6]. In addition, the ROC curve and calculated Youden’s index were utilized to determine the threshold [19].

Table 1 Patient characteristics and clinical outcomes
Full size table

Model performance and validation

Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), area under the ROC curve (AUROC), and corresponding 95% CIs were calculated to assess model performance. Positive likelihood ratio (PLR) and negative likelihood ratio (NLR) were also calculated. These values were compared among prediction models using the validation cohort. Validation analyses were performed in patients without missing values.

Severity assessment tools for comparisons

The PSI [2], CURB-65 [3], 2007 IDSA/ATS criteria [1], simplified 2007 IDSA/ATS minor criteria [20], SMART-COP [6], and España SCAP rule [5] were assessed for comparisons. For each prediction method, the diagnostic accuracy was assessed using the following thresholds reported in the original articles on these tools: PSI, ≥ class IV; CURB-65, ≥ 3 points; España SCAP rule, ≥ 10 points; SMART-COP, ≥ 3 points; 2007 IDSA/ATS criteria (minor criteria), ≥ 3 factors [21]; and simplified 2007 IDSA/ATS minor criteria, ≥ 3 factors [20]. During calculations for the 2007 IDSA/ATS criteria, simplified 2007 IDSA/ATS minor criteria, SMART-COP, and España SCAP rule, multilobar radiological infiltration was substituted by bilateral or chest radiographical infiltration extending to more than 2/3 of the unilateral lung.

Subgroup analyses

Subgroup analyses were performed in patients without potential treatment restrictions, those with CAP (excluding HCAP) or HCAP, those without invasive MV/VS at admission, and those without immunosuppression. Patients without potential treatment restriction were defined as ambulatory patients under 80 years. Those with potential treatment restrictions were defined as those aged 80 years or older or any patient with nonambulatory status [22]. Invasive MV/VS at admission corresponded to the major criteria of the 2007 IDSA/ATS criteria [1]. Immunosuppression included any immunosuppressive diseases, such as congenital or acquired immunodeficiency, hematologic diseases, asplenia and neutropenia (< 1000 cells/μL), treatment with immunosuppressive agents, chemotherapy within the previous 30 days, and use of corticosteroids in daily doses of at least 10 mg/day prednisone equivalent for more than 2 weeks [11].

Statistical analysis

Categorical data were summarized as frequencies presented as percentages. All tests were two-tailed. Statistical data were analyzed using SPSS Statistics (version 25; IBM, Armonk, NY, USA) unless otherwise indicated. Prism (version 7.04; GraphPad, San Diego, CA, USA) was used to calculate 95% CIs for sensitivity, specificity, PPV, and NPV. PLR and NLR were calculated using EZR (version 1.54; Saitama Medical Center, Jichi Medical University, Saitama, Japan).

Results

Characteristics of the cohorts

A total of 1413 patients were eligible in the derivation cohort, and 721 patients were included in the validation cohort. After excluding patients with missing data, the derivation and validation cohorts comprised 1334 and 643 patients, respectively (Fig. 1). The baseline patient characteristics were closely comparable between the derivation and validation cohorts (Table 1). The percentages of patients aged 80 years or older were identical in both the derivation and validation cohorts (42%). The percentages of patients with hyperpotassemia (K ≥ 5.0 mEq/L) and hypercapnia (PaCO2 > 45 mmHg) were higher in the derivation cohort than in the validation one. The incidence proportions of SAEs in the derivation and validation cohorts were 20.8% and 18.7%, respectively (Table 1).

Fig. 1
figure 1

Patient flowchart in the derivation and validation cohorts. Model development and validation were performed in patients without missing values (complete-case analysis)

Full size image

Prediction model development for SAEs in the derivation cohort

The results of the univariate analysis between 33 candidate variables and SAEs are shown in Additional file 2. As a result of the multivariate logistic regression analysis with forward stepwise selection, the prediction model for SAEs comprised 13 significant variables (Table 2). By rounding the β-coefficients of these variables, a simple scoring system was developed, named ALL-COP SCORE rule (Table 2, Fig. 2), which included the following variables: serum albumin level (< 2 g/dL, 2 points; 2–3 g/dL, 1 point); leukopenia (white blood cell count; < 4000 cells/μL, 3 points); chronic lung disease (1 point); confusion (2 points); PaO2/FIO2 ratio (< 200 mmHg, 3 points; 200–300 mmHg, 1 point); potassium level (≥ 5.0 mEq/L, 2 points); arterial pH (< 7.35, 2 points); systolic blood pressure (< 90 mmHg, 2 points); PaCO2 (> 45 mmHg, 2 points); HCO3− (< 20 mmol/L, 1 point); respiratory rate (≥ 30 breaths/min, 1 point); pleural effusion (1 point); and extent of total chest radiographical infiltration of unilateral lung (> 2/3, 2 points; 1/2–2/3, 1 point). The ROC curve was similar between the ALL-COP SCORE rule and the original logistic regression model, and the AUROC was the same between the two models (0.84 [95% CI, 0.81–0.86] and 0.84 [95% CI, 0.81–0.87] for the ALL-COP SCORE rule and the original logistic regression model, respectively) (Additional file 3A). The sensitivity, specificity, and Youden’s index (sensitivity plus specificity minus one) for each point of the ALL-COP SCORE rule in the derivation cohort are shown in Additional file 3B. Since an ALL-COP SCORE threshold of ≥ 4 points had a specificity of ≥ 60% (Fig. 3, Additional file 3B) and as higher sensitivity was preferable to determine inpatients who should receive higher-intensity treatment, as described in the Methods, we determined ≥ 4 points as the ALL-COP SCORE threshold for identifying patients with CAP at high risk for SAEs at the time of pneumonia diagnosis.

Table 2 Multivariate analysis for severe adverse events in the derivation cohort
Full size table
Fig. 2
figure 2

ALL-COP SCORE rule. The ALL-COP SCORE, a scoring system that was developed based on a multivariate logistic regression model, comprises 13 variables. Each variable is assigned between 1 and 3 points by rounding β-coefficients (Table 2). The point range of the ALL-COP SCORE is between 0 (min) and 24 (max)

Full size image
Fig. 3
figure 3

A Prevalence of patients with severe adverse events according to ALL-COP SCORE points in the validation cohort. B Sensitivity and specificity of the ALL-COP SCORE rule in the validation cohort. An ALL-COP SCORE ≥ 4 points has a sensitivity of 92% and a specificity of 60%

Full size image

Prediction model performance for SAEs in the validation cohort

Figure 3A shows the probabilities of SAEs within 30 days after pneumonia diagnosis according to the ALL-COP SCORE rule in the validation cohort. Patients with 4–5, 6–7, and ≥ 8 points had 17%, 35%, and 52% increase in the probability of SAEs, respectively, whereas the probability of SAEs was 3% in patients with ≤ 3 points. An ALL-COP SCORE threshold of ≥ 4 points had a sensitivity of 91.7% and a specificity of 60.0%, with a PPV of 34.5%, a NPV of 96.9%, a PLR of 2.3, and a NLR of 0.1. The ALL-COP SCORE rule exhibited a higher AUROC (0.85) compared with the other predictive models (Fig. 4, Table 3). The sensitivity of the ALL-COP SCORE rule was higher than those of the España SCAP rule, SMART-COP, 2007 IDSA/ATS minor criteria, and simplified 2007 IDSA/ATS minor criteria. Comparison of the sensitivities of the tested predictive models at the same level of specificity (0.60, the specificity of ≥ 4 points by the ALL-COP SCORE rule) and the same comparisons conducted after changing the thresholds so that the specificities were close to between 60 and 70% among the tested models revealed that the sensitivity of the ALL-COP SCORE rule was the highest among the evaluated predictive models (Fig. 4, Table 3).

Fig. 4
figure 4

Comparison of the receiver operating characteristic curves of the prediction rules for severe adverse events in the validation cohort. Definition of abbreviations: SMART-COP, systolic blood pressure, multilobar chest X-ray involvement, albumin, respiratory rate, tachycardia, confusion, oxygenation, and arterial pH; IDSA/ATS, Infectious Disease Society of America/American Thoracic Society; SCAP, severe community-acquired pneumonia; PSI, Pneumonia Severity Index; CURB-65, confusion, urea level, respiratory rate, blood pressure, and age ≥ 65 years

Full size image
Table 3 Comparison of the prediction rules for adverse outcomes in the validation cohort
Full size table

Subgroup analyses

Table 4 shows the results of the predictive performance of the ALL-COP SCORE rule in patient subgroups. The predictivity of the ALL-COP SCORE rule was more accurate when the rule was applied to patients without potential treatment restrictions. Specifically, compared to all patients in the validation cohort, the sensitivity was the same (91.2%), whereas the specificity increased by 10% (70.5%) (Table 3) in the subgroup of patients without potential treatment restrictions, which included 280 (81.9%) patients with CAP excluding HCAP and 62 (18.1%) patients with HCAP. The assessment of patients without invasive MV/VS at admission and those without immunosuppression revealed that the predictive performance of the ALL-COP SCORE rule was almost identical between each subgroup and the entire validation cohort.

Table 4 Performance of the ALL-COP SCORE for predicting severe adverse events in validation cohort subgroups
Full size table

The assessment for the performance of the ALL-COP SCORE rule in secondary endpoints in the validation cohort revealed that an ALL-COP SCORE threshold of ≥ 4 points had a sensitivity of 90.0% and a specificity of 54.6% for 30-day mortality, a sensitivity of 93.2% and a specificity of 56.1% for the requirement of MV/VS, and a sensitivity of 90.9% and a specificity of 55.1% for ICU admission (Additional file 4).

Discussion

In the present study, we developed and validated the ALL-COP SCORE prediction rule for its utility in all patients with CAP including those with HCAP. The ALL-COP SCORE rule exhibited a high AUROC (0.85), and an ALL-COP SCORE threshold of ≥ 4 points showed a high sensitivity (92%) with 60% specificity for predicting SAEs. These results indicated that the proposed ALL-COP SCORE prediction rule may be useful for identifying patients at high risk for SAEs at the time of CAP diagnosis.

The optimal definition of severe CAP, which remains a topic of debate [5,6,7,8,9, 23], should be used as a guide during decision-making on identifying patients who require a higher level of inpatient treatment intensity. The 30-day mortality was adopted as a primary outcome measure to identify patients with mild CAP who could be treated as outpatients, as described in the original studies on the PSI and CURB-65 [1,2,3, 24, 25]. Conversely, several previous studies adopted ICU admission as the primary outcome measure to identify patients who should receive higher-intensity treatment for severe pneumonia [26,27,28,29]. However, some concerns remain that using ICU admission as an outcome measure might prevent the generalizability of findings due to regional differences in the ICU admission criteria depending on medical resources [5, 6, 9, 21]. Thus, España and colleagues adopted a composite endpoint including MV, septic shock, and death to develop the España SCAP rule [5], whereas Charles and colleagues employed intensive respiratory or VS as a primary outcome measure to develop the SMART-COP [6]. Regarding this outcome measure, some researchers adovocate death should not be combined with MV or VS. However, Torsten and colleagues showed that, even patients residing in nursing homes and bedridden patients were excluded, 76.3% of nonsurvivors of CAP did not receive ventilatory support (62.6% of those aged < 65 years) [30]. They stated that the number of nonsurvivors without obvious reasons for withholding ventilatory support was high, particularly in younger patients, and suggested that these high percentages might reflect either treatment restrictions or deficient clinical performance. Therefore, considering future contribution to improve the clinical performance, we defined SAEs as a primary outcome measure and included death and requirement of MV or VS within 30 days after pneumonia diagnosis.

We considered that a model with high sensitivity would be preferable when identifying patients at high risk for SAEs and those who should receive higher-intensity treatment. The 2019 ATS/IDSA CAP guidelines suggest that the 2007 IDSA/ATS severity criteria should be used. However, the AUROC of the ALL-COP SCORE rule was highest among the tested predictive models including the 2007 IDSA/ATS criteria. The sensitivity of the ALL-COP SCORE rule was high at 92%, with a specificity of 60%, when the threshold to identify patients with SAEs was set at ≥ 4 points. The sensitivity of the ALL-COP SCORE rule was the highest when compared to the other tested predictive rules at the same specificity level of 60%. Several reasons might potentially explain the high predictive ability of the ALL-COP SCORE rule. First, the ALL-COP SCORE rule includes several variables such as hypercapnia and hyperkalemia, which are not included in the other tested prediction models, although they are reported as prognostic factors for pneumonia [16, 17]. Second, we adopted ternary or quaternary variables, as well as binary variables, because ternary or quaternary variables provided more accurate effects for SAEs compared with the binary variables that were used in the existing prediction rules [1,2,3, 5, 6]. Findings of the present study also suggest that the examination of some variables, such as serum albumin, PaO2, and pH, which might not be universally available for real-time clinical decision-making [4], might still be worthwhile to assess at the time of pneumonia diagnosis to improve patient care.

The predictive performance of the ALL-COP SCORE rule was improved when patients with possible treatment restrictions were excluded, suggesting that this prediction model may be more useful as a decision-making tool to identify patients requiring intensive care among those without treatment restrictions. Specifically, in the absence of treatment restrictions, physicians should aggressively consider higher-intensity treatment at the time of pneumonia diagnosis in patients with ≥ 4 points according to the ALL-COP SCORE rule. Furthermore, these patients might be strongly recommended for early interventions such as CAP bundle of care [31] (e.g., rapid empirical antibiotic administration with a β-lactam and macrolide [32,33,34], rapid resuscitation, thromboembolic prophylaxis, and appropriate management for hypoxia [35,36,37,38]).

The present study has several limitations. First, the biggest one is that data in two cohorts used in this study was obtained before the coronavirus disease 2019 (COVID-19) pandemic. We are going to validate whether the ALL-COP SCORE rule can be useful in both patient groups of COVID-19 and non-COVID-19 pneumonia. One important future research question is whether prediction models for worse outcomes can be adopted in all CAP including COVID-19 or should be separately considered in COVID-19 and non-COVID-19. Second, there were 13 constitutive variables as part of the ALL-COP SCORE rule, more than those of the other predictive rules [1, 3, 5, 6, 20]. This limitation can be overcome if the ALL-COP SCORE rule is used as a calculation tool in a medical application software or can be combined with an electric medical record system. Third, the present study did not confirm do-not-resuscitate orders at the time of admission. Elderly patients or those with poor functional status might have had various potential treatment restrictions even in the absence of a do-not-resuscitate order. Thus, we defined patients aged 80 years or older and any patient without independent ambulation as those with potential treatment restrictions. Fourth, the present study was performed in Japan which might hinder the international generalizability of the ALL-COP SCORE rule. Therefore, international collaborative studies for further validation of this model are necessary in the future. Furthermore, the efficacy and safety of the ALL-COP SCORE rule as a decision aid to guide treatment intensity should be assessed.

Conclusions

The ALL-COP SCORE is a potentially useful tool to predict patients at high risk for SAEs within 30 days after pneumonia diagnosis and can be adopted to all patients with CAP including those with HCAP. Higher-intensity treatment should be considered in patients with CAP and an ALL-COP SCORE threshold of ≥ 4 points.