BACKGROUND

Four years into the COVID-19 pandemic, there are still many unanswered questions about how best to care for patients affected by this disease. One important question is determining when it is safe to discharge patients from the hospital. Unlike other infectious diseases, early in the pandemic, it became clear that COVID-19 often got worse in the second week of illness after a period of initial stability.1 Discharging patients too soon can increase the risk that patients will get sicker after leaving the hospital, potentially leading to readmission, delays in care, and worse outcomes. But keeping patients in the hospital for an unnecessarily long time can mean overwhelming health care systems already exceeding full capacity, increasing the risk of iatrogenic complications, and creating financial and resource allocation strain on healthcare systems. It is thus critical to understand who can be discharged safely, minimizing length of stay and bed occupancy, while also minimizing post-discharge adverse outcomes such as readmission and death. Yet, we have demonstrated tremendous variability among health care systems in terms of clinical criteria for discharge.2

Observational studies on this subject have found associations between shorter hospital length of stay and readmission risk,3,4,5,6,7 suggesting that some patients were indeed discharged too soon. Other commonly found risk factors for readmission after COVID-19 have included comorbidities, older age, male sex, history of smoking, obesity, being febrile at discharge, and discharge to skilled nursing facilities or home health care services.3,4,5,6,7,8,9,10,11,12,13,14 Remdesivir treatment has been shown to be protective.15 However, most studies using detailed EHR-level data were too small to evaluate other risk factors such as vital signs other than temperature or inflammatory markers, and, to our knowledge, none compared adverse post-discharge outcomes in patients who met particular combinations of discharge criteria.

The goals of this study were to determine the risks for post-discharge adverse outcomes among patients hospitalized with COVID-19 respiratory disease, using a large, multi-center database; to develop a model to distinguish patients at high vs. low risk based on criteria present at the time of discharge; to internally and externally validate the model; and to compare its performance to other sets of published or commonly used criteria.

METHODS

Overview

We conducted a retrospective, multi-center, observational cohort study using the COVID-19 Consortium of HCA Healthcare and Academia for Research Generation (CHARGE) dataset.16 HCA Healthcare (Nashville, TN) is comprised of 185 hospitals mostly located in the Southern and Western United States. The CHARGE dataset was curated by personnel at HCA, Sarah Cannon (the cancer institute of HCA), and Genospace (Boston, MA) using clinical, billing, and administrative data from the electronic health record used at its sites (Meditech, Westwood, MA). Governance was provided by a steering committee with representation from HCA leadership, AHRQ, and ten partner academic institutions, including the Hospital Medicine Re-engineering Network (HOMERuN), a consortium of hospitalist leaders and researchers at academic medical centers throughout the USA. The study was approved by the WCG Institutional Review Board.

Setting and Participants

Inclusion criteria consisted of the following: Hospitalized at an HCA Healthcare-affiliated hospital from March 2, 2020, through February 11, 2021; positive PCR test for SARS-CoV-2 no earlier than 14 days prior to hospital admission; age 18 years or older; and primary admitting or final diagnosis of COVID-19 respiratory disease using a validated set of diagnostic ICD-10 codes.16

Exclusion criteria were nosocomial COVID-19 (first positive test > 7 days after admission); discharged against medical advice, with hospice care, or to another acute care hospital; expired during the hospitalization; admission to a non-Meditech hospital; elective or otherwise ineligible index admission (e.g., cancelled); encounters with missing diagnosis codes or discharge disposition; and discharged less than 30 days before February 11, 2021 (i.e., incomplete follow-up data).

Outcomes

The primary outcome was readmission to or hospital death at any HCA hospital within 30 days of discharge. We did not count readmissions to rehabilitation or psychiatric facilities or elective readmissions (but they would count as a death if a patient died during or subsequent to that event).

Predictors and Potential Confounders

Candidate predictors (see Appendix Table 4) included covariates generally known to be associated with readmission and ones unique to COVID-19, including patient demographics; specific comorbidities and van Walraven-Elixhauser comorbidity score;17 fever status in the 72 h prior to discharge; respiratory rate, heart rate, and blood pressure in the 24 h prior to discharge; worst oxygenation requirement in the last 24 and 72 h prior to discharge; worst and last laboratory values during hospitalization; inpatient complications of COVID-19; COVID-19 treatments; other treatments (e.g., anticoagulation at prophylactic or treatment doses); admission source; discharge disposition; days since first positive SARS-CoV-2 test; ICU stay during hospitalization; and time period of the pandemic by quarter. Potential confounders included the HOSPITAL score, a predictive tool for 30-day, potentially avoidable readmissions established and validated in 4 countries by our team.18,19 See Appendix Table 4 for categorization and proportion of missing data for each variable.

Development and Validation of the Model

Patients who suffered the primary outcome were compared to those who did not with respect to several demographic and clinical variables using descriptive statistics. The cohort was randomly divided into training (two-thirds of the eligible HCA cohort; n = 40,847) and internal validation (n = 21,347) sets. Missing categorical variables were imputed with the mode or “missing” was added as a category level, depending on the degree of missingness (see Appendix Table 4), and continuous variables were imputed with the median. To create a clinically meaningful and interpretable subset of candidate predictors, we used a two-step procedure. First, we fit a logistic model on the training data using least absolute shrinkage and selection operator (LASSO) regression.20 We used tenfold cross-validation with deviance as the loss to select the optimal regularization penalty tuning parameter (lambda). We chose the largest value of lambda with a deviance within one standard error of the minimum to prioritize parsimony.20 We then fit a logistic regression model using unpenalized maximum likelihood to the training data set using the variables selected in the LASSO regression to report adjusted odds ratios and confidence intervals for each variable.21 To obtain unbiased estimates of the confidence intervals in this final model, we employed multiple imputation (using the MICE package in R)22 for any remaining continuous variables by creating ten multiply-imputed data sets, using predictive mean matching and Rubin’s rules to combine parameter estimates and derive standard errors. In these models, we added a small set of interaction terms based on subgroup analyses to look for effect modification: by discharge destination (home, SNF/Rehab/LTAC, other or missing), respiratory rate at discharge, receipt of remdesivir, receipt of corticosteroids, and pandemic quarter.

To evaluate the predictive performance of our model, we calculated the area under the receiver operating characteristic (ROC) curve (AUC) for the training data set and the remaining one-third of the CHARGE cohort (internal validation). We also produced calibration diagrams for the internal validation cohort. We identified a high-risk and low-risk cohort for our outcome using the cutoff of risk that maximized the product of sensitivity and specificity using the cutpointr package in R.23 To externally validate our model, we collected comparable variables from March 2 to December 31, 2020, from seven hospitals in the HOMERuN consortium who agreed to provide their data, including Brigham and Women’s Hospital, BWH Faulkner Hospital (BWH’s community affiliate), Massachusetts General Hospital, University of California San Francisco Medical Center, Northwestern Memorial Hospital, Johns Hopkins Hospital, and NYU Langone Health. Johns Hopkins Hospital could not provide an EHR indicator for ICU admission (because of the conversion of general wards to temporary ICUs); therefore, the initiation and discontinuation of mechanical ventilation or ECMO was used as a proxy for ICU admission and discharge, respectively.

Lastly, we compared the model developed by our analyses with discharge criteria from the published literature or available online from hospitals in the HOMERuN network. Comparators included guidelines from CDC, University of Michigan, Johns Hopkins, Levine et al.,24 and a clinical gestalt. The guidelines for these criteria are included in Appendix Table 5. Using the CHARGE internal validation dataset, we compared the proportion of patients determined to be high risk, the risk of readmission or death in the high-risk vs. low-risk cohorts, the diagnostic odds ratio (and 95% confidence intervals), and the likelihood ratio positive and negative (and 95% confidence intervals) for being high risk and low risk, respectively.

Analyses were conducted using SAS v.9.4 (SAS Institute, Cary, NC) and R v.4.1.3 (R Foundation for Statistical Computing, Vienna, Austria). Unless otherwise stated, two-sided p-values < 0.05 were considered significant.

RESULTS

Figure 1 illustrates the flow diagram. The original cohort consisted of 125,436 merged hospital encounters, while the final cohort, after data cleaning and exclusions, resulted in 62,195 patients. Most patients were excluded due to the absence of COVID-19 respiratory disease; died in the hospital, transferred to another acute hospital, or discharged against medical advice; or did not have 30 days of post-discharge follow-up by the dataset cutoff date. Mean age of the cohort was 61.9 years, and 51.9% were male. In this cohort, 4704 (7.6%) were readmitted or died within 30 days of discharge (7.2% were readmitted, 1.6% died). Characteristics of those patients who were readmitted or died within 30 days of discharge compared to those who did not are shown in Table 1. Those who suffered the primary outcome were generally older, more likely to be male, to be White, to be non-Hispanic, to be a former smoker, to have a shorter length of stay, to be discharged to a destination other than home, to be with a COVID-19 complication of CHF or AKI in the hospital, and less likely to receive remdesivir.

Figure 1
figure 1

Study flow diagram.

Full size image
Table 1 Patient Characteristics
Full size table

In the final multivariable logistic regression model (Table 2), several factors were independently associated with 30-day readmission or death. These included fever within 72 h of discharge; tachypnea, tachycardia, or use of supplementary oxygen in the 24 h prior to discharge; lack of improvement in oxygen requirement (i.e., worst requirement in the last 24 h no better than the worst requirement in the 72 h prior to discharge); lymphopenia or thrombocytopenia at the time of discharge; Elixhauser comorbidity score of 6 or greater; current or former smoker; history of hypertension, chronic obstructive pulmonary disease (COPD), diabetes mellitus, chronic kidney disease, cardiovascular disease, or cancer; hospitalization complicated by heart failure; discharge ≤ 7 days since first positive test for SARS-CoV-2; age 50 or older (compared with age 18–39); male sex; treatment with corticosteroids any time during the hospitalization; discharge destination other than home; and HOSPITAL readmission risk score 5 or greater. Inpatient treatment with remdesivir and prophylactic or treatment-dose anticoagulation were associated with lower odds, as were number of days in the intensive care unit (ICU) and discharge directly from ICU. Notably non-significant factors included BMI and improvement or normalization of inflammatory markers such as C-reactive protein, d-dimer, ferritin, lactate dehydrogenase, and troponin. Using the beta-coefficients from Table 2, the odds of death or readmission in a given patient can be calculated as e[−4.53+ ß1(fever status) + ß2(respiratory rate) + ß3(Heart Rate) + …], and the probability of death or readmission is odds/(1 + odds) (see Supplementary Material for an Excel spreadsheet to calculate risk).

Table 2 Independent Risk Factors for Primary Outcome in Logistic Regression
Full size table

Out of a select few chosen interaction terms identified by subgroup analyses, the only two that were significant were abnormal respiratory rate at discharge by receipt of remdesivir (higher odds of primary outcome than otherwise expected from the two terms individually) and receipt of corticosteroids by discharge to Other location (shelter, prison, or congregate living facility) (lower odds than expected).

The model was well calibrated, as the observed and expected 30-day mortality or re-admission rates in the validation data were highly correlated. Calibration was less precise at the lowest decile (2.1% observed vs. 1.1% predicted) and highest decile (22.4% observed vs. 23.7%) compared with the middle deciles of risk (Fig. 2a). The model discrimination as measured by AUC in the derivation set was 0.75 (95% confidence interval (CI) 0.74–0.76). In the internal validation set (N = 21,347), the AUC was 0.73 (95% CI 0.71–0.74) (Fig. 2b). Using the cutoff of risk (7.8%) that maximized the product of sensitivity and specificity, the model distinguished between 7464 high-risk patients (35% of the internal validation cohort) with a 14.6% probability of readmission or death (i.e., positive predictive value) from 13,883 low-risk patients (65% of the internal validation cohort) with a 4.1% probability of readmission or death (i.e., negative predictive value of 95.9%). The sensitivity of the model was 65%, specificity 68%, and accuracy 67%.

Figure 2
figure 2

a Model calibration plot for internal validation cohort. b Receiver operating characteristic (ROC) curve for internal validation cohort.

Full size image

In the external validation cohort (N = 11,338), 8.7% were readmitted, 0.7% died, and 9.0% were readmitted or died. Compared to the CHARGE cohort, the external validation cohort was younger, with more current smokers, fewer complications of CHF and AKI, and less use of remdesivir and corticosteroids (Appendix Table 6). The AUC of our model in the external validation cohort was 0.66 (95% CI 0.64–0.67). In this cohort, the model could distinguish between 21% of the patients as high risk, with 16.8% probability of readmission or death, and the remaining 79% of patients with a 6.9% probability of readmission or death. As shown in Table 3, using the CHARGE cohort, our model (HCA) outperformed the other models it was tested against. Specifically, the likelihood ratio for the primary outcome of being high risk in our model was 2.02 (95% CI 1.94–2.10), while it varied from 1.02 to 1.16 for the other models.

Table 3 Comparison of Clinical Criteria for Discharge
Full size table

DISCUSSION

In this large, multi-center retrospective cohort study of patients hospitalized with COVID-19 respiratory disease during the first year of the pandemic, 7.6% of patients were readmitted or died within 30 days of discharge. Independent predictors of the primary outcome included a variety of demographic and clinical factors present at the time of discharge. Our model had moderate discrimination when internally validated, fair discrimination when externally validated, and performed better than other criteria that have been recently published or were in common use in hospitals in our consortium.

The proportion of readmission or death within 30 days in our cohort is consistent with other COVID-19 studies,7,10,25 including a meta-analysis that found a 30-day readmission rate of 9.0% and mortality rate of 7.9%.26 These rates are lower than the readmission rates seen in typical medical patients.27 We hypothesize several reasons for this finding: COVID-19 patients tended to have less bio-psychosocial complexity than typical medical patients and longer lengths of stay, and once patients recovered from COVID-19 respiratory disease and/or were past their first week of illness, they tended not to have recrudescent disease. This hypothesis may also explain why days in the ICU and discharge directly from the ICU were associated with lower risk. It is also notable that shorter length of stay was associated with higher risk, which has been noted in other studies, and again lends strength to the hypothesis that some patients with COVID-19 were indeed discharged too soon. In almost every other observational study of medical patients, higher length of stay is associated with increased risk of readmission because both are markers of clinical severity.19,28 For this positive association to be reversed in the COVID-19 cohort, that is, to overcome confounding by severity of illness, suggests that the negative effect of premature discharge might be quite large. Our study also lends support to the observation that patients discharged in the first week of illness can deteriorate later in their course, at least during the first year of the pandemic in an immune-naïve population and prior to variant evolution. Patients discharged to facilities instead of home also had a higher risk of readmission or death; this association has been shown in other cohorts with4,7 and without COVID-1929 and is likely because discharge to facilities other than home is a surrogate marker of frailty.

It is notable that patients who received prophylactic or treatment dose anticoagulation were at lower risk for readmission or death. The effect of prophylactic vs. treatment-dose anticoagulation on non-critically ill hospitalized patients with COVID-19 has been controversial, with some studies (REMAP-CAP, ACTIV-4a, ATTACC, HEP-COVID)30,31 showing benefit of therapeutic dosing, and others (ACTION)32 not. These studies did not include readmission as an outcome. Our study lends support to the hypothesis that anticoagulation might be beneficial in preventing poor post-discharge outcomes in patients with COVID-19, but this would require further and longer-term investigation. It is also notable that receipt of remdesivir was associated with lower risk of poor outcomes (as has been noted in other studies)15 but that receipt of corticosteroids was observed to have the opposite effect (also shown in other studies),11 even after adjustment for several risk factors. It is biologically plausible that stopping anti-inflammatory medications such as dexamethasone at the time of discharge, as is commonly done, could lead to recrudescence of disease, but we cannot also rule out residual confounding by indication (i.e., patients perceived as sicker by their clinicians received steroids, based on unmeasured factors such as shortness of breath, abnormal findings on lung exam, and abnormal findings on chest imaging).

Most of the clinical factors associated with readmission or death were not surprising, but it was notable that none of the inflammatory markers we tested were significant in our models. We introduced these variables into the models in several ways, including absolute levels at discharge and relative improvement during the hospitalization, but none of these influenced the results. It may also be that the amount of missing data for these fields limited our ability to find significant effects. BMI also did not matter despite being a known risk factor for severe disease at the time of admission; BMI may be collinear with other comorbidities that have a greater association with risk. Race and ethnicity were associated with outcome in bivariable analyses, possibly reflecting differences in exposure to COVID-19 and access to care, but they were not significant after multivariable adjustment.

This model could be used to influence decision-making as a patient nears time for hospital discharge. For example, high-risk patients with modifiable risk factors (e.g., lack of improvement in oxygen requirement, fever, tachypnea, or tachycardia) are worthy of consideration for delaying discharge until these risk factors resolve, whereas high-risk patients with non-modifiable factors (e.g., age, comorbidities, nosocomial complications) may be considered for discharge to a more monitored post-acute setting or enrollment in a home monitoring program.33,34,35 Both of these strategies, and use of our model, are worthy of evaluation in prospective studies.

We used a cutoff for high risk that maximized the product of sensitivity and specificity in our model. One might argue that to prevent an outcome such as readmission or death, a cutoff that favors sensitivity might be preferable. However, a 4.1% probability in the low-risk cohort is already quite low compared with typical medical inpatients. Using a lower cutoff would reduce the risk in the low-risk cohort but at the cost of identifying more patients at high risk, potentially prolonging length of stay in these patients.

An important question is the relevance of this model, based on data from the first year of the pandemic, in the context of the evolving epidemiology of COVID-19. Many of the predictive factors are simply markers of clinical instability or comorbidity and are unlikely to have changed substantially. At least one study, from the UK, showed the continued relevance of risk scores generated during the first wave of the pandemic to predict outcomes during the second wave.12 The main unanswered question is the extent to which patients can still deteriorate after a period of initial improvement. There are emerging (unpublished) data that recrudescence after the first week of illness is currently rare but still present in patients who are immunocompromised, especially on rituximab and other B cell–depleting treatments, where there is evidence that the period of viremia is prolonged.36,37 There is also the phenomenon of rebound after treatment with oral antiviral treatment such as nirmatrelvir/ritonavir, but these events are relatively transient and mild.38,39,40

The results of our study are consistent with prior studies of readmission after COVID-19 hospitalization41,42 but also adds to what is currently known. Several studies have suggested that being febrile at discharge might be a risk factor; our study strongly supports that finding. Somani et al. found a trend towards less anticoagulation as a risk factor,6 again something our study more strongly supports. Other studies have also found the association with discharge to destinations other than home as a risk factor.4,7,14 Our study, due to its size, identifies several additional risk factors not previously established, including tachycardia and lack of improvement in oxygen requirement in the 72 h prior to discharge, and the predictive value of the HOSPITAL score, which has been shown to be useful in general medical patients but has not previously been shown to be useful in patients with COVID-19.

The results of this study should be viewed in light of its limitations. While large and diverse, this cohort is from one health care system (mostly community hospitals in the Southern US), which could limit its generalizability (and the model performed less well when externally validated in other health care systems). The analysis suffers from missing data, especially for some laboratory values, that might have limited our ability to find significant effects, as noted above. Missing data was particularly an issue with the external validation cohort, which may partly explain the poorer performance of the model in that cohort; another possible explanation are differences in demographics and in the use of treatments for COVID-19 in the two cohorts. Specifically, the external validation cohort consisted of academic medical centers, which likely cared for patients with higher comorbidity and where different factors may play a role in readmission risk. The study is also limited by the types of variables that could be collected; e.g., it did not include the presence of social support systems that can impact readmission risk.14 Outcome assessment may also have been incomplete, e.g., deaths not known to the health systems in the cohort, which could have limited the discrimination of our model. This current analysis only covers the first year of the pandemic; changes in the standard of care (e.g., in the use of dexamethasone), vaccination, and in the nature of the newest strains of SARS-CoV-2 could change these results, as noted above.

We also did not measure contextual factors, like the effects of periodic surges of patients with COVID-19 on bed capacity, that could have influenced outcomes.43 On the other hand, to our knowledge, this is one of the largest studies to evaluate post-discharge outcomes in the USA using detailed, EHR-level data.

In conclusion, in one of the largest retrospective studies to be conducted on this subject, we identified several factors that were associated with post-discharge readmission or death with moderate discrimination and good calibration, many of which could influence clinical decision-making at the time of discharge in patients hospitalized for COVID-19.