Skip to main content

Electronic Health Record Mortality Prediction Model for Targeted Palliative Care Among Hospitalized Medical Patients: a Pilot Quasi-experimental Study

Abstract

Background

Development of electronic health record (EHR) prediction models to improve palliative care delivery is on the rise, yet the clinical impact of such models has not been evaluated.

Objective

To assess the clinical impact of triggering palliative care using an EHR prediction model.

Design

Pilot prospective before-after study on the general medical wards at an urban academic medical center.

Participants

Adults with a predicted probability of 6-month mortality of ≥ 0.3.

Intervention

Triggered (with opt-out) palliative care consult on hospital day 2.

Main Measures

Frequencies of consults, advance care planning (ACP) documentation, home palliative care and hospice referrals, code status changes, and pre-consult length of stay (LOS).

Key Results

The control and intervention periods included 8 weeks each and 138 admissions and 134 admissions, respectively. Characteristics between the groups were similar, with a mean (standard deviation) risk of 6-month mortality of 0.5 (0.2). Seventy-seven (57%) triggered consults were accepted by the primary team and 8 consults were requested per usual care during the intervention period. Compared to historical controls, consultation increased by 74% (22 [16%] vs 85 [63%], P < .001), median (interquartile range) pre-consult LOS decreased by 1.4 days (2.6 [1.1, 6.2] vs 1.2 [0.8, 2.7], P = .02), ACP documentation increased by 38% (23 [17%] vs 37 [28%], P = .03), and home palliative care referrals increased by 61% (9 [7%] vs 23 [17%], P = .01). There were no differences between the control and intervention groups in hospice referrals (14 [10] vs 22 [16], P = .13), code status changes (42 [30] vs 39 [29]; P = .81), or consult requests for lower risk (< 0.3) patients (48/1004 [5] vs 33/798 [4]; P = .48).

Conclusions

Targeting hospital-based palliative care using an EHR mortality prediction model is a clinically promising approach to improve the quality of care among seriously ill medical patients. More evidence is needed to determine the generalizability of this approach and its impact on patient- and caregiver-reported outcomes.

INTRODUCTION

Hospitals nationwide have invested in palliative care programs to improve the quality of care for seriously ill patients.1 Hospital-based palliative care consultation has been shown to decrease length of stay (LOS), readmission rates, and health care costs.2,3,4,5,6 Palliative care has also been shown to increase survival in some patient populations.7 Emerging evidence further suggests that earlier palliative care consultation leads to greater family satisfaction with care.8 Despite these benefits, the frequency and timing of consultation is highly variable among patients with non-cancer diagnoses,9 due largely to a reliance upon clinicians to identify palliative care needs and refer patients for consultation. Thus, innovative and systematic strategies are needed to augment clinician referral of patients most likely to benefit from earlier palliative care consultation.10,11,12

There has been a growing interest in triggering palliative care consultation using several different criteria, such as selected diagnoses,13, 14 disease-specific prognostic indicators,15, 16 or a patient’s location.17,18,19 Although sound in principle, these approaches are nonspecific and assume palliative care needs and are therefore unlikely to be scalable given the growing number of seriously ill patients and the limited palliative care specialist workforce in the USA.20, 21 The rapid growth of data science combined with the breadth of available data in the electronic health record (EHR) may allow for more timely identification of patients based on actual palliative care needs using predictive analytics. Indeed, several machine learning EHR models recently developed to predict individual risk of mortality and other adverse health outcomes among hospitalized patients represent an important step in this direction.22,23,24,25,26,27 Such prediction models offer great promise for systematically identifying patients most in need, yet none have been coupled with a palliative care intervention to evaluate clinical impact.28, 29

We hypothesized that Palliative Connect, an intervention that combines predictive analytics with early triggered consultation, would improve the delivery of hospital-based palliative care. Thus, we conducted a pilot study to evaluate the clinical impact of triggering palliative care consultation (with clinician opt-out) on hospital day 2 among a diverse general medical population with high predicted risk of death within 6 months. We also explored the acceptability of Palliative Connect among clinicians and spill-over effects among lower risk patients.

METHODS

This pilot study was conducted among general medical patients on the hospitalist services at the Hospital of the University of Pennsylvania, a 791-bed tertiary care institution with over 34,000 adult admissions annually. Nine hospitalist teams care for an average of 85 patients daily. Each team is staffed by one attending hospitalist physician and either a certified registered nurse practitioner or three resident physicians. A multidisciplinary palliative care program that was established 9 years ago completes approximately 10 new consults and maintains a daily census of approximately 60 each day.

An 8-week intervention period was established a priori from December 1, 2017, to February 8, 2018 (excluding the holiday week) and compared with an 8-week historical control period. There were no other palliative care initiatives on the hospitalist services during the study period. This project was determined to qualify as quality improvement by the University of Pennsylvania Institutional Review Board; need for informed consent was waived.

Palliative Connect Intervention

Most existing EHR machine learning mortality prediction models are limited by inclusion of a restricted population,22, 23, 25 reliance on data only available at the end of a hospitalization,24 or use of computationally complex methods that can hinder clinician interpretability and future deployment into the EHR production environment,27, 30 and none have been externally validated. Thus, we first developed and validated an EHR mortality prediction model with this health system’s data using a machine learning approach and regression analysis. We chose the predicted outcome of death within 6 months to identify seriously ill patients who would likely benefit from advance care planning (ACP) discussions in a timeframe sufficient for such discussions to potentially influence the likelihood that they receive goal-concordant care in the final stages of disease.31, 32

We trained the supervised machine learning model using 64,246 admissions among 46,305 unique patients (excluded patients < 18 years of age and observation, obstetrics, hospice, or rehabilitation admissions) to three large urban hospitals at the University of Pennsylvania Health System in 2016; all used EPIC v2017 (EPIC Systems Corporation, Verona, WI). Characteristics between the training and test cohorts are summarized in eTable1 in the online supplement. Six-month mortality was determined from the date of hospital admission and using death dates in the EHR or Social Security Death Index (SSDI).33 The 6-month mortality rates were 9.9% and 10.1% in the training and test sets, respectively. See the online supplement for additional details of the machine learning methods used for prediction model development and analyses. The final prediction model retained 35 covariates (age, gender, admission type, 18 Elixhauser comorbidities, and 14 laboratory values) significantly associated with risk of six-month mortality (online supplement eTable2). The model demonstrated excellent discrimination in the test set, with a c-statistic of 0.86 (95% CI 0.84–0.88) and good calibration, with the expected mortality within the 95% CIs of observed mortality except at the extreme deciles where there were few cases with the outcome or low sample sizes to inform the model.34 eFigure1 and eFigure2 in the online supplement summarize the performance characteristics of the final model in the test set.

We then performed a chart review to determine the mortality risk (Palliative Connect score) threshold above which patients were most likely to need a palliative care consult for ACP discussion and documentation.35, 36 Two palliative medicine specialists (KC and NO) reviewed 30 random charts using data available in the EHR through hospital day 1 while blinded to the predicted mortality risk. Among those with a Palliative Connect score of ≥ 0.3 (13/30), the inter-rater reliability was 100% (Cohen’s Kappa = 1.0).

Finally, we built an intranet-based user interface to bring eligible patients to the attention of the palliative care team each day in a user-friendly format, and to track communication with the primary team clinicians (online supplement eFigure3). Patients ≥ 18 years of age at the time of hospital admission, on the hospitalist service, and with a Palliative Connect score ≥ 0.3 were eligible for a triggered consult using the intervention workflow shown in Fig. 1. Recognizing that patient identification alone is likely insufficient to change clinicians’ palliative care referral patterns, we applied insights from behavioral economics regarding the use of default (opt-out) options to nudge clinicians to deliver recommended care without restricting their choices.37,38,39 Thus, the palliative care triage nurse offered a triggered consult to the primary clinician of patients on the list in descending order until a maximum of two consults were accepted. This approach preserved the palliative care team’s capacity for usual care consults without adding resources.40 Remaining patients were carried over on the list each day until they were offered a triggered consult or discharged or transferred to another service. The content of triggered consults was left to the discretion of the palliative care clinician.

Figure 1
figure1

Study process flow for palliative connect intervention. Each weekday, a Palliative Connect score (predicted risk of death within 6 months) was calculated for patients on hospital day 2. Patients with a score ≥ 0.3 populated a web-based user interface list in order from highest to lowest risk (actual prediction not shown). The palliative care team’s triage nurse called the primary clinicians of patients in descending order to offer an opt-out of the triggered consult until the maximum of two consults were accepted. Remaining patients were carried over on the list each day until they were offered a triggered consult, or they were discharged or transferred to another service. Consults requested per usual care were accepted. Palliative care clinicians and hospitalists completed surveys and clinical data was obtained from the clinical data warehouse. Figure 1 and 2 contains poor quality of text inside the artwork. Please do not re-use the file that we have rejected or attempt to increase its resolution and re-save. It is originally poor, therefore, increasing the resolution will not solve the quality problem. We suggest that you provide us the original format. We prefer replacement figures containing vector/editable objects rather than embedded images. Preferred file formats are eps, ai, tiff and pdf.fully vectorized .eps files attached along with original powerpoint figure file

Historical Patient Controls

To create a comparable cohort of control patients, we ran the prediction risk algorithm silently for 8 weeks prior to the intervention period. Patients were selected for inclusion in the control group using the same eligibility criteria as the intervention group, followed by a stochastic simulation41 that mimicked the descending Palliative Connect scores on the user interface, the triggered consult decline rate observed in the intervention period, and the limit of two accepted triggered consults per day. This approach could not account for other potential unmeasured factors in the opt-out decision process.

Outcomes

For all control and intervention patients, we evaluated several palliative care processes and outcomes, including completed consults (triggered and usual care), pre-consult length of stay (LOS), ACP documentation (defined by the presence of a completed “ACP note” type in the EHR), home palliative care and hospice referrals, change in code status (defined by a new do-not-attempt-resuscitation [DNAR] order in the EHR), hospital mortality, hospital LOS, intensive care unit (ICU) admission and LOS, and all-cause readmission within 30 days of discharge.

For intervention patients, we explored acceptability of the intervention by the proportions of (1) triggered consults declined by a patient and/or caregiver after the primary clinician accepted it; (2) hospitalists who agreed or strongly agreed that triggering palliative care for patients most likely to benefit is acceptable; and (3) completed consults determined to be appropriate by the palliative care clinician. We also collected triggered consult decline reasons. Clinician data were collected from post-consult surveys.

Finally, we evaluated for spill-over effects on palliative care consultation rates among two non-study populations on the hospitalist services: (1) patients with a risk score < 0.3 and (2) patients with a risk score ≥ 0.3 who were discharged or transferred services before their clinician was offered a triggered consult. All clinical data were retrieved from Penn Data Store, a data warehouse that includes EHR and post-discharge coded data.

Statistical Analyses

Analyses for this pilot study were primarily descriptive using summary statistics as appropriate. The analytic sample included all admissions offered a triggered consult. Comparisons between intervention and control groups were performed using Pearson’s chi-square or Fisher exact tests for categorical variables, and two-tailed t tests and Wilcoxon rank-sum tests for normally and non-normally distributed continuous variables, respectively. Statsmodels v0.6.142 in the Python programming language was used for all analyses. Data are presented as number (percent) unless otherwise specified. Findings were considered statistically significant at P < 0.05. The datasets generated and/or analyzed during the study are not publicly available due to protected patient data, but may be made available from the corresponding author on reasonable request.

RESULTS

During the 8-week intervention period, 946 medical admissions to the hospitalist services were automatically screened in the EHR, 134 (14) of which were offered a triggered a consult (Fig. 2). Patient demographics and mortality risk scores in the control (n = 138) and intervention cohorts were similar (Tables 1 and 2).

Figure 2
figure2

Palliative Connect study screening and enrollment. Screening occurred automatically in the EHR every day at 0630. The palliative care triage nurse called the primary clinicians of patients with a Palliative Connect score ≥ 0.3.

Table 1 Patient Characteristics
Table 2 Palliative Connect Outcomes

Process Measures and Outcomes

During the intervention period, primary clinicians accepted 77 (57) triggered consults and 8 additional consults were requested per usual care, with 3 occurring after the triggered consult was declined. Table 2 summarizes the clinical findings from this study. Palliative care consults increased by 74% in the intervention group compared to control (22 [16] vs 85 [63], P < .001), and occurred 1.4 days earlier after admission (2.6 [1.1, 6.2] vs 1.2 [0.8, 2.7], P = .02). There was a 38% increase in ACP documentation (23 [17] vs 37 [28], P = .03) and 61% increase in home palliative care referrals (9 [7] vs 23 [17], P = .01) during the intervention period. There were fewer ICU admissions, in-hospital deaths and 30-day readmissions, and more hospice referrals among the intervention group compared to historical controls, but these differences were not statistically significant. The median hospital LOS and proportion of patients with a change in code status were unchanged between the groups.

Table 3 Reasons Provided by Primary Team for Declining Triggered Palliative Care Consult (n = 57)

Acceptability of Triggered Palliative Care

Among the 57 (43) declined triggered consults, the most common reason cited by primary clinicians was “the patient has no palliative care needs at this time” (Table 3). No accepted triggered consults were declined by a patient or caregiver. Of the hospitalists who completed a post-consult survey (13/29; RR 45%), 10 (77) agreed or strongly agreed that triggering palliative care consults (with opt-out) for patients most likely to benefit was acceptable, and 3 (23) neither agreed nor disagreed. For all completed consults among the intervention cohort (81/85; RR 95%), palliative care clinicians were more likely to report that traditionally requested consults were appropriate compared to triggered consults (7 [100] vs 49 [66]; P = .04). However, when the 33 additional consults requested for patients with a risk score < 0.3 were included, there was no difference in reported appropriateness (P = .18).

Spill-Over Effects of Triggered Palliative Care

Consultations among hospitalist admissions with a Palliative Connect score < 0.3 remained unchanged between the intervention and control periods (33/798 [4] vs 48/1004 [5]; P = .48), and similarly for admissions with a risk score ≥ 0.3 who were discharged or transferred services before the trigger (0/11 [0] vs 1/17 [6]; P = .41).

DISCUSSION

Increasing access to palliative care is a major focus of national efforts to improve the quality of serious illness care.43, 44 This pilot study suggests that using an EHR mortality prediction model to trigger palliative care may improve timely identification of seriously ill medical patients and increase consultation among this vulnerable population. Specifically, we found that palliative care consultation increased nearly fourfold compared to historical controls and occurred a day and a half earlier, along with higher ACP documentation rates.

The Palliative Connect prediction model identified hospitalized patients with a high risk of death within 6 months of admission who would not have otherwise received a consult. When nudged to consider palliative care for these patients, the primary team accepted more than half of the triggered consults. For declined consults, clinicians most often cited either a lack of palliative care needs or a belief that palliative care needs were already being met. Importantly, we did not assess clinician-patient agreement in these cases. Striking a balance between the needs of patients, hospitalist teams, and palliative care clinicians was essential to ensure feasibility and acceptability of this intervention. Primary teams were not prevented from requesting consults at any time during the study, but rather by limiting the triggered consults seen each day the palliative care team was still able to see all consults requested per usual care. However, it is evident from the triggered consult declination rates and palliative care clinicians’ assessments of consult appropriateness that further evaluation is needed to define the optimal trigger timing across different diseases and care settings.

Another important result of Palliative Connect was the increase in referrals to home palliative care. Such increased access to palliative care across the care continuum is essential to improving the quality of serious illness care, as patients spend the majority of time out of the hospital and the evidence for community-based palliative care to improve patient and caregiver outcomes is strong.45 We also saw a trend toward increased hospice referrals, which is consistent with findings from a retrospective study of outcomes after palliative care consultation conducted in this study’s health system.5 Coupled with the increase in ACP documentation, this suggests that implementation of Palliative Connect may facilitate delivery of goal-concordant care, although this remains an elusive outcome to directly measure. Importantly, despite more palliative care consults and ACP documentation, there was not an increase in new DNAR orders during the intervention period. This finding should alleviate, at least in part, persistent misconceptions that palliative care is analogous to hospice or limiting treatment options.

In addition to providing new insights on key questions about clinician acceptability of triggering palliative care for hospitalized medical patients and its potential impact on important palliative care process metrics, this study also revealed promising trends in clinical outcomes. There were clinically meaningful reductions in-hospital mortality, 30-day all-cause readmissions, ICU admissions, and ICU LOS. Although we did not design this pilot study to detect statistically significant differences, these early findings are supported by existing observational evidence of the value of palliative care.

The Palliative Connect intervention has several key strengths that lend it to real-world applicability, including (1) systematic identification of patients at risk for poor health outcomes, (2) risk stratification to allow for selection of a desired threshold, (3) delivery of actionable information to clinical teams in real time, and (4) preservation of primary clinician decision autonomy. Moreover, as there is significant heterogeneity across hospitals and palliative care programs,46, 47 this intervention design enables adaptation to meet local implementation needs with subsequent evaluation of outcomes. For example, the optimal number of triggered consults seen each day, type of clinician nudge, and mortality risk threshold are likely to differ by site based on available resources, the culture of palliative care, and patient case-mix. Similarly, considerations to mitigate potentially unmanageable increases in consult volume21 and to facilitate scalability and sustainability might include using the prediction model to deliver primary clinician-directed educational interventions or nudges for “primary” palliative care before a consult is triggered, or for direct referrals to community-based palliative care, although it is not yet clear which patients would benefit from which intervention. Finally, incorporation of the perspectives and preferences of patients and caregivers for implementation could enhance acceptability.

Limitations

There are several important limitations of this study. First, this was a non-randomized, single-center study among general medical patients that used historical controls, so whether Palliative Connect could be generalizable to other inpatient populations is yet unknown. We opted not to randomize individual patients because of the potential for significant spill-over effects of increased consultation among non-study patients being treated by the same primary clinician, and alternate study design options that would randomize at the clinician or unit level were not practical for this small pilot study. Thus, differences in outcomes between the historical control and intervention groups may have been attributable to unmeasured patient characteristics, temporal trends, or a Hawthorne Effect among the primary team or palliative care clinicians. Second, this study was underpowered to detect statistically meaningful differences in clinical outcomes, so caution is warranted in interpretation of the intervention’s effectiveness. Third, a mortality risk-based trigger relies on the widely held belief that patients with higher risk have greater palliative care needs. In reality, patients’ actual palliative care needs are influenced by a combination of their prognosis, diagnoses, clinical team, and sociodemographic factors. Finally, all prediction model clinical decision-making tools are at risk for perpetuating human biases reflected in the underlying data, and for capturing practice patterns and case-mix at one time point.30 Thus, rigorous assessments of fairness to promote health equity48 and periodic model re-evaluation and recalibration are essential to ensure that the model remains valid and useful.49

CONCLUSION

The Palliative Connect intervention demonstrated promising evidence of clinician acceptability and clinical impact among a general medical population at a large academic hospital. Such innovative palliative care delivery approaches that target patients most likely to benefit are critical to the field’s ability to sustainably provide high-value care. A randomized evaluation of Palliative Connect is needed in a more diverse population and setting to determine its effectiveness, including assessments of patient- and caregiver-reported outcomes.

References

  1. 1.

    Dumanovsky T, Augustin R, Rogers M, Lettang K, Meier DE, Morrison RS. The Growth of Palliative Care in U.S. Hospitals: A Status Report. Journal of Palliative Medicine 2016;19:8–15.

    PubMed  PubMed Central  Google Scholar 

  2. 2.

    Bharadwaj P, Helfen KM, Deleon LJ, et al. Making the Case for Palliative Care at the System Level: Outcomes Data. Journal of Palliative Medicine 2016;19:255–8.

    PubMed  Google Scholar 

  3. 3.

    May P, Normand C, Cassel JB, et al. Economics of Palliative Care for Hospitalized Adults With Serious Illness: A Meta-analysis. JAMA Intern Med 2018;178:820–9.

    PubMed  PubMed Central  Google Scholar 

  4. 4.

    Morrison RS, Penrod JD, Cassel JB, et al. Cost savings associated with US hospital palliative care consultation programs. Archives of Internal Medicine 2008;168:1783–90.

    PubMed  Google Scholar 

  5. 5.

    O’Connor NR, Junker P, Appel SM, Stetson RL, Rohrbach J, Meghani SH. Palliative Care Consultation for Goals of Care and Future Acute Care Costs: A Propensity-Matched Study. Am J Hosp Palliat Care 2018;35:966–71.

    PubMed  Google Scholar 

  6. 6.

    O’Connor NR, Moyer ME, Behta M, Casarett DJ. The Impact of Inpatient Palliative Care Consultations on 30-Day Hospital Readmissions. Journal of Palliative Medicine 2015;18:956–61.

    PubMed  Google Scholar 

  7. 7.

    Temel JS, Greer JA, Muzikansky A, et al. Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med 2010;363:733–42.

    CAS  Google Scholar 

  8. 8.

    Carpenter JG, McDarby M, Smith D, Johnson M, Thorpe J, Ersek M. Associations between Timing of Palliative Care Consults and Family Evaluation of Care for Veterans Who Die in a Hospice/Palliative Care Unit. Journal of Palliative Medicine 2017;20:745–51.

    PubMed  PubMed Central  Google Scholar 

  9. 9.

    Beernaert K, Cohen J, Deliens L, et al. Referral to palliative care in COPD and other chronic diseases: a population-based study. Respir Med 2013;107:1731–9.

    PubMed  Google Scholar 

  10. 10.

    Abedini NC, Chopra V. A Model to Improve Hospital-Based Palliative Care: The Palliative Care Redistribution Integrated System Model (PRISM). J Hosp Med 2018;13:868–71.

    PubMed  Google Scholar 

  11. 11.

    Courtright KR, Cassel JB, Halpern SD. A research agenda for high-value palliative care. Ann Intern Med 2018;168:71–2.

    PubMed  Google Scholar 

  12. 12.

    Hui D, Mori M, Meng YC, et al. Automatic referral to standardize palliative care access: an international Delphi survey. Supportive Care in Cancer 2018;26:175–80.

    PubMed  Google Scholar 

  13. 13.

    Gruhler H, Krutka A, Luetke-Stahlman H, Gardner E. Determining Palliative Care Penetration Rates in the Acute Care Setting. J Pain Symptom Manage 2018;55:226–35.

    PubMed  Google Scholar 

  14. 14.

    Courtright KR, Madden V, Gabler NB, et al. Rationale and design of the Randomized Evaluation of Default Access to Palliative Services (REDAPS) trial. Annals of the American Thoracic Society 2016;13:1629–39.

    PubMed  PubMed Central  Google Scholar 

  15. 15.

    Hussain J, Adams D, Allgar V, Campbell C. Triggers in advanced neurological conditions: prediction and management of the terminal phase. BMJ Support Palliat Care 2014;4:30–7.

    PubMed  Google Scholar 

  16. 16.

    Rocque GB, Campbell TC, Johnson SK, et al. A Quantitative Study of Triggered Palliative Care Consultation for Hospitalized Patients With Advanced Cancer. J Pain Symptom Manage 2015;50:462–9.

    PubMed  Google Scholar 

  17. 17.

    Hurst E, Yessayan L, Mendez M, Hammad A, Jennings J. Preliminary Analysis of a Modified Screening Tool to Increase the Frequency of Palliative Care Consults. Am J Hosp Palliat Care 2018;35:417–22.

    PubMed  Google Scholar 

  18. 18.

    Finkelstein M, Goldstein NE, Horton JR, Eshak D, Lee EJ, Kohli-Seth R. Developing triggers for the surgical intensive care unit for palliative care integration. J Crit Care 2016;35:7–11.

    PubMed  Google Scholar 

  19. 19.

    Glajchen M, Lawson R, Homel P, Desandre P, Todd KH. A rapid two-stage screening protocol for palliative care in the emergency department: a quality improvement initiative. J Pain Symptom Manage 2011;42:657–62.

    PubMed  PubMed Central  Google Scholar 

  20. 20.

    Lupu D, American Academy of H, Palliative Medicine Workforce Task F. Estimate of current hospice and palliative medicine physician workforce shortage. J Pain Symptom Manage 2010;40:899–911.

    PubMed  Google Scholar 

  21. 21.

    Dumanovsky T, Rogers M, Spragens LH, Morrison RS, Meier DE. Impact of Staffing on Access to Palliative Care in U.S. Hospitals. Journal of Palliative Medicine 2015;18:998–9.

    PubMed  PubMed Central  Google Scholar 

  22. 22.

    Bihorac A, Ozrazgat-Baslanti T, Ebadi A, et al. MySurgeryRisk: Development and Validation of a Machine-learning Risk Algorithm for Major Complications and Death After Surgery. Ann Surg 2018.

  23. 23.

    Delahanty RJ, Kaufman D, Jones SS. Development and Evaluation of an Automated Machine Learning Algorithm for In-Hospital Mortality Risk Adjustment Among Critical Care Patients. Crit Care Med 2018;46:e481-e8.

    PubMed  Google Scholar 

  24. 24.

    Sahni N, Simon G, Arora R. Development and Validation of Machine Learning Models for Prediction of 1-Year Mortality Utilizing Electronic Medical Record Data Available at the End of Hospitalization in Multicondition Patients: a Proof-of-Concept Study. J Gen Intern Med 2018;33:921–8.

    PubMed  PubMed Central  Google Scholar 

  25. 25.

    Weissman GE, Hubbard RA, Ungar LH, et al. Inclusion of Unstructured Clinical Text Improves Early Prediction of Death or Prolonged ICU Stay. Crit Care Med 2018;46:1125–32.

    PubMed  PubMed Central  Google Scholar 

  26. 26.

    Umscheid CA, Betesh J, VanZandbergen C, et al. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015;10:26–31.

    PubMed  Google Scholar 

  27. 27.

    Avati A, Jung K, Harman S, Downing L, Ng A, Shah NH. Improving palliative care with deep learning. BMC Med Inform Decis Mak 2018;18:122.

    PubMed  PubMed Central  Google Scholar 

  28. 28.

    Cohen IG, Amarasingham R, Shah A, Xie B, Lo B. The legal and ethical concerns that arise from using complex predictive analytics in health care. Health Aff (Millwood) 2014;33:1139–47.

    Google Scholar 

  29. 29.

    Shah ND, Steyerberg EW, Kent DM. Big Data and Predictive Analytics: Recalibrating Expectations. JAMA 2018.

  30. 30.

    Beam AL, Kohane IS. Big Data and Machine Learning in Health Care. JAMA 2018;319:1317–8.

    PubMed  Google Scholar 

  31. 31.

    Sanders JJ, Curtis JR, Tulsky JA. Achieving Goal-Concordant Care: A Conceptual Model and Approach to Measuring Serious Illness Communication and Its Impact. Journal of Palliative Medicine 2018;21:S17-S27.

    PubMed  Google Scholar 

  32. 32.

    Turnbull AE, Hartog CS. Goal-concordant care in the ICU: a conceptual framework for future research. Intensive Care Med 2017;43:1847–9.

    PubMed  PubMed Central  Google Scholar 

  33. 33.

    Quinn J, Kramer N, McDermott D. Validation of the Social Security Death Index (SSDI): An Important Readily-Available Outcomes Database for Researchers. West J Emerg Med 2008;9:6–8.

    PubMed  PubMed Central  Google Scholar 

  34. 34.

    Zhu BP, Lemeshow S, Hosmer DW, Klar J, Avrunin J, Teres D. Factors affecting the performance of the models in the Mortality Probability Model II system and strategies of customization: a simulation study. Crit Care Med 1996;24:57–63.

    CAS  PubMed  Google Scholar 

  35. 35.

    Sudore RL, Heyland DK, Lum HD, et al. Outcomes That Define Successful Advance Care Planning: A Delphi Panel Consensus. J Pain Symptom Manage 2018;55:245–55 e8.

    PubMed  Google Scholar 

  36. 36.

    Weissman DE, Meier DE. Identifying Patients in Need of a Palliative Care Assessment in the Hospital Setting A Consensus Report from the Center to Advance Palliative Care. Journal of Palliative Medicine 2011;14:17–23.

    PubMed  Google Scholar 

  37. 37.

    Halpern SD, Ubel PA, Asch DA. Harnessing the power of default options to improve health care. N Engl J Med 2007;357:1340–4.

    CAS  PubMed  Google Scholar 

  38. 38.

    Halpern SD. Using Default Options and Other Nudges to Improve Critical Care. Crit Care Med 2018;46:460–4.

    PubMed  PubMed Central  Google Scholar 

  39. 39.

    Emanuel EJ, Ubel PA, Kessler JB, et al. Using Behavioral Economics to Design Physician Incentives That Deliver High-Value Care. Annals of Internal Medicine 2016;164:114−+.

    PubMed  Google Scholar 

  40. 40.

    O’Connor NR, Casarett DJ. Which Patients Need Palliative Care Most? Challenges of Rationing in Medicine's Newest Specialty. Journal of Palliative Medicine 2016;19:696–7.

    PubMed  Google Scholar 

  41. 41.

    Modelling and Simulation: Exploring Dynamic System Behaviour. 2nd ed. London: Springer; 2013.

  42. 42.

    J SSaP. Statsmodels: Econometric and statistical modeling with python. Proceedings of the 9th Python in Science Conference; 2010.

    Google Scholar 

  43. 43.

    Meier DE. Increased access to palliative care and hospice services: opportunities to improve value in health care. Milbank Q 2011;89:343–80.

    PubMed  PubMed Central  Google Scholar 

  44. 44.

    Meier DE, Back AL, Berman A, Block SD, Corrigan JM, Morrison RS. A National Strategy For Palliative Care. Health Aff (Millwood) 2017;36:1265–73.

    Google Scholar 

  45. 45.

    Kavalieratos D, Corbelli J, Zhang D, et al. Association Between Palliative Care and Patient and Caregiver Outcomes: A Systematic Review and Meta-analysis. JAMA 2016;316:2104–14.

    PubMed  PubMed Central  Google Scholar 

  46. 46.

    Smith AK, Thai JN, Bakitas MA, et al. The diverse landscape of palliative care clinics. Journal of Palliative Medicine 2013;16:661–8.

    PubMed  PubMed Central  Google Scholar 

  47. 47.

    Spetz J, Dudley N, Trupin L, Rogers M, Meier DE, Dumanovsky T. Few Hospital Palliative Care Programs Meet National Staffing Recommendations. Health Aff (Millwood) 2016;35:1690–7.

    Google Scholar 

  48. 48.

    Rajkomar A, Hardt M, Howell MD, Corrado G, Chin MH. Ensuring Fairness in Machine Learning to Advance Health Equity. Ann Intern Med 2018;169:866–72.

    PubMed  PubMed Central  Google Scholar 

  49. 49.

    Davis SE, Lasko TA, Chen G, Matheny ME. Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality. AMIA Annu Symp Proc 2017;2017:625–34.

    PubMed  Google Scholar 

Download references

Acknowledgments

The authors are grateful to the palliative care team at the Hospital of the University of Pennsylvania for their support of Palliative Connect.

Funding

This work was funded in part by a career development award from the National Palliative Care Research Center (KRC).

Author information

Affiliations

Authors

Contributions

Study concept and design: KC, CC, MB, SR, MD, NO

Collection, management, analysis, and interpretation of the data: KC, CC, MB, SR, LP, MD, NO

Drafted or critically revised the manuscript for important intellectual content: KC, CC, MB, SR, LP, MD, NO

All authors read and approved the final manuscript.

Corresponding author

Correspondence to Katherine R. Courtright MD, MS.

Ethics declarations

Conflict of Interest

The authors declare that they do not have a conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic Supplementary Material

ESM 1

(DOCX 713 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Courtright, K.R., Chivers, C., Becker, M. et al. Electronic Health Record Mortality Prediction Model for Targeted Palliative Care Among Hospitalized Medical Patients: a Pilot Quasi-experimental Study. J GEN INTERN MED 34, 1841–1847 (2019). https://doi.org/10.1007/s11606-019-05169-2

Download citation

KEY WORDS

  • palliative care
  • prediction model
  • triggers