Predictive models based on electronic health records (EHRs) are used to identify patients at high risk for 30-day hospital readmission. However, these models’ ability to accurately detect who could benefit from inclusion in prevention interventions, also termed “perceived impactibility”, has yet to be realized.
We aimed to explore healthcare providers’ perspectives of patient characteristics associated with decisions about which patients should be referred to readmission prevention programs (RPPs) beyond the EHR preadmission readmission detection model (PREADM).
This cross-sectional study employed a multi-source mixed-method design, combining EHR data with nurses’ and physicians’ self-reported surveys from 15 internal medicine units in three general hospitals in Israel between May 2016 and June 2017, using a mini-Delphi approach.
Nurses and physicians were asked to provide information about patients 65 years or older who were hospitalized at least one night.
We performed a decision-tree analysis to identify characteristics for consideration when deciding whether a patient should be included in an RPP.
We collected 817 questionnaires on 435 patients. PREADM score and RPP inclusion were congruent in 65% of patients, whereas 19% had a high PREADM score but were not referred to an RPP, and 16% had a low-medium PREADM score but were referred to an RPP. The decision-tree analysis identified five patient characteristics that were statistically associated with RPP referral: high PREADM score, eligibility for a nursing home, having a condition not under control, need for social-services support, and need for special equipment at home.
Our study provides empirical evidence for the partial congruence between classifications of a high PREADM score and perceived impactibility. Findings emphasize the need for additional research to understand the extent to which combining EHR data with provider insights leads to better selection of patients for RPP inclusion.
Similar content being viewed by others
Healthcare organizations strive to reduce readmission rates through targeted readmission prevention programs (RPPs), including discharge planning, coupled with post-discharge follow-up, which have reported promising results.1,2,3 Comprehensive care approaches that incorporate patient and caregiver engagement, education, and well-being promotion, as well as complexity management and promotion of continuity of care, have been identified as hallmarks of high-quality transitional care.4 However, RPPs are resource-intensive, and given the need to efficiently allocate resources, identifying which patients would benefit most from RPPs is of growing interest.5 To maximize efficiency, big data from electronic health records (EHRs) are increasingly used to identify patients at high-risk for 30-day hospital readmission for inclusion in RPPs6; however, whether these models can accurately detect patients potentially benefiting from RPP inclusion, also termed “perceived impactibility”, has yet to be realized.7
Patients who are perceived as “impactible”7,8,9 or “care-sensitive”10 have clinical and social characteristics that make their situation “modifiable”, “addressable”, or “intervenable” and are therefore candidates for RPPs. Impactibility indicates characteristics that may be key contributors to patients’ risk and to suitability for inclusion in RPPs, yet may not necessarily be detected by automated prediction scores.9 For example, deteriorating health status increases the chance for readmission, but this does not imply that the deterioration is preventable, and therefore is not “intervenable” in terms of readmission prevention.7 To better align advances in automated prediction models and clinical care, practitioners might use data on individual and social factors to improve high-risk-case identification in terms of prediction accuracy and impactibility.9
To study the appropriate selection of high-risk patients to RPPs, we used a large EHR data warehouse from the largest integrated payer-provider healthcare organization in Israel, Clalit Health Services (CHS). CHS has developed and implemented the Preadmission Readmission Detection Model (PREADM),11 which enables early identification of patients at high risk for 30-day hospital readmission to internal medicine units. The PREADM score guides physicians and nurses in prioritizing patients for inclusion in RPPs, tailored to meet their ongoing care needs. However, anecdotal evidence since the introduction of the program in 2012 has shown that inclusion in RPPs is determined not only by PREADM score but also by healthcare provider perspectives on patients’ clinical, functional, cognitive, and social characteristics. Moreover, some patients who are not detected as high risk for 30-day readmission according to their PREADM score are referred to RPPs based on healthcare providers’ perceptions. These considerations, however, are not uniform, and the specific characteristics that deem patients impactible are unclear.
The objective of this study was to explore healthcare providers’ perspectives about patients’ personal and clinical characteristics associated with the decision whether to refer to an RPP (operationalized as “perceived impactibility”), beyond the classification of the PREADM score.
In Israel, all citizens receive healthcare coverage from one of four integrated payer-provider healthcare organizations. CHS, which serves over half of Israel’s population, operates eight general hospitals and over 1400 primary and specialty clinics nationwide. A broad range of RPPs are performed in CHS hospitals and primary care clinics, including employing transitional care nurses in hospitals who assess the patients’ physical, mental, and social conditions; developing tailored interventions in collaboration with patients and their healthcare providers in the hospital and primary care clinic settings; and conducting routine assessments of patients’ condition throughout hospitalization. This study was performed at three medium-to-large general CHS hospitals.
Study Design and Survey Development
This study employed a cross-sectional evaluation of data from a larger cohort study examining hospitals’ and primary care providers’ perspectives of patients’ social, functional, and cognitive characteristics and subsequent readmission risk (forthcoming). As part of this study, we developed a survey to understand the perspectives of healthcare providers based on risk factors for readmission and characteristics of patients included in RPPs. A three-round mini-Delphi method was used to design and develop the survey for healthcare provider perspectives on patient characteristics that might be associated with RPP referral decision-making. In the mini-Delphi method, field experts discuss a pool of items for potential inclusion in a survey and provide reasons and justifications including them.12,13,14,15
The mini-Delphi method included the following: (1) three in-person workshops of 30 participants (10/group); (2) a face-to-face panel of two staff nurses and five nursing and medical managers reviewing the 30-item survey in a second round; and (3) a review of the 20 highest-agreement items. The final round resulted in the deletion of seven items deemed irrelevant or repetitive. All healthcare-provider participants were experts in RPPs and represented nursing or medical sectors from hospital and primary care settings, including four chief physicians and six head nurses from internal medicine units in various CHS hospitals, six nursing managers at different organizational levels (district or sub-district), eight primary care clinic nursing managers, and six transitional care nurses. This study was approved by the CHS Ethics Committee.
Between May 2016 and June 2017, medical and nursing teams from 15 internal medicine units at three general CHS hospitals were surveyed. Nurses and internal medicine physicians (in charge of direct patient care) were asked for information about patients 65 years or older who were hospitalized in internal medicine wards at least one night. Data were collected by a trained research assistant, twice a week (during weekdays), and then typed and stored in a secure CHS database. Each day about 70–100 patients at each hospital from all internal medicine wards met study-inclusion criteria. Patients who met the inclusion criteria were selected sequentially from the randomized list in each ward until roughly 25% were included in the study. Physicians and nurses both indicated whether patients were or should have been referred to an RPP. Responses on the 13-item survey were collected from nurses only, as they were determined to be the key informants who were most knowledgeable about patient characteristics (according to the focus group discussions).
A patient’s referral (or intended referral) to an RPP was the dependent variable (from the survey) and was coded as 1 (= yes) if either the nurse or physician indicated that the patient was or should be referred to an RPP. Otherwise, the variable was coded 0 (= no) if both physician and nurse indicated a non-RPP referral.
For each patient, the assigned nurse was asked to indicate whether the patient had a specific clinical, functional, cognitive, and/or social characteristic (“yes” = 1) or not (“no” = 0): for example, “Does the patient need support of social services?” or “Does the patient have low language proficiency?” See Appendix 1 for the final survey.
We obtained EHR data from the centralized CHS data warehouse, which stores complete demographic and administrative data for each CHS member, along with complete inpatient records. We extracted data on each patient’s PREADM score, which considers six chronic conditions (congestive heart failure [CHF], chronic renal failure, chronic obstructive pulmonary disease [COPD], malignancy, arrhythmia, disability), previous use of health services (number of hospital admissions, past-year primary care and specialist visits; days from last hospitalization), body mass index, and an indicator of the hospital’s catchment area.11 A high PREADM score was defined as a score of 40 or higher, reflecting the risk status of 55% of the hospitalized population and a readmission rate of 30%.11 Additionally, we extracted separate data on age and sex.
Bivariate analyses, using Pearson’s χ2 tests, were performed to assess differences in characteristics (according to the 13-item survey) between patients who were and were not referred (or were eligible for referral) to an RPP. To ultimately identify characteristics that should be considered in decisions for inclusion in RPPs, we performed a decision-tree analysis, one of the most widely used data-mining algorithms in clinical decision support systems16 (with a conservative cutoff of < 0.1 for inclusion of variables), applying Classification And Regression Tree (CART) using the recursive partitioning and regression trees (rpart) package in R. We also conducted sensitivity analysis by developing a decision-tree model for patients with responses from both nurses’ and physicians’ surveys. All analyses were performed using R version 3.2.2.
We collected 817 questionnaires on 435 patients. Sixty-two patients died during their hospitalization; therefore, the final cohort included 373 patients with 704 healthcare provider surveys (373 from nurses, 331 from physicians who were only rating whether the patient should have been referred to an RPP).
About a quarter (28.7%) of the patients were 8 5+ years old, and 56% were female. The share of those with chronic diseases ranged from 22.8% with COPD to 41.3% with a disability. Just under two thirds (59.8%) had a high PREADM score (i.e. high-risk of readmission). For 212 (56.8%) patients, the physician or nurse indicated or recommended referral to an RPP. For 42 patients (11.3%), information on RPP referral was not provided by physicians. The characteristics of these 42 patients were statistically significantly similar to those of the 331 patients for whom both nurse and physician assessments were available, in terms of demographics and co-morbidity (Appendix 2). Our sample was similar to the overall sample of admitted patients in terms of age and comorbidities.17
Table 1 shows that those referred to an RPP had higher rates of a high PREADM score (72.2% vs. 43.5%, P < 0.001), of CHF (35.4% vs. 23.6%, P = 0.019), and of disability (47.2% vs. 33.5%, P = 0.011). Only two of 13 patient characteristics identified by healthcare providers were not significantly related to the likelihood of an RPP referral (P > 0.1) (i.e. living alone and low language proficiency); therefore, these characteristics were excluded from the decision-tree analysis.
Table 2 demonstrates that having a high PREADM score is only partially congruent with high perceived impactibility. Of the total study population, 41% had a high PREADM score and were also referred to an RPP (high perceived impactibility), 19% had a high PREADM score but were not referred to an RPP (low-medium perceived impactibility), 16% had a low-medium PREADM score but were referred to an RPP (high perceived impactibility), and 24% had a low-medium PREADM score and were not referred to an RPP (low-medium perceived impactibility). Thus, PREADM score and RPP inclusion were congruent in 65% of patients.
The decision-tree analysis identified five patient characteristics that were statistically significantly associated with RPP referral and subdivided all patients into six segments (nodes). The chance of being referred to an RPP varied from 17 to 75% (Fig. 1). The first question in the tree (highest importance), also called the root node, was “Does the patient have a high PREADM score?” In classification trees, positive or “yes” responses branch to the right. If “yes” (69% probability, 223 patients), the second question was “Is the patient eligible to be referred to a nursing home?” Patients who met these criteria were classified as suitable for RPP referral (terminal node 1, 75% probability, 150 patients). For patients with a high PREADM score but who were ineligible for nursing-home referral, the third question was “Are any of the patient’s diseases not under control?” (i.e. the patient’s chronic condition is not stable or above threshold levels). If “yes”, patients were classified as suitable for RPP referral (terminal node 2, 68% probability, 40 patients); if “no”, patients were classified as unsuitable for referral to RPPs (terminal node 3, 42% probability, 33 patients).
Among the 150 patients who had a low-medium PREADM score (question 1), the root node branched left; the next question (highest importance) was “Does the patient need support of social services?” If “yes”, patients were classified as suitable for RPP referral (terminal node 4, 67), with a 58% probability. Finally, for patients having a low-medium PREADM score and no need of social-services support, the third question was “Does the patient need special equipment at home?” If “yes”, patients were classified as suitable for referral to a RPP (terminal node 5, 11), with a 73% probability; if “no”, patients were classified as unsuitable for RPP referral (terminal node 6, 72), with a 17% probability.
Sensitivity analysis for 331 patients with responses from both nurses and physicians about RPP referral yielded similar results.
Our study provides empirical evidence of the partial mismatch between classification of patients at high risk for hospital readmission and perceived impactibility. PREADM score and RPP inclusion were congruent in 65% of patients, whereas 19% had a high PREADM score but were not referred to an RPP and 16% had a low-medium PREADM score but were referred to an RPP. The decision-tree analysis identified five patient characteristics that were statistically significantly related to the reason for patient RPP referral: high PREADM score, eligibility for nursing-home referral, having a condition not under control, need for social-services support, and need for special equipment at home.
The finding that about two thirds of patients were both high risk and perceived as impactible is congruent with other recent findings.8 The remaining third were almost evenly classified between those who were high risk but perceived as not impactible and those who were perceived as impactible but not high risk. This supports the notion that for decisions about RPP referrals, identifying individuals at high risk for readmission is insufficient; it is also important to understand who would benefit from inclusion in RPPs.7,8,9,10 The decision-tree analysis showed that the most important driver of the RPP inclusion decision was the PREADM score, justifying nurses’ and physicians’ initial use of the PREADM tool to select appropriate individuals for RPP inclusion.18, 19
This study also shows that patients with low-medium PREADM scores might need to be referred to an RPP. This finding expands the common focus of readmission-risk identification. While it is common practice to concentrate on identifying high-risk patients, our findings point to the value of considering RPP inclusion according to clinicians’ assessments even where predictive models fail to accurately identify high-risk patients.7 Our results show that individuals with low-medium PREADM scores who needed social-services support or special equipment at home were highly likely to be referred to an RPP. Thus, healthcare provider assessments may be beneficial to incorporate into the current RPP decision-making process to consider inclusion of patients with low-medium PREADM scores.
The current study provides insight into the characteristics that hospital nurses and physicians consider when selecting patients as potential participants in RPPs beyond an automated prediction score. For example, non-nursing-home-eligible patients with controlled disease, despite having a high PREADM score, were unlikely to be referred to an RPP. Such a decision may be due to the low likelihood of those patients to benefit from existing care-management-based approaches to readmission prevention.
The decision-tree supervised learning methodology used in this study is rarely used in the readmission literature but may offer clinicians a practical, user-friendly output for addressing patient perceived impactibility. Decisions for RPP inclusion could be guided by two two-step approaches: first, among high-risk patients, nursing-home eligibility is determined, and then the degree to which the disease is under control is ascertained. Second, among non-high-risk patients, those who need social support are included in RPP, as are those who need special equipment at home.
Ultimately, linked EHR data and information from nurses and physicians can enable structured decision-making for care-management patient selection. Based on these characteristics, the future goal is to develop an on-site, user-friendly decision tool for choosing individuals perceived as impactible to refer to RPPs. Nonetheless, development of on-site user-friendly decision tool requires additional research to first establish the evidence for the decision criteria on which such a tool can lead to standardized RPP selection.
Strengths and Limitations
The main strength of our study was our multi-source and mixed-method design, which combined EHR data with self-reported surveys of nurses and physicians, along with a mini-Delphi approach. Mixed-method studies employing both quantitative and qualitative approaches allow for better understanding of research problems and complex phenomena than either approach alone20 while reducing the threat to construct validity.21 Thus, this combination enabled a comprehensive understanding of readmission prevention. Additionally, the multi-source component of our study reduced same-source bias, thus increasing our confidence in our conclusions.22
An additional strength was the use of the decision-tree model. A decision tree is generally intuitive, and, rather than showing the factors that together compose the likelihood of an outcome (as in regression analysis), it presents the main characteristics that can be followed linearly to guide decisions such as those presented here.16
Despite the current study’s strengths, some limitations should be noted. First, there is no gold standard for the RPP measure. Referral to an RPP aimed at readmission reduction, such as patient discharge counselling and follow-up, or involvement of a transitional care nurse, is commonly performed according to physicians’ or nurses’ recommendation. Accordingly, we used information from physicians or nurses, thus reflecting an all-inclusive approach encompassing both nursing and medical considerations. For the purposes of this study, provider referral for RPP implies perceived impactibility. However, to truly address the issue of impactibility, we need to know whether those selected for intervention are in fact “impactible” according to a set of criterions or standards. Future studies should develop such standards to assess the distinctive contribution of providers’ perceived impactibility.
Second, in the decision-tree analyses, the third branches included relatively small sample sizes, which can increase the likelihood of type II errors skewing the results within these sub-populations. Larger sample studies are needed to increase the power of future studies. Future studies should also consider expanding the target population to those younger than 65. Nonetheless, as much of the focus of current healthcare delivery systems is on better streamlining care for the growing, complex population of older adults, findings from this study are pertinent to other healthcare systems and populations.
Finally, although disentangling the effect of accurate identification and effectiveness of readmission prevention interventions in terms of readmission reduction rates is beyond the scope of this study, it is important to study perceived impactibility in the context of actual readmission reduction and improved clinical outcomes.
Our study provides empirical evidence for the partial congruence between classifications of a high PREADM score and perceived impactibility. The findings emphasize the need for additional research to understand the extent to which combining EHR data with provider insights leads to better selection of patients for inclusion in RPPs.
Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306:1688–98.
Hansen LO, Young RS, Hinami K, et al. Interventions to reduce 30-day rehospitalization: a systematic review. Ann Intern Med. 2011;155:520–28.
Kripalani S, Theobald CN, Anctil B, et al. Reducing hospital readmission rates: current strategies and future directions. Annu Rev Med. 2014;65:471–85.
Naylor MD, Shaid EC, Carpenter D, et al. Components of comprehensive and effective transitional care. J Am Geriatr Soc. 2017;5: 1119–25.
Leppin AL, Gionfriddo MR, Kessler M, et al. Preventing 30-day hospital readmissions: a systematic review and meta-analysis of randomized trials. JAMA Intern Med. 2014;174:1095–107.
Zhou H, Della PR, Roberts P, et al. Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review. BMJ Open. 2016;6.
Lewis GH. “Impactibility models”: identifying the subgroup of high-risk patients most amenable to hospital-avoidance programs. Milbank Q. 2010; 88:240–55.
Jackson C, DuBard A. It’s all about impactability! Optimizing targeting for care management of complex patients. Community Care of North Carolina. 2015.
Steventon A, Billings J. Preventing hospital readmissions: the importance of considering ‘impactibility,’ not just predicted risk. BMJ Qual Saf. 2017;26:782–5.
Freund T, Mahler C, Erler A, et al. Identification of patients likely to benefit from care management programs. Am J Manag Care. 2011;17:345–352.
Shadmi E, Flaks-Manov N, Hoshen M, et al. Predicting 30-day readmissions with preadmission electronic health record data. Med Care. 2015;53:283–89.
Armstrong JS. (2008). Methods to elicit forecasts from groups: Delphi and prediction markets compared. Available at SSRN 1153124.
Khosrow-Pour M. Encyclopedia of information science and technology 2005; IGI Global.
Eubank BH, Mohtadi NG, Lafave MR, et al. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Med Res Methodol. 2016;16:56.
Skulmoski GJ, Hartman FT, Krahn J. The Delphi method for graduate research. J Inf Technol Educ Res. 2007;6:1–21.
Song YY, Ying LU. Decision tree methods: applications for classification and prediction. Shanghai Arch Psychiatry. 2015;27(2):130.
Flaks-Manov N, Topaz M, Hoshen M, et al. Identifying patients at highest-risk: the best timing to apply a readmission predictive model. BMC Med Inform Decis Mak. 2019;19(1):118.
Balicer RD, Shadmi E, Israeli A. Interventions for reducing readmissions–are we barking up the right tree? Isr J Health Policy Res. 2013;2(1):2.
Shadmi E, Zeltzer D, Flaks-Manov N, et al. Reducing readmission rates: evidence from a large intervention in Israel. The Practice of Patient Centered Care: Empowering and Engaging Patients in the Digital Era, vol 244; 2017:109.
Creswell JW, Clark VLP. Designing and conducting mixed methods research. Sage Publications, 2017.
Cook TD, Campbell DT, Shadish W. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin 2002.
Masyn KE, Little T. The Oxford handbook of quantitative methods in psychology. 2013.
The authors would like to thank Sydney Krispin (Clalit Research Institute, Israel) for her assistance in editing and reviewing the manuscript.
This study was funded by the Israel National Institute for Health Policy Research (NIHP) (2015/114).
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Annual Conference of the Israel Society for Medical Information Systems (ILAMI), October 2017, Tel-Aviv, Israel; ICIC18 – 18th International Conference on Integrated Care, May 2018, Utrecht, Netherlands.
Electronic supplementary material
About this article
Cite this article
Flaks-Manov, N., Srulovici, E., Yahalom, R. et al. Preventing Hospital Readmissions: Healthcare Providers’ Perspectives on “Impactibility” Beyond EHR 30-Day Readmission Risk Prediction. J GEN INTERN MED 35, 1484–1489 (2020). https://doi.org/10.1007/s11606-020-05739-9