Background

More than 25 million people experience a hospital inpatient stay at U.S. hospitals during the course of a year [1]. Subsequent transitions from hospital to home and community settings are too often uncoordinated, disruptive, and costly [2,3,4]. Unplanned hospital readmissions represent an important consequence of poorly managed care transitions, resulting in poorer patient experience, elevated risks of hospital-acquired conditions and injury, and more than $25 billion annually in healthcare costs [5]. During the past decade, the U.S. government invested billions of dollars in demonstration programs, quality improvement initiatives, and value-based payment incentives designed to improve care transitions and reduce unplanned hospital readmissions [6, 7]. Specifically, the U.S. Centers for Medicare and Medicaid Services (CMS) supported organized, evidence-based TC programs such as Project RED (Re-Engineering Discharge), [8] Project BOOST (Better Outcomes by Optimizing Safe Transitions, [9] the Care Transitions Intervention (CTI), [10] and the Transitional Care Model (TCM) [11]. Encouraging signs of progress emerged in recent years, including a growing collection of research-tested interventions to improve care transitions, [7, 10, 12, 13] and a downward trend in readmission rates among Medicare beneficiaries [14,15,16]. This progress has been uneven among U.S. hospitals and the extent disputed, [17] with wide variation in readmission rates and persistently elevated rates among low-income patients and other vulnerable subgroups at safety net hospitals [18, 19]. Notably, all the various evidence-based TC programs are characterized by multiple components, share some interventions and have some unique ones.

Uneven progress in improving care transitions may result at least in part from variation in the practices adopted and implemented by hospitals to manage care transitions. A 2010 survey of hospitals participating in a national quality improvement initiative for care transitions found that the average hospital implemented less than half of the 10 recommended practices, such as alerting outpatient physicians about patient discharges within 48 h (39% of hospitals), sending a discharge summary to the patient’s primary physician (23% of hospitals), and following up on test results that returned after patients are discharged (34% of hospitals) [20]. A follow-up study of these same hospitals in 2012 found increases in hospitals’ use of some practices, but no change in use of others [21]. Hospitals often provide multiple care transition interventions to patients, sometimes selecting components from several different models based on staff knowledge, preferences, and expectations about feasibility and effectiveness [22, 23]. Little systematic evidence currently exists about the combinations of care transition practices that hospitals group together and implement. Identifying and understanding which practices hospitals use and which they group to deliver as a bundle aimed at improving care transitions are necessary first steps for evaluating the comparative effectiveness of varying approaches to care transitions experienced by patients as they go through the hospital discharge process.

Project ACHIEVE (Achieving Patient-Centered Care and Optimized Health In Care Transitions by Evaluating the Value of Evidence) [24] was funded by the Patient-Centered Outcomes Research Institute (PCORI®) to convene patients and family caregivers with nationally recognized healthcare researchers to identify which TC services and outcomes matter most to patients and family caregivers [25] and to rigorously evaluate ongoing efforts seeking to improve care transitions. This 5-year multi-component, mixed methods project in the U.S. involved a research team with principal investigators from three academic medical centers, an integrated health system and a large survey company, guided by a Stakeholder Advisory Group and a Scientific Advisory Committee. The research team included experts in health services research, survey methodology, qualitative research, care transitions, health literacy, family caregivers, implementation science, and organizational behavior. This complex initiative involved multiple simultaneous components carried out at different times as shown in Fig. 1. One of the specific aims of ACHIEVE was to determine which evidence-based transitional care (TC) strategies or groups of them most effectively yield patient and caregiver desired outcomes. An essential step in this process was identifying which groups of TC strategies were commonly delivered as a bundle by hospitals. We report here the methodology used to identify and define combinations of transitional care strategies, or groups of activities, implemented among a large and diverse cohort of U.S. short-term acute-care hospitals that aimed to improve an array of patient outcomes. The primary goal of this manuscript is to describe this methodology by which the Project ACHIEVE research team utilized information from multiple components of Project ACHIEVE to determine the groupings of TC strategies with relevant relationship to 30-day readmissions and patient reported outcomes.

Fig. 1
figure 1

Project ACHIEVE Components and Timeline (attached file)

Full size image

Methods/design

Study design

The methodology presented in this manuscript informed one of the specific aims of the larger Project ACHIEVE [24]. Project ACHIEVE is an observational study with the overall aims of identifying what matters most to patients and family caregivers during care transitions, and identifying which combinations of TC strategies yield desired outcomes for diverse groups of patients and family caregivers. ACHIEVE evaluated the natural experiment [26, 27] being conducted in the U.S. regarding the selective implementation of evidence-based transitional care models to accomplish this overall aim. Of note, we define TC Strategies as TC practices or efforts that aim to improve care transitions; such TC strategies are often bundled together by hospitals (e.g., Projects RED or BOOST). Our methodology called for identifying and defining a priori groups of TC strategies commonly delivered as bundles by hospitals. This process included survey of hospital employees actively engaged in care transitions; survey of Medicare beneficiaries discharged from hospitals; use of exploratory factor analysis, latent class analysis, and finite mixture model; and iterative review by research team members, many of whom are nationally recognized experts in care transitions research. The process was informed by ACHIEVE’S prior retrospective analysis of 2.4 million Medicare beneficiaries across 5 years [28]. These a priori defined groups of TC strategies are delineated and defined in Tables 3, 4. Detailed explanation as to how they were derived is provided in the TC Strategy Development section of the Methods section.

Setting

Forty short-term acute care hospitals and two critical access hospitals in the U.S.

Participants

Medicare beneficiaries discharged from hospitals that participated in Project ACHIEVE’s prospective study were eligible to be participants. Using a purposive sampling strategy, hospitals were recruited by the research team and its partners to ensure diversity in: 1) urbanicity; 2) safety-net status; 3) critical access status; 4) member of an integrated delivery system; and at least some participation in 5) alternative payment models, and/or 6) formal evidence-based TC programs (e.g., Project RED, Re-engineering Discharge; Project BOOST, Better Outcomes by Optimizing Safe Transitions) or community-based TC programs (e.g., Centers for Medicare and Medicaid Services’ Community-based Care Transitions Program – CMS CCTP). A hospital recruitment email was sent by ACHIEVE partners (American Hospital Association, America’s Essential Hospitals, Joint Commission Resources) to hospitals known to have a formal TC effort through their participation in one of the following programs: CCTP; Quality Improvement Organization Integrating Care for Populations and Communities (QIO ICPC); and Hospital Engagement Network (HEN), which became the Hospital Innovation Improvement Networks (HIINs) in 2016. The ACHIEVE team followed up with interested hospitals and confirmed participation.

Hospital survey

Representatives from all 42 hospitals agreeing to participate in Project ACHIEVE completed a cross-sectional, web-based survey (See Additional file 1: Appendix 1; it is not under license), to examine their hospital’s implementation of TC efforts from October 2016 through December 2017. Project ACHIEVE hospital coordinators at each site were nominated by hospitals based on their participation in implementing TC strategies in the hospital. These staff were sent web-based survey links through REDCap, [29] a HIPAA-compliant survey platform. The overlapping roles of responding staff included quality improvement (38%), case management/care coordination/social work (52%), patient safety (5%), clinical role—e.g., physician or nurse (14%), other non-clinical (7%), or roles such as research or project management staff (12%). Survey content was generated by the research team and informed by an updated review of available literature, [30] ACHIEVE patient and family caregiver focus groups, [25] input from the project’s Scientific Advisory Council (SAC) and Stakeholder Advisory Group (SAG), ACHIEVE’s Phase 1 hospital site visits, [23] and was revised based on results from ACHIEVE’s retrospective study which also included a hospital survey [28]. Respondents also reported on the range of organized TC programs supported by CMS—e.g., including Project RED, [8] Project BOOST, [31] Coleman’s Care Transitions Initiative, [12] and Naylor’s Transitional Care Model [32]—and indicated whether or not their organization implemented specific TC practices or strategies included in evidence-based care transition models and/or recommended by professional and scientific organizations. Research staff validated survey data through follow up phone calls to staff or through subsequent 1 to 2 day, in-person site visits with various transitional care stakeholders from the hospital and its patient community.

Hospital survey data were linked with corresponding records from the 2017 American Hospital Association (AHA) annual survey and the 2018 CMS Impact File (Table 1) to obtain information on hospital facility characteristics, including number of staffed beds, ownership, and teaching status.

Table 1 Characteristics of Study Hospitals Compared to National Samples
Full size table

Characteristics of the 42 participating hospitals are available in Table 1. Compared to the 2017 nationally representative American Hospital Association Survey, [33] ACHIEVE hospitals were more often in the West (40% vs. 20%), had at least 300 beds (57% vs 19%), were teaching (88% vs. 51%), and nongovernment, non-profit (79% vs. 59%); 13 of the 42 (31%) hospitals were members of one large regional integrated health system (Kaiser).

Patient survey

We developed the post-hospital discharge patient survey to assess: patient perception of receipt of TC strategies received at the hospital or since being home, care experience, patient-reported health outcomes, caregiver effort and stress, and background and demographic questions. We used existing validated instruments when possible, such as items in the NIH PROMIS [34] repository, but most items derived from constructs elicited from study focus groups/interviews [25] and hospital site visits [23]. Survey questions and response options were evaluated through cognitive interviews involving 34 patients and 34 family caregivers. The patient survey contained assessment of 11 TC strategies, 5 of which were also assessed in the above hospital survey. While both sources of data (hospital and patient) were used in the analytic process to define TC strategy groups, in the final prospective analysis, we used only one data source for each strategy. Ultimately, the hospital survey was used for (n = 16 strategies), due to the targeted nature of many TC efforts (e.g., community service referrals). We assessed 6 of the final 22 TC strategies through the patient survey, as they were viewed as being most reliably measured from the patient perspective (e.g., plain language communication). Survey instruments were revised based on these findings from the earlier Project ACHIEVE study components and extensive conversations among the research team, Stakeholder Advisory Group (SAG) and Scientific Advisory Council (SAC). The updated instruments were pilot-tested during a five-month period among five participating hospitals. Based on pilot findings, and again with input from the research team, SAG and SAC, the surveys were further refined. The patient survey is provided in Additional file 2: Appendix 2; it is not under license.

Patient recruitment occurred in hospitals from June 2017 to April 2018. Adult participants were eligible if they were hospitalized on the medical or surgical units at the participating hospitals. For all non-Kaiser hospitals (n = 29), inclusion criteria required patients to have traditional Medicare Fee for Service (FFS); for Kaiser hospitals (n = 13), inclusion criteria included patients with Medicare Advantage or FFS. Exclusion criteria included: 1) in-hospital death, 2) transferred to another acute-care hospital, 3) discharged against medical advice, 4) admission for primary diagnosis of psychiatric condition, rehabilitation, or medical treatment of cancer; 5) current prisoner; or 6) currently under suicide watch. Participating hospital staff identified eligible patients during their hospitalization and obtained HIPAA authorization and contact information for those interested in being contacted to complete a mail or phone survey about their TC experiences in the hospital and at home. Recruited patients were mailed survey packets (in English or Spanish) beginning 51 days post-discharge as per CMS requirements to avoid conflicts with Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) surveys. Non-responders received a reminder after 7 days, a second survey after 24 days, a phone call after 34 days; and were retired after up to 5 call attempts.

From July 2017 to July 2018, 57 % of recruited patients (N = 17,639) responded to the survey. On average, patients completed surveys 75 days post-discharge (Range 52–259). See Fig. 2 for patient survey flow chart. Ultimately, 8080 patients were included in the process of identifying TC strategy groups with a mean age of 72.3 (SD = 10.1). Consistent with the 2017 Medicare population [35], of which 54% were female and 76% were White, a majority of participants were female (53.6%), White (78.9%), and were eligible for Medicare due to their age (78.9%). See Table 2.

Fig. 2
figure 2

Patient Sample Flow Chart (attached file)

Full size image
Table 2 Patient Characteristics, N = 8080
Full size table

TC strategy development

The selection of individual TC strategies to be evaluated in Project ACHIEVE, and their definitions, resulted from a multistep process including: 1) an extensive review of the TC literature, [30] 2) focus groups and interviews conducted with nearly 250 patients and family caregivers, [25] 3) a retrospective analysis of TC strategies at 370 hospitals, [28] 4) input and feedback from the study’s Scientific Advisory Council (SAC) and the Stakeholder Advisory Group (SAG), and 5) through iterative conversations among the research team’s investigators which included experts and leaders in transitional care and research methodology using a modified Delphi process aiming to identify the most important ones that could be reasonably evaluated through surveys of hospitals and patients experiencing the hospital discharge process [36].. See Fig. 3 for a flow chart describing this multi-step, iterative process. This process identified a priori 22 important TC strategies that are variably implemented by hospitals to be evaluated in ACHIEVE’s prospective survey.

Fig. 3
figure 3

TC Strategy Development Process (attached file)

Full size image

Determining whether or not a patient received a TC strategy was accomplished through two approaches (see Tables 3,4 for a list of final TC strategies, their required activities—i.e., definition—and their frequencies of implementation). Recognizing the importance of patients’ perception of receiving some TC strategies, we deemed six to be more reliably measured from the patient perspective (e.g., plain language communication), and therefore used patient survey data to measure them in the final prospective analysis. Thus, for the remaining 16 of the 22 TC strategies, we used hospital-reported data with site visit validation to evaluate implementation based on their reported provision to ‘all’ or ‘most’ patients. For example, many TC strategies implemented by hospitals are based on risk stratification (e.g., referral to community services), and therefore most patients might not report having experienced the strategy though it was applied appropriately.

Table 3 Transitional Care (TC) Strategy Prevalence and Definitionsa
Full size table
Table 4 Transitional Care (TC) Strategy Prevalence and Definitionsd
Full size table

Methodology for identifying groups of TC strategies for evaluation

A multi-step process was undertaken to determine the groups of TC strategies for analysis in the prospective study, initially building upon ACHIEVE’s retrospective study [28]. A key aspect of the retrospective analysis was developing analytic approaches for the Medicare Beneficiary data files that would later be used for the prospective study reported in this manuscript. The retrospective study evaluated hospitals’ reported implementation of 13 TC strategies to determine associated changes in readmission rates and post-discharge ED utilization from 2010 to 2014. Using factor and latent class analyses, this retrospective longitudinal cohort study used survey data from 370 U.S. hospitals to classify hospitals’ TC strategy implementation into the five groups listed in Additional file 3: Appendix 3. From this retrospective study we determined analytic methodologies for identifying groups of TC strategies delivered by hospitals, and identified a group associated with the most reduction in 30-day readmissions after adjusting for the policy effect of the hospital readmission reduction penalty. We applied this methodology to an expanded number of TC strategies in the prospective study based on information gleaned from ACHIEVE focus groups conducted with patients and family caregivers [25].

Statistical analysis

Initially, descriptive statistics were performed to measure the frequency of each TC strategy (i.e., 16 hospital-sourced and 6 patient-sourced strategies). Among the 16 strategies measured through the hospital survey, TC implementation was dichotomously coded as ‘Yes’ when all required activities were provided for ‘all’ or ‘most’ patients, and ‘No’ when they were not. Patient exposure to the hospital-sourced strategies was calculated by aggregating the number of patients discharged from all hospitals reporting ‘yes’ to the strategy. Among the 6 TC strategies measured through the patient survey, patient exposure to strategies was dichotomously coded as ‘Yes’ when all required activities were reported as ‘Yes definitely’ received by the patientFootnote 1 and ‘No’ when they were not. See Tables 3, 4.

Three separate analyses informed the categorization of TC strategies into groups. First, we employed exploratory factor analysis (EFA) which identified the correlation structure among the TC strategies; i.e., patterns in how hospitals naturally implement strategies in bundles or groups, as well as patterns in the groups of TC strategies to which patients are exposed. Specifically, we calculated principal component (PC) estimates in factor analysis separately for the initial 11 dichotomous TC strategy variables from the patient survey (e.g., ‘Yes’ when received and ‘No’ when not; note, only 6 of these strategies were included in final groupings) and for the 16 dichotomous TC strategy variables from the hospital survey (e.g., ‘Yes’ when implemented and ‘No’ when not), each using a polychoric correlation matrix [37] and varimax rotation. The five factors whose eigenvalues exceeded one—indicating unity among variables— identified combinations of TC strategies for further analysis. The resulting factors include overlap, where individual TC strategies may load onto more than one factor, and thus, strategies may be members of more than one TC strategy group.

Next, we conducted latent class analysis (LCA) to identify unmeasured, or latent, class membership among subjects with observed variables. LCA is a nonparametric statistical method used to identify distinct but partially unobservable subgroups (classes) within a population, based on patterns of response across multiple measures [38]. Unlike factor analysis, latent class analysis identifies classes that are mutually exclusive with no overlap in TC strategies, and thus, achieves higher contrast among classes or groups than factor analytic methods. We performed LCA in order to complement our factor analysis results, in which TC strategies could overlap across different groups, thus diminishing the contrast among them. Latent class analyses were applied to TC strategy variables separately according to the source (i.e., hospital survey vs. patient survey).

Next, we used a finite mixture model (FMM) separately for hospital and patient survey data to model the probability of individual patients belonging to each unobserved group that includes different TC strategies and to classify individuals into the groups. Finite mixture modeling was conducted to determine how healthcare utilization was predicted by the selected TC strategies. It was used to further refine the TC groups as it draws inferences about how each group performs.

In addition to the analytic steps described above, the TC strategies and grouping of them were subject to expert review in the context of their conceptual or practical relevance before decisions were made regarding inclusion or exclusion of strategies in the final groups. In addition to ensuring groups with conceptual coherence and analytic significance, the team also sought to decrease overlap among groups and minimize redundancy. See Table 5 for each methodological step and its purpose.

Table 5 Methodological Steps for TC Strategy Group Formation
Full size table

After the proposed groups were defined, random forest (RF) analysis, a supervised machine learning algorithm, [39] was conducted as a supplementary tool to investigate group composition. Random forest analyses work by creating multiple decision trees, or models, to identify the combinations among them that are influential in predicting an outcome. For the purposes of this analysis, individual TC strategies, groups of TC strategies, and key covariates (e.g., comorbidity, integrated health system affiliation, and Hierarchical Condition Category cohortFootnote 2) were entered into a random forest analysis to ascertain their relative importance in predicting 30-day hospital readmission. Importantly, we did not use random forest to edit the TC groups’ configurations, but rather aimed to further consider our hybrid approach to identifying groups of TC strategies by examining whether random forest analysis would suggest an alternate group or method of grouping. Finally, as a form of sensitivity analysis, we ran a boosted trees analysis to determine if results were comparable to the random forest analysis. Boosted trees is a popular machine learning technique that makes predictions by combining decisions from a series of base models (decision trees or other algorithms) for a weighted average instead of simple average over a series of bootstrapped samples [40].

Results

Apart from two TC strategies (Symptom Management and Teach Back for Information and Skills), a majority of hospitals and patients indicated implementation of or exposure to all TC strategies (Tables 3,4). Three TC strategies—Identification of Caregiver, Interdisciplinary Approach, and Standard Protocols—were implemented almost universally. Given their ubiquity, these three TC strategies were not included in any of the groups of TC strategies, as they would not help to differentiate the groups and were presumed fundamentally to be included in each TC strategy group.

Referencing ACHIEVE’s prior retrospective analysis combined with results from the exploratory factor analysis, latent class analysis, finite mixture model analysis, and expert review, our analyses supported the bundling of TC strategies into five groups of TC strategies (See Table 6): 1) Patient Communication and Care Management, a combination of strategies pertaining to facilitating clear and collaborative communication between providers and patients throughout the care transition from hospital to post-discharge including identification of patients’ goals or preferences and contacting patients post-discharge to address issues; 2) Hospital-Based Trust, Plain language, and Coordination, a combination of strategies emphasizing trust and plain language communication between providers and patients, as well as tailored care planning and pre- and post-discharge activities including medication reconciliation and contacting patients post-discharge to address issues; 3) Home-Based Trust, Plain Language, and Coordination, a combination of strategies emphasizing trust and plain language communication between providers and patients at the home as well as use of a specific transition team to coordinate activities including referral to community services and follow-up appointments; 4) Patient/Family Caregiver Assessment and Information Exchange Among Providers, assessment of transitional care needs and patients’ goals/preferences with undertaking interventions for high-risk patients combined with cross-setting information exchange among providers; and 5) Assessment and Teach Back, includes language assessment, use of teach back and contacting patients post-discharge to address issues. Each of these TC strategy groups comprises three to six, non-mutually exclusive TC strategies; i.e., some strategies are included in more than one TC strategy group (See Table 6). The No TC Group includes patients who were not exposed to any of the five TC groups, though they were exposed to other combinations of TC strategies (Range 5 to 15 TC strategies).

Table 6 Five Transitional Care Strategy Groups in Prospective Analysis and Requisite Strategies
Full size table

Factor analyses conducted with patient-derived TC strategies (N = 11) and those conducted with hospital-derived TC strategies (N = 16) each resulted in five factors (Table 7). Slightly different results were obtained from the latent class analysis which found seven classes of strategies from patient survey data and three classes of TC strategies from hospital survey data (Table 8). The findings from the finite.

Table 7 Factor Analysis Results
Full size table
Table 8 Latent Class Analysis Results
Full size table

mixture model results using patient survey data resulted in three groups, with overall variance explained by the groups ranging from 0.012 to 0.140 (Table 9). The finite mixture model conducted with only hospital survey data showed one group (with explained variance of 0.007), including eight TC strategies. The following section details how these findings guided assignment of TC strategies to specific groups (i.e., bundles) for evaluation as interventions received by patients.

Table 9 Finite Mixture Model Results
Full size table

The TC strategy group Patient Communication and Care Management emerged from factor and latent class analyses, in which Post-Discharge Care Consultation, Plain Language Communication at Home, Plain Language Communication at the Hospital, Symptom Management, and Transition Summary for Patients and Family Caregivers loaded onto the same factors or groups. Several of these strategies, or their close approximations, similarly bundled together in the retrospective study’s TC strategy groups (e.g., Symptom Management,Footnote 3 Post-Discharge Care Coordination,Footnote 4 and/or Transition Summary for Patients and Family Caregivers).

The TC strategy group Hospital-Based Trust, Plain Language and Coordination in part emerged from latent class analysis results, which showed that the TC strategy Identify High-Risk Patients and Intervene loaded with Medication Reconciliation and Transition Summary for Patients and Family Caregivers; the latter two TC strategies also grouped together in the retrospective study analysis. In addition, finite mixture modeling using hospital survey data supported the grouping of Post-Discharge Care Consultation, Medication Reconciliation, Plain Language Communication, and Transition Summary for Patients and Family Caregivers together. Based on factor and latent class results showing consistent affinity among Promote Trust at Hospital, Plain Language Communication at Hospital, and Post-Discharge Care Consultation, as well as the conceptual cohesion of these strategies, they were each included in the Hospital-based trust, plain language and coordination group.

The TC strategy group Home-Based Trust, Plain Language and Coordination emerged from latent class analysis (LCA) results and finite mixture model results in which Home visits, Referral to Community Services, Follow-up Appointment showed cohesion. In the LCA, Transition Team also loaded into the same class as these three TC strategies. LCA results also supported the combination of Plain Language Communication at Home and Promote Trust at Home, which conceptually, fit this TC strategy group’s emphasis on home-based post-discharge support activities. The retrospective study similarly identified a TC strategy group including Referral to Community Services and Transition Team.

The TC strategy group Patient/Family Caregiver Assessment and Information Exchange Among Providers emerged from results of both factor and latent class analyses in which the following TC strategies loaded together: Timely Exchange of Critical Patient Information Among Providers, Patient Goal/Preference Assessment, Identify High-Risk Patients and Intervene and Patient and Family Caregiver TC Needs Assessment. Two of these strategies (Timely Exchange of Critical Patient Information and Patient and Family Caregiver TC Needs Assessment) were similarly grouped in the retrospective study.

For the TC strategy group, Assessment and Teach Back, both factor and latent class analyses supported the grouping of Teach Back for Information and Skills, Language Assessment and Post-Discharge Care Consultation; evidence for grouping the latter two strategies also emerged from finite mixture model results. While none of these TC strategies were grouped in the retrospective study, analytically and conceptually these strategies showed cohesion pertaining to strategies employed by hospitals to increase adherence to patient care plans.

These series of analyses—complemented by expert review to ensure 1) consistency with prior ACHIEVE research (e.g., retrospective TC groups and patient and family caregiver focus group findings [25]), 2) consideration of service setting (e.g., hospital-based, bridging from hospital to home, or home-based), 3) practical relevance (e.g., sample size in each group must enable group comparison); and 3) minimal redundancy and overlap among groups— resulted in the groups as outlined in Table 6.

Additional analytic confirmation

Random forest analyses were conducted as a confirmatory process to ascertain if machine learning methods would suggest additional important factors related to hospital readmissions by providing information about the relative “importance” of each variable in predicting 30-day readmissions (Table 10). Variables included in the random forest analysis were individual transitional care strategies, covariates (e.g., comorbidity, integrated health system affiliation), and the five newly defined groups of TC strategies. Results showed that, relative to other covariates and TC strategies, patient comorbidities and hospital cohort (e.g., medical, surgical, neurology) were most influential in predicting readmissions, accounting for a total of >200G2 and ~ 30% relative importance. Regarding the individual TC strategies and groups of TC strategies, those assessed through patient survey data (e.g., Helpful Healthcare Contact, Promote Trust, Plain Language Communication) were more important than the TC strategies assessed through the hospital survey data (e.g., Follow-up appointment, Teach Back for Information and Skills), with G2 ranging from 54 to 16, and relative importance from 7 to 2%. Importantly, random forest results did not suggest alternate groups nor an alternate approach to defining or classifying groups of TC strategies, with the key TC strategies from each group landing on the top of the importance tree. For example, Helpful Healthcare Contact has 7.2% relative importance, and Plain Language Communication at Home has 5.1% relative importance. As most information was used by splitting these two key TC strategies, the group including these two key TC strategies---Patient Communication and Care Management has only 2.1% relative importance. Notably, results were similar when we ran a boosted trees analysis, with patient comorbidity, Hierarchical Condition Category (HCC) cohort, and several patient-reported TC strategies among the most influential in the model for predicting 30-day hospital readmissions.

Table 10 Random Forest Analysis Results
Full size table

Discussion

Project ACHIEVE aimed to provide practical, actionable guidance to hospitals searching for information about how to strategically invest in transitional care strategies in order to optimize patient outcomes. Prior systematic reviews found no individual strategy consistently associated with reduced readmissions, suggesting the need to evaluate TC strategies as implemented in groups or bundles [41]. Although numerous evidence-based, multifaceted transitional care models exist with evidence supporting their ability to reduce readmissions, [8, 9, 12, 32] adaptation of such models by implementation sites is common [23, 31, 42]. Little evidence exists regarding the question ‘Which transitional care strategies in which combination are most effective at improving patient outcomes in diverse populations and settings’? ACHIEVE capitalized on the natural experiment [26, 27] being conducted in the U.S. regarding the selective implementation of evidence-based transitional care models to answer this question. The present analysis describes the methodology for identifying the groups of TC strategies most commonly implemented by hospitals nationwide for subsequent evaluation of their comparative effectiveness. Our process resulted in five overlapping groups of TC strategies reflecting analytic and conceptual cohesion.

The five groups of TC strategies reported here that emerged from this process are being evaluated through the study’s prospective cohort analysis, the details of which will be reported elsewhere, to determine their detailed relationship with a broad spectrum of outcomes [43]. We believe the analyses that yielded these groups of TC strategies (See Table 5) provide methodological guidance for others seeking to conceptualize and evaluate complex interventions. We believe our findings provide a roadmap for the multistep, hybrid application of sophisticated analytic and conceptual techniques to categorize and define the ways in which TC strategies are naturally clustered so that these clusters (i.e., TC strategy groups) may then be evaluated.

Results of the sophisticated analyses conducted by the ACHIEVE research team have important practice implications. For example, one of the contextual analyses conducted, random forest analysis, demonstrated that patients’ underlying health conditions (e.g., comorbidities and hierarchical condition classification) were among the most important variables influencing readmissions, consistent with prior research [44,45,46,47]. This finding highlights the difficulties faced by hospitals and care providers attempting to improve transitional care. The next most important variables were implementation of TC strategies measured through survey of patients. This finding suggests that patient reporting of TC experience may be a more reliable indicator of TC implementation than hospital reporting for certain strategies—e.g., if caring and trust were fostered—given that implementation may vary by provider, care team, or circumstance. Patient survey data may more accurately reflect patients’ perception and experience more accurately reflecting impact of hospitals’ efforts. Another possible reason that patient-reported TC strategies exerted more influence over hospital readmissions may simply be that these strategies were meaningful components of discharge planning. Trust in one’s health care provider has emerged repeatedly in the literature as being associated with positive patient health outcomes [48], better care plan adherence [49], and patient satisfaction [50]. In fact, in focus groups and individual interviews conducted with 248 patients and family caregivers [25], participants voiced their strong desires to feel 1) cared for and about (Promote Trust), 2) prepared to implement the care plan (Plain Language Communication, Symptom Management), and 3) accountability on behalf of healthcare professionals regarding who to contact (Helpful Healthcare Contact).

Although the TC strategies in our model that were measured by patient survey were more important for predicting readmissions, strong rationale remains for measuring implementation of some TC strategies through hospital assessment. First, some TC strategies known to impact the quality of care do not have a patient-facing component, rendering it difficult for a patient to report on its appropriate application. For example, patients may not be aware if Medication Reconciliation was conducted by a designated pharmacist who contacted outpatient physicians or pharmacies, or whether their care team participates in Interdisciplinary Rounds. Second, some TC strategies are selectively applied based on risk stratification and are not universally appropriate for all patients. As most patients would screen negative for such interventions, their self-report of implementation may skew negative, even if the hospital appropriately implemented the strategy. Ultimately, we believe our hybrid approach of measuring implementation of certain TC strategies (e.g., Medication Reconciliation, Identify High Risk Patients and Intervene) from the hospital perspective, and measuring others (e.g., Plain Language Communication, Trust) from the patient perspective provides a more comprehensive assessment of implementation of TC strategies.

Strengths and limitations

The hybrid approach outlined in Project ACHIEVE’s methodology—combining both analytic and conceptual methods— for defining groups of TC strategies for analysis represents a novel approach. The strengths of comprehensive, sophisticated analytic methods to identify patterns in how hospitals implement and patients experience transitional care efforts, which is important due to the wide variation in fidelity that hospitals report even when implementing evidence-based models for transitional care. The inclusion of expert interpretation and review to complement our analytic methods strengthens our approach, providing clinical experience and extensive research insight; an approach recommended when evidence informing clinical practice guidelines is inadequate [51]. By having clinical practitioners review the findings in context of their conceptual and practical relevance, we were able to ensure a degree of conceptual cohesion to enhance a TC strategy group’s relevance to real-world. Practice. By having expert researchers in transitional care also review the findings and inform groupings of TC strategies, we ensured there was sufficient distinction among the groups so that their comparative effectiveness could be evaluated.

While our approach of collecting data from both hospitals and patients is a strength, each source has its limitations. First, although most hospitals’ survey dataFootnote 5 were validated by a one to two-day site visit in which ACHIEVE researchers met with TC stakeholders across the hospital, the potential for self-report bias or incomplete implementation of certain strategies across all units or providers remains. Similarly, although our collection of some TC strategy data from the patient perspective is a strength, it has limitations. First, it is possible that patient outcomes following a care transition (e.g., readmission to hospital or emergency department) may have influenced patients’ perceptions of the transitional care strategies they received. The unfortunate CMS-imposed delay of patient survey administration to at least 51 days after hospital discharge may have exacerbated the potential for recall bias as well as selection bias. Sicker patients may have died or become less capable of participating. Further, patients with cognitive impairment were necessarily excluded due to their lack of consenting capacity, so our results may not apply as directly to that population. While our response rate of 57% for the patient survey compares favorably with similarly designed surveys—e.g., H-CAHPS surveys of patient experience typically average approximately 30% response rate [52]--the possibility remains that participants were systematically different than non-participants in ways that we were unable to measure. Because patients were initially recruited by hospitals, and not formally consented until contacted for the survey, we were unable to collect demographic information about those who did not consent. Therefore, we are unable to compare characteristics of participants with non-participants.

Finally, although our list of 22 transitional care strategies is extensive, and was rigorously developed through review of evidence-based TC models, review by an active and engaged Stakeholder Advisory Council and Scientific Advisory Council, and extensive structured discussions with the Project ACHIEVE research team [53], it is not exhaustive of all TC strategies used by hospitals nationwide, or even of the hospitals in our study. Some important TC practices used by hospitals may not have been included on our survey and therefore were likely omitted from study. Nevertheless, our findings provide practical guidance regarding a method of identifying patterns of patient exposure to bundles of TC strategies as a means of evaluating complex interventions.

Conclusions

Our findings provide support that a data-driven approach using sophisticated statistical tools complemented by content experts can help identify underlying patterns of hospitals’ TC implementation efforts that correspond with better outcomes. Using such tools, this study identified five groups of TC strategies that have potential to improve patient outcomes.