Background

Translating complex multi-component evidence-based interventions from study conditions and protocols to actual practice necessarily requires minor to major adaptation of intervention content, format, personnel roles and processes. Adaptations may be desired or necessary to customize to local circumstances or to account for new technology or temporal changes in disease epidemiology or clinical practice [1, 2]. Traditionally, it has been thought that variance from an evidence-based protocol is accompanied with at least some diminution of effectiveness. This effect has been termed “program drift” (i.e., decreased benefit arising from deviations from study protocols) and “voltage drop” (i.e., loss of benefit when interventions are applied in more heterogeneous patient populations and settings as they move from efficacy to effectiveness and into actual practice) [3].

More recently, the inevitability of diminished benefits due to program drift and voltage drop has been questioned. While fidelity to a study population, setting, and protocol may be critical for a biological or a clinical intervention to maintain the studied benefits, this may not be the case for health system interventions or more complex multi-component interventions. Adaptations may be made as the intervention is applied in more heterogenous patient populations or practice settings, thereby potentiating voltage drop. Other adaptations may be made to the intervention protocol to improve fit with the practice setting and the environment, thereby potentiating program drift. Other adaptations may address technological or medical advances that were not available or formally manualized at the time of the evidence-based study. In theory, these adaptations may have neutral effects. Some adaptations may enhance the effectiveness of the original intervention3, thereby neutralizing any diminished effectiveness from program drift or voltage drop or perhaps even increasing the net benefit beyond that observed in evidence-based studies. Other adaptations may inadvertently diminish benefits. Indeed, some have called for the systematic study of adaptations to evidence-based interventions and a formalized structure for updating the evidence base to account for adaptation [4].

To improve our understanding of the effect of adaptations, we took advantage of an implementation of an evidence-based hospital admission avoidance Hospital at Home (HaH) intervention [5]. This implementation necessitated extensive adaptation of studied procedures to local setting and culture and evolving medical practice. In 2014, The Mount Sinai Health System, a seven-hospital system in New York City, began implementing HaH with 30-day post-acute care follow-up of patients. For select patients with specific diagnoses (e.g., pneumonia) who would otherwise be admitted, HaH services (e.g., intravenous antibiotics, fluids, oxygen, etc.) and daily clinician visits are provided at home along with durable medical equipment, phlebotomy, and home x-ray as needed. We examined whether HaH associations with outcomes changed over time as adaptations were put into place.

Funded as an innovation award from the Center for Medicare and Medicaid Innovation, this implementation of HaH was instigated with the goal of eventual sustainment in the local health system and scaling and dissemination nationally. The phased plan for implementation within the local health system was consistent with the concepts outlined in the Dynamic Sustainability Framework (DSF) [3]. The DSF posits that interventions should be implemented with consideration of fit with practice settings and the external ecological system of competitors, regulation, market forces, and population needs. The framework also puts forth that these elements are not static and will change with time. Our phased implementation anticipated a piloting phase followed by initial implementation in one hospital emergency department with phased expansion over several quarters to other health system settings. We anticipated the need to adapt HaH to each hospital’s unique culture, medical staff, electronic record system, community, and labor practices. Similarly, we anticipated that HaH might need to be adapted to changing policy and regulations, different payers, and a healthcare market in dynamic flux due, in part, to the implementation of the Affordable Care Act.

Methods

Patients and settings

Patients were enrolled in HaH starting in November 2014 for 33 months through August 2017. An initial 6-month run-in pilot period was followed by nine quarters of implementation activity.

Details of HaH patient engagement procedures have been reported elsewhere [6]. Briefly, potentially eligible patients were identified in the emergency departments of Mount Sinai Health System hospitals, or by referral from physicians in outpatient clinical practices or a home-based primary care practice.

Patients were eligible for HaH if they were ≥ 18 years of age, required inpatient admission, and had fee-for-service Medicare or coverage from a single private insurer that contracted with Mount Sinai for HaH services. Patients were excluded if they were clinically unstable (e.g., had very low blood pressure), required cardiac monitoring or intensive care, lived in an unsafe home environment, or resided outside the specified catchment area.

Hospital at Home, Core components, and adaptations and their coding

The HaH program retained many of the core components of original evidence-based studies of HaH (see Table 1). These included a) targeting to participants who needed to be hospitalized; b) delivering hospital-level services at home instead of the hospital; c) daily registered nurse visits to the home; d) clinician (physician or nurse practitioner) home visits; and e) 24/7availability to patients and family members.

Table 1 Core Programmatic Elements of HaH from Studies Treating Multiple Diagnoses
Full size table

Adaptations to the original HaH model were made at inception and throughout the 33-month implementation to enhance enrollment, to add new hospital sites, to improve workflow, to enhance patient care, to extend hours of intake, and to take advantage of new opportunities (availability of new technology or services such as community paramedicine). As described later in the list of adaptations made to the HaH program, HaH service lines evolved to include palliative care at home, observation unit at home, and rehabilitation at home (see full list in Table 2). We used an established coding system to categorize who made the adaptations, what was modified, for whom the adaptations were made, and the nature of the adaptations [1].

Table 2 Adaptations of HaH Model
Full size table

Data collection and measures

Patient-level data were collected for the purposes of performance monitoring and final evaluation of program performance. Interviews were conducted at the bedside in the emergency department (ED) (see Additional file 1 for instrument guide), with follow-up interviews conducted at 2 and 4 weeks after admission by phone (see Additional files 2 and 3 for instrument guide). We collected information on the following patient outcomes: length of stay (LOS), 30-day readmissions and emergency department (ED) visits, escalations (i.e., needing to suspend a HaH episode in the acute rather than post-acute phase to transfer the patient to the hospital), and patient ratings of care measured with the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey. HCAHPS was scored per guidelines from the Centers for Medicare and Medicaid Services [13], specifically, the proportion of individuals who gave a top-box rating for the measure (highest possible rating), adjusted for age, education, interview language (Spanish or English), and general health. Interviewers also documented patient demographics, performance in 5 activities of daily living (ADL), and general health (rated poor to excellent). Functional impairment was defined as needing some help or unable to perform one or more of 12 ADLs. Data on all outcomes, with the exception of patient ratings, which were less available to us in a timely fashion, were monitored monthly with feedback to program staff for review and continuous quality improvement.

Analysis

Because the outcomes collected were most relevant to hospital stays, we report outcomes for patients admitted to the program where the intervention was used to substitute for a hospital stay. That is, we excluded patients receiving rehabilitation at home (n = 264). We also exclude patients who received observation services for a single overnight period (n = 41). If hospital services were extended overnight past a second midnight for patients initially admitted to observation services, we considered the case to have been converted into a regular HaH stay and include them in our analyses; this simulates what would have happened if a patient had been admitted to a regular hospital observation unit and required services past the second midnight. Patients receiving palliative care services at home were receiving HaH for what substitutes as a hospital stay and are included as well. After exclusions, our sample includes data on 295 patients.

We present outcomes by quarter of HaH patient admission. We used logistic or linear regression (in the case of LOS) with an independent variable included for the numerical quarter of enrollment after an initial 6-month pilot phase. We estimated the average marginal effect for number of quarters since the program began (marginal effect per each additional quarter) for LOS. Odds ratios were estimated for the other models. The models controlled for season of HaH admission, patient age, sex, race, ethnicity, education, insurance type, functional impairment, and self-rated health.

More than 10% of data were missing for race and ethnicity (n = 64), physical function (n = 115), and general health (n = 84). To maximize the number of patients in the models, we used multiple imputation, modeling the probability of missing data for each variable on age, sex, race and ethnicity, education, insurance type, pre-acute physical function, and general health. Analyses were performed with SAS version 9.3 (SAS Institute, Cary, NC).

Because the data in these analyses were used for internal program evaluation and reporting to Medicare, their collection was exempt from Mount Sinai Institutional Review Board (IRB) review and patient consent was not required. We requested and received approval from our IRB to conduct a retrospective analysis of these data.

Results

Over the 33 months of implementation, a series of program adaptations were implemented (see Table 2), including a) adding a 30-day post-acute transition component to the HaH model (at inception); b) expanding target diagnoses and modifying exclusions from those originally studied (Quarter 1) due to changes in medical practice (e.g., removed HIV exclusion from decade-old protocol of original HaH studies); c) outside contracting for certain services (e.g., infusion) rather than providing the service directly with program staff (Quarter 2); d) implementing HaH variations due to change in medical practice (e.g., palliative care unit at home [Quarter 1], observation unit services at home [Quarter 3]) or opportunity (e.g., rehabilitation at home services [Quarter 3]); e) adapting intake procedures (e.g., holding patients overnight in the emergency room [Quarter 4]) for patients entering the program late at night when delivering supplies to the home to initiate a HaH episode was less available; f) launching telehealth visits (Quarter 4); and g) changing staffing and staff roles (Quarters 2, 6, 8). Other more minor adaptations were made throughout the implementation period.

All adaptations were initiated by the program team (see Table 3). Modifications were most commonly made in format (or how the intervention was delivered) and in personnel and roles. Modifications were made at the level of the cohort (e.g., for patients who would otherwise have been admitted to an observation unit in the hospital), patient population (e.g., for patients with new diagnoses or with palliative care needs that would not have been eligible for the original HaH intervention) or organization (e.g., staffing modifications that affected the entire program regardless of hospital site). The nature of adaptations was most commonly to add or to substitute new program elements. In three instances, these adaptations were sufficiently extensive for the cohort or population that it involved integrating the HaH approach to additional patient populations (palliative care patients) or to other clinical constructs (e.g., observation unit services or postacute rehabilitation).

Table 3 Coding of Adaptations of HaH
Full size table

HaH services substituting for a hospital stay were received by 295 patients. The average LOS was 3.2 days. Of our sample, 12.2% had escalations, 8.6% had readmissions within 30 days, and 5.8% had ED visits within 30 days. The overall rating for hospital care was rated at the highest level by 68.8% of patients.

Graphs of these outcomes, adjusted for patient characteristics and season, over the nine quarters of implementation are provided in Figs. 1, 2, 3 and 4. A median of 33 patients (range 11–44) received services each quarter. The graphs show occasional spikes in events such as readmissions. Confidence intervals and standard errors were moderately large at any point in time; however they indicate that changes, if any, were small over time. The graphs indicate the quarter of initiation for the adaptations listed in Tables 2 and 3.

Fig. 1
figure 1

Mean Length of Stay (LOS) in days, Change over quarters SE 0.02, p = 0.75

Full size image
Fig. 2
figure 2

Hospital Rating (High) HCAHPS. Change over quarters OR 0.99, p = 0.66

Full size image
Fig. 3
figure 3

Percent Hospital Readmissions, Change over quarters, OR 1.00, p = 0.99

Full size image
Fig. 4
figure 4

Percent ED Visits, Change over quarters, OR 0.93, p = 0.09

Full size image

Regression models controlling for patient characteristics and season of admission are available in Additional file 4. Older age was associated with higher ED revisits and higher ratings of care. African American patients had shorter LOS and higher 30-day ED revisits and readmissions. Having Medicaid was associated with higher ED revisits and lower ratings of care. Poor self-rated health was associated with longer LOS, greater escalations, and higher ratings of care.

In the regressions, we did not observe a relationship between quarter from program inception and LOS (− 0.007 days/quarter; SE 0.02, p = 0.75). The odds ratios (OR) per quarter from program inception was mildly increased for escalations (OR 1.09, 95% CI 1.01 to 1.18, p = 0.03), but was not increased or decreased for 30 day ED visit (OR 0.93, 95% CI 0.86 to 1.01, p = 0.09), 30-day readmission (OR 1.00, 95% CI 0.93 to 1.08, p = 0.99), and the patient providing the highest rating for overall hospital care (OR 0.99, 95% CI 0.93 to 1.05, p = 0.66).

Discussion

We made adaptations to an evidence-based model of HaH at program inception and throughout a 33-month implementation. Our experience reinforces concepts from the DSF. Many adaptations were made to improve HaH operations. Others were made to respond to patient needs and referring provider requests such as for palliative care patients who needed acute hospital-level care. Some adaptations were made to improve scalability such as extending intake hours into nights and weekends. Other adaptations were made because new opportunities became available to enhance the patient experience such as the initiation of video telehealth and community paramedicine visits. These adaptations would have been difficult to include at program inception or even to script into phased introductions as the availability of the new opportunities or need for adaptation was unknown at that time. In some cases, adaptations were necessary to adapt to differing practice contexts at different hospitals. Expansion to other hospitals usually required new intake procedures to adjust to local culture, differing payer mix and social determinants of health in the surrounding community, and even different unions. The intervention also had to be adapted to payment and market forces affecting our hospitals (e.g., unanticipated creation and expansion of observation services during the period of implementation) and vendors (e.g., vendor budget issues, including vendor bankruptcy). We believe that the need and opportunity for dynamic adaptation occurs to varying degrees in the implementation of all new programs and is not limited to the HaH implementation experience we chart in this paper [2].

We modified the protocols used in our HaH program from the existing research evidence base described in the literature. For example, the addition of new diagnoses, alteration of clinical exclusions, and the addition of 30-day postacute follow up were changes from studied protocols. Despite this, for selected findings of LOS and patient satisfaction, our outcomes were similar to those reported in the original studies in Table 1. Further, in a previous report [6], these outcomes over the period of implementation compared favorably to those of a comparison group of similar patients who were hospitalized in regular inpatient units. Thus, our implementation of an evidence-based intervention with its associated adaptations was accomplished with no obvious “program drift.”

Additionally, we made adaptations in which we modified the HaH approach to changing context each quarter. For example, we added palliative care and other programs, as well as telehealth and community paramedicine visits. However, we did not observe significant changes in effectiveness across a variety of measures. LOS, 30-day readmissions, ED visits, and patient ratings of care did not significantly change over time. Tracking and reviewing these outcomes on a monthly basis as part of a continuous improvement activity may have been important in maintaining these effects through the implementation period. In the case of escalations, we observed a trend toward higher rates in later quarters. This was expected as our clinicians purposely started the program with less complex patients and enrolled increasingly complex patients as the team gained experience. The escalation rate was actually closely monitored throughout implementation, the program team was conscious of the trend, and cases of escalation were reviewed by the team on an ongoing basis in the interest of continuous improvement. Thus, we believe that the implementation of serial adaptations may have also avoided the phenomenon of “voltage drop.”

Our study was limited by sample size by quarter. There are also occasional quarters where readmissions or other outcomes appear to increase. However, these occurrences are transient and the trend over the nine quarters is one of no change. A further limitation is that individual adaptations could be hypothesized to potentially enhance or diminish net benefits or program efficiency. For example, the community paramedicine intervention was intended to improve response time, enhance ability to adjust treatments in the home, and to reduce trips to the ED. One could hypothesize that this might enhance effectiveness. On the other hand, observation services at home were a departure from the evidence-based model, and one could hypothesize that these patients might be less likely to benefit. Because adaptations were made as the need or opportunity arose, it is not possible for us to isolate and examine the effect of any individual adaptation in this analysis.

Conclusion

The implementation of evidence-based programs necessarily requires adaptation. We made significant adaptations to HaH at inception and then serial adaptations over the course of implementation. Over the course of the implementation, many of these important outcomes were tracked and fed back to the program leadership leading to further adaptations. Our findings may indicate that adaptations to evidence-based programs may avoid diminished benefits due to potential ‘program drift’ or ‘voltage drop’.