INTRODUCTION

Emergency Departments (EDs) care for society’s most vulnerable older adults who often present with exacerbations of chronic disease at the end of life, yet the clinical paradigm continues to focus on the treatment of acute illness and injury. Palliative care interventions in the ED capture high-risk patients at a time of crisis and can dramatically improve patient-centered outcomes.1,2 As a substantial number of older adults visit the ED during times of crisis, emergency providers set the trajectory for their care, including whether an older adult is hospitalized and to which setting. Emergency medicine providers can play an integral role in transforming care for older adults through evidence-based models of care delivery that emphasize tradeoffs between potential benefits and harms.

Until recently, little emphasis has been placed on delivering goal-concordant emergency care although palliative care in the ED improves quality of life and reduces healthcare use.1 The Primary Palliative Care for Emergency Medicine (PRIM-ER) Quality Improvement (QI) trial aims to reduce admissions to inpatient settings, decrease inpatient length of stay and ICU admissions, and increase 6-month survival by improving emergency providers’ primary palliative care skills and knowledge in an effort to align care plans with patient goals impacting clinical outcomes.

Developed to broaden the focus of research from efficacy of controlled interventions to the effectiveness and generalizability of interventions across varying settings and populations,3 the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework is particularly useful for evaluating pragmatic trials.4 In trials such as PRIM-ER, RE-AIM can assist in transforming research findings into real-world settings impacting everyday clinical practice. One gap identified in the RE-AIM literature is challenges related to staff participation in clinical interventions.4 For this study, we aim to understand in greater detail why and how the PRIM-ER QI project was successful in the first two pilot sites using quantitative and qualitative methods. Using these findings, we will disseminate the lessons learned to each of the 33 additional clinical EDs participating in the PRIM-ER QI clinical trial over the next 2 years of the implementation phase. This will address a need to provide scientific evidence to local QI sites to guide implementation.5

METHODS

Study and Intervention Design

PRIM-ER is a pragmatic, cluster-randomized stepped wedge trial that focuses on improving care of older adults with serious, life-limiting illness by changing EM provider norms on primary palliative care and day-to-day clinical practice via the implementation of QI intervention components. The QI intervention consists of four components: (1) delivery of provider (physicians and physician assistants (PAs)) education through a 4-h in-person simulation based communication skills training in end-of-life care, called EM Talk,6 (2) delivery of a one-hour online didactic nurse education course through End-of-Life Nursing Education Consortium (ELNEC),7 (3) implementation of a clinical decision support (CDS) tool, and (4) implementation of an audit and feedback system. Each site is expected to complete all four intervention components during their 3-week intervention period.8 For the two education components targeting providers and nurses, the goal is to train 75% of the EM providers.

The PRIM-ER protocol has been previously described.8 To test the effectiveness of the intervention, it will be implemented at 35 clinical EDs across the USA over a 2-year period. For this study, we conducted a secondary research analysis of the PRIM-ER trial using quantitative and qualitative data from the first two pilot sites in order to determine the reach and effectiveness of the intervention using the RE-AIM framework.

Study Population

This study took place at two large tertiary academic urban city hospitals located in the New York-Northern New Jersey Metropolitan Statistical area9 between August and September 2018.

The inclusion criteria for participation in the QI intervention included (1) working as a full-time employee (FTE) in the ED at either of the two pilot sites enrolled and (2) having the role of either an emergency physician, PAs, nurse practitioner (NP), or nurse. All residents, or postgraduate physicians in training, other members of the care team (imaging technologists, phlebotomists, etc.), and part-time and/or per-diem EM physicians, PAs, and nurses were all excluded. NPs were not employed by either ED, thus were excluded.

Table 1 outlines each site’s characteristics, in an effort to better understand the context, similarities, and differences of the health systems where the pilot QI intervention occurred. Both PRIM-ER pilot sites were academic medical centers, located in similar urban regions, each with over 80,000 ED visits per year. Site 1 is a public safety net hospital. Site 2 is a private quaternary care hospital. Site 1 had 28 FTE emergency physician but no PAs and 89 FTE EM nurses. Site 2 had 21 FTE emergency physicians, 38 FTE EM PAs, and 108 FTE EM nurses, as described in Table 1. Site 2 had a larger provider pool eligible for training, as EM clinical leadership decided to include PAs in the training series. PAs at Site 2 work under an attending physician and care for high acuity patients, thus leadership decided that training the FTE EM PA staff on goals-of-care communication skills was clinically relevant. Physicians work alongside nurses across both sites. Providers at both sites were under the leadership of the department chair but independent management structures exist at both sites for different clinician types (e.g., nurse managers).

Table 1 Site Characteristics
Full size table

During each site’s pre-intervention period, they were required to convene a PRIM-ER QI leadership team, who would be responsible for garnering buy-in and support from their EM staff in the implementation of the QI intervention components and communicating with the NYU research team. Each site’s team had a site principal investigator (PI), physician champion, nurse champion, and informatics analyst.

Data Collection

To assess components of the RE-AIM framework, both quantitative and qualitative methods were used.

Quantitative Data Sources

Site characteristics were provided by each Site PI. All participants meeting inclusion criteria were asked to anonymously complete a validated baseline survey prior to the QI intervention (either online or in-person) assessing their attitudes and knowledge towards end-of-life care, including self-rated knowledge, support for hospice practice and philosophy, and views on physician-patient communication.10 Participant characteristic data was also collected through this survey. Online and in-person responses were managed using REDCap, a secure electronic data capture platform hosted at NYU School of Medicine.11,12 Intervention completion data (Reach in RE-AIM) was collected through completed attendance sheets from both the provider (physician and PA) in-person EM Talk trainings and nurse ELNEC online training. All attendance data was verified by each of the site PIs to ensure fidelity.

Qualitative Data Sources

For the key informant qualitative interviews, a purposive sampling approach was used to recruit interviewees to obtain diversity regarding their role and experiences working on implementing the pilot intervention. In order to capture a broad perspective of all components of the intervention, a member of the PRIM-ER study team reached out to each identified champion via institutional email, with up to two email follow-ups for non-responders. Eleven participants were invited to participate in an interview. Unfortunately, two participants left their respective institution (site 1) and were unable to be contacted for an interview, and were excluded immediately from the outreach participant list. The final sample included six participants with varying roles from both institutions. From site 1, one site PI and one physician champion were interviewed. From site 2, one site PI, one physician champion, one nurse champion, and one informatics analyst were interviewed.

Investigators considered additional interviews but felt as though sampling and thematic saturation were achieved as each interviewee as part of their site’s PRIM-ER QI leadership team was able to speak to all key components of the QI intervention regardless of their role and no new themes were emerging in the data.13

An interview guide was developed using portions of the RE-AIM framework to discuss local site context, experiences in the development of the intervention components, barriers, and facilitators that contributed to participation/nonparticipation (Reach in RE-AIM), successful implementation strategies (Effectiveness in RE-AIM), and health system–level barriers that may have impacted adoption (A in RE-AIM). Questions related to RE-AIM implementation and maintenance were not emphasized as these elements will be examined in depth with PRIM-ER main outcome results. Interviews were conducted by NYU research staff (AC, FRC; female and male respectively) with qualitative interviewing training and previous experience. FRC previously worked with the interviewees during the PRIM-ER intervention and thus all interviewees were familiar with the goal of interviews. The interviews and secondary mixed methods analyses were conducted post intervention, between April and June 2019. Each interview was approximately 30 min and conducted telephonically or in participants’ private offices. Verbal informed consent was obtained, and no non-participants were present. Interviews were recorded and transcribed. Field notes were made post-interview and transcripts were not returned to participants to make corrections. No repeat interviews were conducted.

Data Management and Analysis

The responses to the 10-question Physician Attitudes and Knowledge Survey using a five-point Likert scale were collapsed into three categories (strongly agree/agree, neutral, disagree/strongly disagree) mirroring the validated implementation of this instrument and analysis was conducted in R 3.5.2.10

Qualitative analyses were conducted and managed in Atlas.ti v8.3.20. Three investigators (AC, FRC, ST) used both deductive and inductive (grounded theory)14 approaches to code all interviews. An initial codebook was developed using preliminary themes that all investigators agreed upon. Two coders (FRC and ST) read each transcript independently and met in person with a third investigator (AC) who was responsible for resolving discrepancies that occurred. This process occurred for all transcripts. The codebook was iteratively modified, as new codes emerged. Once coding was complete, the investigators met to identify common themes, which were then applied to the RE-AIM framework for interpretation.

The study was approved by the NYU School of Medicine Institutional Review Board. We adhered to the consolidated criteria for reporting qualitative research (COREQ).15

RESULTS

Quantitative Results

A total of 197 people completed the Baseline Attitudes and Knowledge survey,10 an overall response rate of 65%. In contrast to the provider education components of the intervention, baseline survey completion data was anonymous and there were no consequences for those not completing the survey for ethical concerns and preservation of the integrity of the private responses. Intervention components could be required by department leadership like other educational clinical programs, but survey completion could not. The demographics of the survey participants are provided in Table 2. A majority of the respondents were nurses (65.5%) and they tended to be female (87.6%), which is consistent with the field of nursing globally.16,17,18 The gender of provider type was almost equally distributed, with there being slightly more female respondents (56.5% female, 43.5% male). Within both sites, most respondents identified as being white (43.5% at site 1 and 64.4% at site 2), and non-Hispanic or Latino (88.2% at site 1 and 87.5% at site 2).

Table 2 Sample Characteristics of PRIM-ER Survey Respondents and Intervention Reach
Full size table

Table 3 outlines survey responses for all factor items identified in the baseline knowledge and attitudes survey including self-rated knowledge, support for hospice and philosophy, and views on physician-patient communication. At baseline, over half of the respondents (117 of 197) agree or strongly agreed that telling patients that they are dying is difficult. Despite this, the vast majority (91%, n = 173) agreed or strongly agreed that patients would benefit if hospice care were initiated earlier in the course of their illness. Additionally, 83% agreed or strongly agreed that hospice care generally meets the needs of the family better than conventional care.

Table 3 Provider’s Knowledge, Attitudes, and End-of-Life Care Practices of 197 Respondents Across Two Pilot Sites
Full size table

Both sites successfully completed all components of the intervention. Results on reach (R of RE-AIM) for the educational components are described in Table 2. Both sites achieved a high level of staff participation for the EM talk (physician/PA training) and ELNEC (nurse) trainings. The goal was to train 75% of the FTE EM providers and nurses, which both sites surpassed.

Qualitative Results

Two major themes emerged as facilitators contributing to the successful outcomes (E in RE-AIM) of the PRIM-ER pilot. These include leveraging (1) institutional leadership support and (2) established QI processes.

Main Theme 1: Institutional Leadership Support

Interviewees expressed that institutional leadership, both within their emergency medicine department and hospital-wide support was important for facilitating success and uptake of the PRIM-ER intervention. Strong support was categorized as engaging leaders that have the authority to make decisions on mandating trainings, substituting PRIM-ER faculty education for normally scheduled educational programming, and providing protected time for individuals responsible for developing and assisting in the implementation of the grant deliverables. One interviewee noted that “If you don’t have leadership support, forget about it.” (Site 1 MD champion). Participants from both institutions expressed that prior to PRIM-ER, QI was a priority to leadership (both departmental and hospital-wide), which was important as initial buy-in already existed. One participant expressed, “I think quality improvement is always at the forefront of leadership and administration in the ED.” (Site 2 MD champion). Reflecting on their past QI initiatives another interviewee noted, “We are engaged in a lot of quality improvement initiatives. This was probably a unique study. I think all the educational components made it a little bit more unusual.” (Site 1 PI).

Reach and effectiveness can also be attributed to departmental leadership at each institution having the autonomy to ensure that the in-person provider trainings were not only substituted for normally scheduled faculty education, but also mandatory.

Both sites valued faculty growth and development as leadership leveraged existing slots of time whereby providers and/or nurses would normally receive education in order to implement the PRIM-ER curricula. This substitution allowed for enhanced faculty adoption via seamless integration.

We basically do faculty development every month. We do all different types of programming during that hour [….] This was really a perfect program for that venue. (Site 1 MD champion)

Leadership allowed multiple sessions to accommodate clinical schedules, while also enforcing attendance. Only staff who had extenuating circumstances (e.g. maternity leave, clinical shift issues like post-night call, etc.) were excluded from participating in PRIM-ER, increasing reach. One PI stated: “Our chairman was like, ‘If you are off, you are coming. This isn’t an ‘Oh, maybe, yay’ activity. This is: We have a grant. You’re coming.’” (Site 1 PI). The other Site PI noted: “We wanted this to be mandatory, and we wanted to message it as such. We explained that people really won’t get their bonus if they don’t attend this.” (Site 2 PI).

Effectiveness was also facilitated by leaders providing the identified QI team protected time to work on the development and implementation of the PRIM-ER components. When asked about successful implementation strategies that contributed to this outcome, the Informatics Analyst expressed having protected time from hospital and departmental leadership crucial. He noted, “Giving me the assignment and the time allotted to work on it, because it was many hours and weeks that were put into this build (of the CDS). Having the support of leadership to understand the workload and what needs to be done.” (Site 2 informatics analyst).

Main Theme 2: Established QI Processes

Leveraging established QI processes at each institution also helped to facilitate effectiveness. QI processes that existed pre-intervention included effective cross-discipline communication and collaborations and audit and feedback improvement strategies (e.g. data monitoring and performance feedback).

Participants expressed that leveraging existing interdisciplinary partnerships and collaborations among the designated QI team (e.g. provider champion and nurse champion cross communication) was a key strategy in unified success in implementing the PRIM-ER components.

One interviewee noted:

I think one thing that is really important is communication […] When there's broken communication at the site, and people don't talk to each other about what they're supposed to be doing and don't have frequent check-ins, it really falls apart. I think communication is just so important that people don't stay in their silos. (Site 2 PI)

Another interviewee described similar strategies impacting success:

I would recommend in the other different sites that the providers and nurses have weekly or biweekly conference calls to keep each other updated on the project as they roll it out. That was one thing that was helpful. It just keeps everybody in line. (Site 2 RN champion)

Capitalizing on the existing process of data monitoring and feedback was an important aspect in the site effectiveness of the CDS component. Interviewees attributed their CDS successes to being employed at data-driven institutions as they were able to effectively develop and monitor clinical data for QI through existing audit and feedback mechanisms.

We have different projects we're working on. We track our issues on a white board right outside the ED. We have meetings with everyone in the ED every week. It's very front and center. We give a lot of personalized feedback to our attendings. (Site 2 PI)

I think the dashboard is a key component to the whole equation because it gives you as the developer of the alert system some feedback. That feedback is really critical to making sure that the CDS is effective.” (Site 2 MD champion)

Interviewees emphasized that the audit and feedback QI process was part of their institution’s culture, which aided in the optimization and implementation of this PRIM-ER component.

We send a report every month to our chair and leaders of the hospital to talk about what are the changes over time for a bunch of quality measures including palliative care. (Site 2 PI)

We have our electronic health record sub-committee that meets each month and goes over improvements. […] We do audit and feedback on a faculty level. Sometimes the chairman might be looking at how often the trigger’s happening or what the faculty are doing. (Site 1 PI)

Successful cross-disciplinary reach and effectiveness of the PRIM-ER educational components and implementation of the CDS tool with accompanying audit and feedback system was facilitated by capitalizing on the existing QI processes previously in place.

DISCUSSION

Since its conception, the RE-AIM framework has been applied across a variety of clinical trials, public health initiatives, and structural QI interventions, and has been cited over 2800 times in over 450 publications.19 As the RE-AIM framework emphasizes external validity and generalizability across diverse, real-world settings, it particularly useful in evaluating pragmatic trials,4 such as PRIM-ER, to understand what factors contributed to participation and success in an effort to maximize effectiveness of large-scale clinical initiatives.

Despite the abundance of literature on the use of RE-AIM, few studies use qualitative research methods when applying the RE-AIM framework, limiting a deeper understanding to explain “how” and “why” results happened.20,21 Purely quantitative results are not sufficient to understand the factors affecting participation in an intervention in clinical settings, and a major gap in the application of the RE-AM framework is reporting both quantitative and qualitative data.19,20,21,22,23 In a comprehensive review, Glasgow et al. indicated there is a need for greater understanding of the contextual factors that influence interventions such as organizational climate (e.g., staff and setting).19 Using quantitative and qualitative data, our findings fill this important gap in understanding in a real-world clinical setting the factors that contributed to implementation success. Using RE-AIM qualitatively can also assist in planning and designing future interventions, as data ascertained can help QI clinical champions understand the contextual factors that aided in obtaining the desired results.24 Ensuring that QI interventions are designed accordingly can aid in high intervention fidelity, as compatible interventions are more likely to be adopted.25

Using qualitative data specifically, we were able to elicit supporting evidence that demonstrated that institutional leadership support and established QI processes were key factors contributing to high reach and effectiveness. These findings support QI literature which emphasizes the role of leadership support in implementing a QI initiative,26,27,28,29 and the importance of evaluating constructs in order to tailor interventions to local context.30 The use of our qualitative data enriched our understanding of our quantitative reach findings.

Expanding on Damschroder’s Consolidated Framework for Implementation Research (CFIR), our findings support the need for exploring inner setting constructs that may impact reach and effectiveness.30 For example, commonly referred to as CFIR’s Networks and Communications, we found cross-discipline communication, and building a strong team that is able to communicate effectively, and coordinate across disciplines is essential for successful implementation.30,31,32,33,34,35 The CDS and audit and feedback components were successful at both sites due to the fact that the intervention was compatible with existing infrastructures, workflows, and technologies, demonstrating a positive climate that allowed and rewarded implementation.30,36 Additionally, we suspect that the high self-reported knowledge and attitudes on palliative care played an important factor in achieving our targeted “reach” outcomes as participants may have been more willing to engage in the intervention educational components. Our findings outlined demonstrate the importance of institutional leadership and support. Institutional leadership and support were critical components in successfully implementing the pilot intervention and was the catalyst for evaluating leaders behaviors (knowledge, perseverance, role modeling, etc.) within the main PRIM-ER grant using the Implementation Leadership Scale.37

Strengths and Limitations

Lack of reach data on nonparticipants remains a limitation in the RE-AIM literature and in our study.20 Both pilot sites had high baseline knowledge and attitudes on palliative care based on the baseline survey results, which may not be generalizable across the additional 33 sites implementing the main PRIM-ER initiative. These baseline results may have also impacted “reach” (of RE-AIM) of the educational components (EM Talk and ELNEC), as we do not have a clear understanding of participants’ knowledge and attitudes on palliative care who did not complete the survey or participate in the trainings, indicating a potential selection bias. The study team will track site-level contextual factors and assess baseline knowledge and attitudes as a potential predictor for adoption and non-adoption for the remaining RE-AIM constructs during the evaluation phase of the larger PRIM-ER initiative. The baseline survey implemented was originally validated in physicians, but has subsequently been adapted and used for advanced practice providers and nurses,38,39,40 a strength of using this instrument.

The qualitative interviews were conducted with a diverse group of individuals from both pilot sites and therefore represent a holistic understanding of the facilitators to implementation at two independent health systems, which is a strength of this analysis; these findings can be applied broadly to the remaining sites implementing the QI initiative in the main PRIM-ER study. Lastly, one limitation of this study is that we only applied two components of the RE-AIM framework to our analyses. This was purposeful, as firstly we wanted to do a deeper qualitative dive regarding the factors that contributed to our successes and secondly, as the main PRIM-ER grant is intending on evaluating all five components across all 35 sites as outlined in our protocol.8

Clinical Implications and Future Direction

For successful “reach” and “effectiveness”, there is a need to capitalize on institutional leadership support and QI processes. With the aim of scaling up the intervention,41 we will use these findings and lessons learned to support the other 33 sites’ PRIM-ER implementation by suggesting the following:

  1. 1.

    Mandating training sessions;

  2. 2.

    Substituting intervention specific education for normally scheduled faculty development time;

  3. 3.

    Providing protected time for individuals responsible for assisting in the implementation of the educational trainings and CDS development; and

  4. 4.

    Building on existing QI processes to enhance cross-disciplinary communication and CDS integration.

These findings can aid in guiding the 33 intervention sites as well as practicing clinicians interested in embedding systemic evidence-based quality improvement initiatives within their clinical practice and institutions.

CONCLUSION

As palliative care interventions in the ED capture high-risk patients at time of crises and can dramatically improve patient-centered outcomes, it is necessary to understand why and how certain factors impact the achievement of the desired outcomes prior to scaling up an initiative. Having strong institutional leadership support and leveraging QI processes can create synergies for success in the implementation of pragmatic QI clinical interventions.