Introduction

Patient-reported outcomes and experiences (PRO) data are an integral component of health care quality measurement (Cella et al., 2015) and PROs are now being collected by many healthcare systems. These data have produced valuable insights at multiple levels, e.g., for patient-provider interactions and decision-making, for informing and monitoring quality improvement (QI) activities within hospitals, and for comparisons across hospitals (Black et al., 2016; Lavallee et al., 2016; Marshall et al., 2021).

The capture of PRO data for comparisons of the patient experience across hospitals has occurred through condition-specific registries that allow patient contact for this purpose (Helsten et al., 2016; O’Connor et al., 2021; Haider et al., 2020; Zheng et al., 2014); in turn, these data may be linked to clinical data and used for research or QI activities (Auffenberg et al., 2021; Marshall et al., 2021).

Despite its importance, the collection of PRO data is often challenging (Bingham et al., 2016; Lavallee et al., 2016; Nordan et al., 2018). It relies on multiple organizational capacities (i.e., resources, capabilities and competencies) and includes strong incentives and leadership, and advanced electronic processes for collecting and linking patient data from multiple sources (Korst et al., 2008). Given that the capacity to collect PRO data is fundamental to the national vision of having PROs integrated within the electronic health record (EHR) (Snyder et al., 2017), further research is needed. We sought to identify the organizational capacities for PRO data collection required by hospitals participating in a nascent QI learning collaborative that has the goal of improving the childbirth experience across multiple hospitals and hospital systems. Childbirth is a highly preference-sensitive condition (Wennberg et al., 2002), and measurement of the childbirth experience is highly reliant on PROs (Nilver et al., 2017). The focus here is on the implementation of the PRO data-collection process in which patients express their values and preferences (V&P) antepartum and report their outcomes (PROs) postpartum; the intervention being evaluated is the collection of these V&P and PRO data.

Project Description

This project involved the implementation of the Childbirth Experience Survey (CBEX) in 16 childbirth hospitals in California from 2018 to 2021. CBEX was developed and implemented by the Childbirth Patient-Reported Outcomes Partnership and includes 18 domains relevant to the childbirth experience (Gregory et al., 2019; Korst et al., 2018). In CBEX part 1, administered in the last month of pregnancy, patients reported the aspects of the childbirth experience that they felt were important (V&P). In CBEX part 2, patients reported what happened to them (PROs) and their satisfaction with that care.

CBEX was administered online through a digital perinatal healthcare coordinator that provided a secure platform to consent and register patients for the research study and to collect and export deidentified data to the research team. Patients were offered a small incentive ($5 gift card) for each of the surveys completed.

In 2018, the research team worked with participating hospitals to assure that Institutional Review Board (IRB) requirements were fulfilled and that recruitment strategies were adapted and implemented. Enrollment was paused for most of 2020 because of the COVID-19 pandemic. In 2021, the data were exported to the research team for analysis, and a benchmarking report of deidentified results was developed.

Methods

This study was approved by the lead IRB under Pro00050845. We used qualitative descriptive case study methodology (Yin, 2018). The authors reviewed all archival documents associated with the project, attended weekly meetings for troubleshooting and monitoring, and observed final meetings between the research team and participating hospital staff. The data sources that were used to examine the objectives and assess the progress of the project with respect to its milestones are listed in Table 1, which also provides a summary of the evidence used to evaluate the importance of organizational capacities by project milestone.

Table 1 Data sources as evidence for the association between organizational capacities/incentives and project milestones
Full size table

We used a modified conceptual framework detailed by Korst et al. to guide our data collection efforts (Korst et al., 2008). This framework hypothesizes that adequate organizational incentives and capacities allow successful achievement of project milestones in a collaborative setting. In addition to incentives, organizational capacities were classified as: Leadership, Policies/IRB, Operating systems, Information technology (IT), Legal aspects, Cross-collaboration, and Patient impact and engagement. The 4 project milestones considered here were (1) Agreements; (2) System Design; (3) System Development and Operations; and (4) Implementation.

Hospitals were initially approached based on their relationship with members of the research team. Team members visited sites and conducted readiness assessments based on the Consolidated Framework for Implementation Research (Damschroeder et al., 2009). Over a 3-year period, 24 hospitals with diverse patient populations were approached, and, of these, 16 enrolled patients.

To evaluate the success of reaching each milestone, critical incident tracking was used to determine the capacities and incentives needed to resolve them (Neale, 2008). Critical incidents were defined as problematic situations that occurred at either the project or hospital level. Tracking these critical incidents and how they impacted the success of meeting milestones provided the evidence for our evaluation of the organizational capacities needed for success.

Results

Participating sites received a CBEX hospital report that provided information for identifying hospital best practices and developing future goals for QI initiatives. Critical incidents were organized by milestone (Table 2). These critical incidents and the data associated with them were used to document the sequence of events involved in the implementation and to explore why milestones were or were not met by each participating hospital and by the project as a whole. Several of the critical incidents are discussed below by milestone.

Table 2 Critical incidents, workarounds, and outcomes by milestones
Full size table

Milestone 1: Agreements

The original proposal for this project involved using electronic patient lists. It was intended that each site create a list of eligible patients for the third-party data collection company, which would initiate patient contact. Most hospitals were unable to pursue antepartum recruitment using such a list because they had paper-based pre-registration systems, with many pregnant patients remaining unregistered until they presented at the hospital for delivery. This compelled a switch to in-person antepartum recruitment. The lead IRB agreement was then established to allow “non-engagement in research” of the participating sites. Non-engagement means that participating hospital staff are limited to notifying patients regarding the study and does not allow for other research activities such as providing informed consent, administering incentives, or collecting protected health information (PHI) (U.S. Department of Health and Human Services, 2008).

Milestone 2: System Design

The research team worked extensively with the sites to develop recruitment strategies that fit within site policies and workflow. Recruitment was dependent on staff motivation and individual interest. Student volunteers could be used at some hospitals to actively recruit, and they were instrumental to recruitment success; however, their participation ended after the start of the COVID-19 pandemic.

Milestone 3: System Development/Operations

A variety of approaches were developed in collaboration with hospital staff to assist with recruitment. At some sites, staff were able to incorporate CBEX into their clinical duties, and such sites had higher recruitment rates. However, at other sites, staff involvement was difficult to develop and maintain.

Milestone 4: Ongoing Implementation

Recruitment was interrupted due to the COVID-19 pandemic. When IRBs allowed research to resume, sites shifted to online recruitment due to ongoing concerns about social distancing and limiting the number of visitors in clinical spaces.

CBEX Part 2 was adapted for “postpartum only” recruitment. The lead hospital received approval to create lists of postpartum patients by provider name. Patients were contacted by e-mail with ~ 10% response rate. This approach was shared with other hospitals, some of which had success.

Discussion

Hospital organizational capacity-building for the collection and sharing of PROs is a complex process, especially when attempted across independent hospitals with varying degrees of experience in research. The achievement of each implementation milestone requires a variety of different activities and leadership skills are essential to developing the workarounds that move organizations into this new territory.

Using the evidence presented in Table 2, the investigators identified a target for each of the capacities listed in the original framework (Korst et al., 2008) based on learning health system goals and suggested next steps for achieving them. This resulted in a capacity-based trajectory for the advancement of childbirth PRO data collection and sharing (Table 3). Descriptions of each of these targets and potential next steps to reaching the target are presented below.

Table 3 Trajectory matrix for hospital capacity building for sharing of patient-reported outcomes and experiences (PROs)
Full size table

Incentives

We posited that, under Incentives, the target is that the collection and use of PRO data be part of the hospital organization’s strategic goals (Cella et al., 2015). External incentives (e.g., federal revenue withholding tied to hospital satisfaction) (The Centers for Medicare & Medicaid Services, 2021) and internal incentives (e.g., improved patient satisfaction) can drive hospital organizations to develop meaningful collection and use of PROs (Korst et al., 2011). A potential next step to improve incentives to integrate PROs with the organization’s strategic goals would be to strengthen the vision of CBEX by demonstrating collaborative improvement in a single PRO and simultaneously demonstrating improved hospital satisfaction scores.

Leadership

The target for the Leadership capacity is to reach the executive-level patient experience officer (CXO) within the organization. The CXO position demonstrates full accountability and support of the executive leadership to drive tangible improvement (Breen et al., 2021; Melder et al., 2020). Potential next steps to improve hospital leadership capacity would be (1) to implement CBEX activities under more formal organizational accountability such as subcontracts that provide a financial incentive or infrastructure with salary support or direct assistance with data collection; and (2) to promote CBEX results at higher organizational levels to encourage interest of those accountable for the patient experience.

Policies/IRB

The target for the Policies/IRB capacity is to have CBEX data included in routine hospital QI activities. Suggested frameworks for learning health systems indicate that QI is a fundamental activity, and that research should be integrated into clinical practice (Damschroeder et al., 2009; Harrison & Grantham, 2018). Folding the collection of PRO data into routine QI activities would ease the burden of implementation (LeRouge et al., 2020), especially given that the risks from such data collection are minimal (Whicher et al., 2015). Revision of the IRB process to determine which QI activities require express prospective consent versus which may be addressed by routine disclosures is critical to the evolution and the future of PRO data collection and use (Finkelstein et al., 2015. A potential next step to improve hospital policies/IRB capacity would be to integrate CBEX into the hospital QI process.

Operating Systems

The target for the Operating Systems capacity is to have electronic processes in place to identify and contact patients eligible for QI activities (Allen et al., 2021). If such processes are in place, then denominators for QI activities can be determined, patients can be contacted by either a third-party or the relevant hospital, and response rates tracked and improved. Although most participating hospitals did not have electronic lists of antepartum patients, postpartum electronic lists could be created from the EHR or hospital discharge data. Potential next steps to improve hospitals’ ability to participate in childbirth QI activities using electronic patient lists would be to provide resources to hospitals to create such lists and to define the primary activity of the CBEX project as QI to enable third-party patient contact.

Information Technology

The target for the IT capacity is to have PRO data integrated with, or at least linked to, the EHR (Snyder et al., 2017). Given the variability in EHR sophistication in the hospitals likely to participate in a learning collaborative, a third-party data collection mechanism may be required, and indeed, advisable. For the CBEX project, several hospitals under one system received informed consent for collection of patient identifiers so that a link between CBEX and hospital data could be created. Potential next steps to improve PRO data collection and EHR linkage capacity would be (1) if relevant, structuring the informed consent to include patient notification of PHI collection and EHR linkage; (2) integrating CBEX recruitment into virtual antepartum and postpartum clinical appointment platforms; and (3) offering CBEX recruitment as a QI initiative that is compatible with clinical workflow.

Legal

The target for the Legal Aspects capacity is to have a formal collaborative governance structure. Legal assistance is required to determine activities and resources to be used by the learning collaborative (Cella et al., 2015). The importance of legal services will be most relevant to later milestones required for such a collaborative, including expansion, governance, and sustainability. Given that legal views regarding data-sharing may vary widely by hospitals and hospital systems, potential next steps toward this target would be to achieve funding for collaborative activities and begin discussions regarding collaborative goals and formal relationships among the participants.

Cross-Collaboration

The target for the Cross-Collaboration capacity is to demonstrate successful and sustainable collaborative activities. The development of the capacity of hospital staff to interpret data and determine optimal and responsive interventions is a critical component of learning health systems (Foley & Vale, 2017). Potential next steps toward the determination of priority QI activities and goal development, implementation of interventions, and data-tracking would be (1) to assure adequate baseline data collection for a priority topic; (2) to identify evidence-based interventions (if possible) or best practices that may address key priority concerns for active participants; and (3) to identify easily addressed environmental/service issues that have been reported by patients as problematic.

Patient Impact and Engagement

The target for the Patient Impact and Engagement capacity is to demonstrate (1) improvement in PROs in the desired direction; and (2) ongoing participation in the collaborative process (Hartley & Seid, 2021; Key & Lewis, 2018; Rubin, 2017). It is important not only that patients provide feedback regarding services, but also that they should be active participants in the collaborative. If that participation is fulfilled, and hospitals respond, then, in theory, patient satisfaction should reach optimal levels. Potential next steps toward improving patient engagement capacity are to (1) develop increased organizational “value” for patient feedback and participation in QI activities (Fjeldstad et al., 2019); and (2) expand the use of patient advisory groups within hospitals and within the collaborative.

Study Limitations

The context of this study was unique, and its generalizability to other healthcare conditions or environments may be limited. However, Table 3 may suggest potential avenues that could be employed by other collaboratives for achieving these suggested targeted capacities. Second, although 16 hospitals were approached based on existing relationships, the sample of hospitals represented diversity in patient populations limiting potential biases. Third, the COVID-19 pandemic undoubtedly impacted all organizational capacities, limiting resources and availability of staff to participate. However, it was the catalyst for virtual recruitment. Finally, an independent evaluator was not budgeted nor required as part of the funding award and this may limit the objectivity of the findings.

Conclusion

This project demonstrates the complexity of hospital data-sharing for QI activities in the context of building a learning collaborative using PRO data. Different organizational capacities are important for each implementation milestone. This evaluation presents a trajectory that outlines the progress needed to bring the implementation to its full maturity and helps clarify next steps towards meeting those targets.