Capacity-Building for Collecting Patient-Reported Outcomes and Experiences (PRO) Data Across Hospitals

Purpose Patient-reported outcomes and experiences (PRO) data are an integral component of health care quality measurement and PROs are now being collected by many healthcare systems. However, hospital organizational capacity-building for the collection and sharing of PROs is a complex process. We sought to identify the factors that facilitated capacity-building for PRO data collection in a nascent quality improvement learning collaborative of 16 hospitals that has the goal of improving the childbirth experience. Description We used standard qualitative case study methodologies based on a conceptual framework that hypothesizes that adequate organizational incentives and capacities allow successful achievement of project milestones in a collaborative setting. The 4 project milestones considered in this study were: (1) Agreements; (2) System Design; (3) System Development and Operations; and (4) Implementation. To evaluate the success of reaching each milestone, critical incidents were logged and tracked to determine the capacities and incentives needed to resolve them. Assessment The pace of the implementation of PRO data collection through the 4 milestones was uneven across hospitals and largely dependent on limited hospital capacities in the following 8 dimensions: (1) Incentives; (2) Leadership; (3) Policies; (4) Operating systems; (5) Information technology; (6) Legal aspects; (7) Cross-hospital collaboration; and (8) Patient engagement. From this case study, a trajectory for capacity-building in each dimension is discussed. Conclusion The implementation of PRO data collection in a quality improvement learning collaborative was dependent on multiple organizational capacities for the achievement of project milestones.


Introduction
Patient-reported outcomes and experiences (PRO) data are an integral component of health care quality measurement (Cella et al., 2015) and PROs are now being collected by many healthcare systems. These data have produced valuable insights at multiple levels, e.g., for patient-provider interactions and decision-making, for informing and monitoring quality improvement (QI) activities within hospitals, and for comparisons across hospitals (Black et al., 2016;Lavallee et al., 2016;Marshall et al., 2021).
The capture of PRO data for comparisons of the patient experience across hospitals has occurred through conditionspecific registries that allow patient contact for this purpose (Helsten et al., 2016;O'Connor et al., 2021;Haider et al., 2020;Zheng et al., 2014); in turn, these data may be linked to clinical data and used for research or QI activities (Auffenberg et al., 2021;Marshall et al., 2021).
Despite its importance, the collection of PRO data is often challenging (Bingham et al., 2016;Lavallee et al., 2016;Nordan et al., 2018). It relies on multiple organizational capacities (i.e., resources, capabilities and competencies) and includes strong incentives and leadership, and advanced electronic processes for collecting and linking patient data from multiple sources (Korst et al., 2008). Given that the capacity to collect PRO data is fundamental to the national vision of having PROs integrated within the electronic health record (EHR) (Snyder et al., 2017), further research is needed. We sought to identify the organizational capacities for PRO data collection required by hospitals participating in a nascent QI learning collaborative that has the goal of improving the childbirth experience across multiple hospitals and hospital systems. Childbirth is a highly preference-sensitive condition (Wennberg et al., 2002), and measurement of the childbirth experience is highly reliant on PROs (Nilver et al., 2017). The focus here is on the implementation of the PRO data-collection process in which patients express their values and preferences (V&P) antepartum and report their outcomes (PROs) postpartum; the intervention being evaluated is the collection of these V&P and PRO data.

Project Description
This project involved the implementation of the Childbirth Experience Survey (CBEX) in 16 childbirth hospitals in California from 2018 to 2021. CBEX was developed and implemented by the Childbirth Patient-Reported Outcomes Partnership and includes 18 domains relevant to the childbirth experience (Gregory et al., 2019;Korst et al., 2018). In CBEX part 1, administered in the last month of pregnancy, patients reported the aspects of the childbirth experience that they felt were important (V&P). In CBEX part 2, patients reported what happened to them (PROs) and their satisfaction with that care.
CBEX was administered online through a digital perinatal healthcare coordinator that provided a secure platform to consent and register patients for the research study and to collect and export deidentified data to the research team.
Patients were offered a small incentive ($5 gift card) for each of the surveys completed.
In 2018, the research team worked with participating hospitals to assure that Institutional Review Board (IRB) requirements were fulfilled and that recruitment strategies were adapted and implemented. Enrollment was paused for most of 2020 because of the COVID-19 pandemic. In 2021, the data were exported to the research team for analysis, and a benchmarking report of deidentified results was developed.

Methods
This study was approved by the lead IRB under Pro00050845. We used qualitative descriptive case study methodology (Yin, 2018). The authors reviewed all archival documents associated with the project, attended weekly meetings for troubleshooting and monitoring, and observed final meetings between the research team and participating hospital staff. The data sources that were used to examine the objectives and assess the progress of the project with respect to its milestones are listed in Table 1, which also provides a summary of the evidence used to evaluate the importance of organizational capacities by project milestone.
We used a modified conceptual framework detailed by Korst et al. to guide our data collection efforts (Korst et al., 2008). This framework hypothesizes that adequate organizational incentives and capacities allow successful achievement of project milestones in a collaborative setting. In addition to incentives, organizational capacities were classified as: Leadership, Policies/IRB, Operating systems, Information technology (IT), Legal aspects, Cross-collaboration, and Patient impact and engagement. The 4 project milestones considered here were (1) Agreements; (2) System Design; (3) System Development and Operations; and (4) Implementation.
Hospitals were initially approached based on their relationship with members of the research team. Team members visited sites and conducted readiness assessments based on the Consolidated Framework for Implementation Research (Damschroeder et al., 2009). Over a 3-year period, 24 hospitals with diverse patient populations were approached, and, of these, 16 enrolled patients.
To evaluate the success of reaching each milestone, critical incident tracking was used to determine the capacities and incentives needed to resolve them (Neale, 2008). Critical incidents were defined as problematic situations that occurred at either the project or hospital level. Tracking these critical incidents and how they impacted the success of meeting milestones provided the evidence for our evaluation of the organizational capacities needed for success.

Results
Participating sites received a CBEX hospital report that provided information for identifying hospital best practices and developing future goals for QI initiatives. Critical incidents were organized by milestone (Table 2). These critical incidents and the data associated with them were used to document the sequence of events involved in the implementation and to explore why milestones were or were not met by each participating hospital and by the project as a whole. Several of the critical incidents are discussed below by milestone.

Milestone 1: Agreements
The original proposal for this project involved using electronic patient lists. It was intended that each site create a list of eligible patients for the third-party data collection company, which would initiate patient contact. Most hospitals were unable to pursue antepartum recruitment using such a list because they had paper-based pre-registration systems, with many pregnant patients remaining unregistered until they presented at the hospital for delivery. This compelled a switch to in-person antepartum recruitment. The lead IRB agreement was then established to allow "non-engagement in research" of the participating sites. Non-engagement means that participating hospital staff are limited to notifying patients regarding the study and does not allow for other research activities such as providing informed consent, administering incentives, or collecting protected health information (PHI) (U.S. Department of Health and Human Services, 2008).  13 7,9,13,17,20 17,18,14,15,20 Incentives 14,15,16,21 9,13 9,13,22 7,14,15,17,18,21,22  Lead hospital and IT vendor: needed to change vendor because of requirement for personal data collection from respondent, specifically birth date.
New vendor acquired. Extensive conversations between team and vendor to communicate anticipated details of patient data collection and security requirements.
For security reasons, vendors may base their platforms on multiple pieces of personal data. A proportion of the patient population for this project were immigrants. To optimize recruitment, the Childbirth Partnership advised limiting personal data collection. Team required that IT vendor be responsible for any PHI breaches.

Agreements: hospitals
Some hospitals with prior verbal agreements never joined the project, with some staff citing "survey fatigue." There were multiple issues with hospital personnel on medical or maternity leave or changing responsibilities (getting promotions).
Some hospitals required separate BAA with IT vendor.
Delays in onboarding sites were accommodated where possible. Staggered on-boarding was done intentionally to allow time for the team to work with each site. Sites without sufficient interest/follow-up, or adequate readiness were dropped. Multiple strategies were used to onboard hospitals, including: Readiness assessments On-site kick-off meetings IRB assistance BAA assistance The project team worked primarily with physician and nurse leadership in hospital obstetrical departments and hospitals were not required to have formal agreements. There was no payment to hospitals. Because of the need for assistance from the study team, hospitals that were in the same region were better able to participate and to receive assistance than hospitals that were farther away. For example, the program manager could visit local hospitals and meet with site leads locally, but it was more difficult with outlying hospitals. This was exacerbated by the COVID-19 pandemic.

Agreements: IRB
Several hospitals did not have internal IRB or the capacity to work with the lead IRB. Several hospitals had limitations placed on IRB, including no enrollment of teens, or mention of birth control.
Several hospitals needed to train staff for IRB certification. Constant revisions and amendments required to IRB process and to consent forms; these needed to be incorporated at all hospitals.
Some hospitals wanted to link to clinical data. Lead IRB had changing personnel. Lead IRB policies evolved over time.
The initial plan to recruit from pre-registration or postpartum database lists could not be put into place initially and the team shifted to an IRB process that allowed participating hospitals to make CBEX available but without hospital research engagement. Hospitals could use their own IRB or the lead IRB to cover their participation and account for variability from the policies of the lead IRB. IRB changes were minimized to allow the project to proceed. An alternative IRB consent form was offered to hospitals that wanted to link the survey data to their own clinical data.
The team's request to use existing database lists for patient recruitment was not feasible, making the team commit to inperson enrollment. The IRB was then initiated under the concept that staff at participating hospitals would not be engaged in research, and thus not be required to have IRB approval. The budget was not sufficient to pursue a more formal approach where clinic staff could be paid to actively seek out those who were eligible and flag charts for recruitment. Several hospitals were eventually able to implement recruitment efforts with antepartum or postpartum database lists approved by patient providers, and this approach was found to be more successful than in-person efforts. Respondents without a phone and an email address could not participate due to security checks.
Catholic hospitals removed survey questions regarding birth control.
One hospital system (separate IRB) had specific new items they wanted included.
Monitoring of survey issues and dates of fixes was an important task. Survey design continued to evolve both because of use as patient-outcome measures and because of topical interest such as the pandemic or changes in childbirth services. A process will be required to track revisions and validation of new items.
It remained unclear if email or text messages were more successful. Many patients did not use email (although email was required for sign-up) and text messages could be ignored or flagged as spam if patients were not expecting it. System design: obtaining Data Patient registration and recruitment into the study was an ongoing concern. Each hospital needed a different approach to recruitment to fit CBEX within the workflow.
The lead hospital was able to recruit using postpartum discharge lists that were approved by patient physicians.
One other hospital was also able to implement this strategy successfully. There was a need for an ongoing IT task list and an additional budget. Used Amazon tablets: required WiFi, ad removal, and programming with instructions to allow patients to take survey on tablet while awaiting appointment Used student volunteers to assist: they could get credit or letter of recommendation IRB regulations did not allow staff to receive incentives to recruit, but marketing materials were permitted. Many staff did not see CBEX recruitment as part of their duties. Some staff were interested and continued to make patients aware of CBEX.

Milestone 4 ongoing implementation Activities
Ongoing staff continuity problems, with staff on leave or changes in responsibilities Links to the survey were only given to patients, however, within this process, hacking of the program did occur, with people using false email addresses and phone numbers (together) to register as a different patient each time. The incentive was minimal ($5 per survey). "Postpartum only" recruitment was not able to be achieved by some hospitals. Covid brought a pause to all research activities for most of 2020, and an increased burden on all staff activity. There was a reduction of patients on tours/childbirth classes.
Lead hospital continued to provide assistance with potential recruitment techniques through site visits, personal assistance, and weekly group calls Hacking scenarios were detected and the affected links removed promptly by IT vendor. New links were issued.
How the breaches occurred was not identified. The team began with antepartum recruitment activities and later folded in the "postpartum only" activities when it was determined that lead staff could focus on this distinct recruitment. Postpartum recruitment involved distinct activities and personnel from antepartum recruitment.
After the pandemic started, the team shifted to online efforts for recruitment, including using the vendor's implementation of online prenatal visits in participating hospitals to leverage recruitment activities. Antepartum testing remained a productive area for recruitment. The team provided hospitals with a slide set for accessing the survey as hospitals shifted to providing online prenatal content. An interim analysis was conducted to assist in maintaining stakeholder engagement.
Partnership meetings allowed debriefing about COVID-19 events and impact within their hospitals and benchmark and provide emotional support to each other.
Recruitment continued to be the most concerning issue for all hospitals. The most successful recruitment strategies were through the antepartum testing area, where the patient spent time with staff who were often known to them, and through the database lists as described above. The use of volunteers and students was also successful until the onset of the pandemic. COVID-19 stimulated increased interest in online activities, and efforts were made to recruit patients in online prenatal visits, education, and childbirth classes.
Many staff were especially burdened during the pandemic. Weekly and monthly partnership meetings continued even when research was halted. This remained a good avenue for stakeholders to voice concerns about the COVID-19 pandemic and its impact on childbirth services and patient experiences.

Milestone 2: System Design
The research team worked extensively with the sites to develop recruitment strategies that fit within site policies and workflow. Recruitment was dependent on staff motivation and individual interest. Student volunteers could be used at some hospitals to actively recruit, and they were instrumental to recruitment success; however, their participation ended after the start of the COVID-19 pandemic.

Milestone 3: System Development/Operations
A variety of approaches were developed in collaboration with hospital staff to assist with recruitment. At some sites, staff were able to incorporate CBEX into their clinical duties, and such sites had higher recruitment rates. However, at other sites, staff involvement was difficult to develop and maintain.

Milestone 4: Ongoing Implementation
Recruitment was interrupted due to the COVID-19 pandemic. When IRBs allowed research to resume, sites shifted to online recruitment due to ongoing concerns about social distancing and limiting the number of visitors in clinical spaces. CBEX Part 2 was adapted for "postpartum only" recruitment. The lead hospital received approval to create lists of postpartum patients by provider name. Patients were contacted by e-mail with ~ 10% response rate. This approach was shared with other hospitals, some of which had success.

Discussion
Hospital organizational capacity-building for the collection and sharing of PROs is a complex process, especially when attempted across independent hospitals with varying degrees of experience in research. The achievement of each implementation milestone requires a variety of different activities and leadership skills are essential to developing the workarounds that move organizations into this new territory.
Using the evidence presented in Table 2, the investigators identified a target for each of the capacities listed in the original framework (Korst et al., 2008) based on learning health system goals and suggested next steps for achieving them. This resulted in a capacity-based trajectory for the advancement of childbirth PRO data collection and sharing ( Table 3). Descriptions of each of these targets and potential next steps to reaching the target are presented below.

Incentives
We posited that, under Incentives, the target is that the collection and use of PRO data be part of the hospital organization's strategic goals (Cella et al., 2015). External incentives (e.g., federal revenue withholding tied to hospital satisfaction) (The Centers for Medicare & Medicaid Services, 2021) and internal incentives (e.g., improved patient satisfaction) can drive hospital organizations to develop meaningful collection and use of PROs (Korst et al., 2011). A potential next step to improve incentives to integrate PROs with the organization's strategic goals would be to strengthen the vision of CBEX by demonstrating collaborative improvement in a single PRO and simultaneously demonstrating improved hospital satisfaction scores.

Leadership
The target for the Leadership capacity is to reach the executive-level patient experience officer (CXO) within the organization. The CXO position demonstrates full accountability and support of the executive leadership to drive tangible improvement (Breen et al., 2021;Melder et al., 2020). Potential next steps to improve hospital leadership capacity would be (1) to implement CBEX activities under more formal organizational accountability such as subcontracts that provide a financial incentive or infrastructure with salary support or direct assistance with data collection; and (2) to promote CBEX results at higher organizational levels to encourage interest of those accountable for the patient experience.

Policies/IRB
The target for the Policies/IRB capacity is to have CBEX data included in routine hospital QI activities. Suggested frameworks for learning health systems indicate that QI is a fundamental activity, and that research should be integrated into clinical practice (Damschroeder et al., 2009;Harrison & Grantham, 2018). Folding the collection of PRO data into routine QI activities would ease the burden of implementation (LeRouge et al., 2020), especially given that the risks from such data collection are minimal (Whicher et al., 2015). Revision of the IRB process to determine which QI activities require express prospective consent versus which may be addressed by routine disclosures is critical to the evolution and the future of PRO data collection and use (Finkelstein et al., 2015. A potential next step to improve hospital policies/IRB capacity would be to integrate CBEX into the hospital QI process.

Operating Systems
The target for the Operating Systems capacity is to have electronic processes in place to identify and contact patients eligible for QI activities (Allen et al., 2021). If such processes are in place, then denominators for QI activities can be determined, patients can be contacted by either a third-party or the relevant hospital, and response rates tracked and improved. Although most participating hospitals did not have electronic lists of antepartum patients, postpartum electronic lists could be created from the EHR or hospital discharge data. Potential next steps to improve hospitals' ability to participate in childbirth QI activities using electronic patient lists would be to provide resources to hospitals to create such lists and to define the primary activity of the CBEX project as QI to enable third-party patient contact.

Information Technology
The target for the IT capacity is to have PRO data integrated with, or at least linked to, the EHR (Snyder et al., 2017). Given the variability in EHR sophistication in the hospitals likely to participate in a learning collaborative, a third-party data collection mechanism may be required, and indeed, advisable. For the CBEX project, several hospitals under one system received informed consent for collection of patient identifiers so that a link between CBEX and hospital data could be created. Potential next steps to improve PRO data collection and EHR linkage capacity would be (1) if relevant, structuring the informed consent to include patient notification of PHI collection and EHR linkage; (2) integrating CBEX recruitment into virtual antepartum and postpartum clinical appointment platforms; and (3) offering CBEX recruitment as a QI initiative that is compatible with clinical workflow.

Legal
The target for the Legal Aspects capacity is to have a formal collaborative governance structure. Legal assistance is required to determine activities and resources to be used by the learning collaborative (Cella et al., 2015). The importance of legal services will be most relevant to later milestones required for such a collaborative, including expansion, governance, and sustainability. Given that legal views regarding data-sharing may vary widely by hospitals and hospital systems, potential next steps toward this target would be to achieve funding for collaborative activities and begin discussions regarding collaborative goals and formal relationships among the participants.

Cross-Collaboration
The target for the Cross-Collaboration capacity is to demonstrate successful and sustainable collaborative activities. The development of the capacity of hospital staff to interpret data and determine optimal and responsive interventions is a critical component of learning health systems (Foley & Vale, 2017). Potential next steps toward the determination of priority QI activities and goal development, implementation of interventions, and data-tracking would be (1) to assure adequate baseline data collection for a priority topic; (2) to identify evidence-based interventions (if possible) or best practices that may address key priority concerns for active participants; and (3) to identify easily addressed environmental/service issues that have been reported by patients as problematic.

Patient Impact and Engagement
The target for the Patient Impact and Engagement capacity is to demonstrate (1) improvement in PROs in the desired direction; and (2) ongoing participation in the collaborative process (Hartley & Seid, 2021;Key & Lewis, 2018;Rubin, 2017). It is important not only that patients provide feedback regarding services, but also that they should be active participants in the collaborative. If that participation is fulfilled, and hospitals respond, then, in theory, patient satisfaction should reach optimal levels. Potential next steps toward improving patient engagement capacity are to (1) develop increased organizational "value" for patient feedback and participation in QI activities (Fjeldstad et al., 2019); and (2) expand the use of patient advisory groups within hospitals and within the collaborative.

Study Limitations
The context of this study was unique, and its generalizability to other healthcare conditions or environments may be limited. However, Table 3 may suggest potential avenues that could be employed by other collaboratives for achieving these suggested targeted capacities. Second, although 16 hospitals were approached based on existing relationships, the sample of hospitals represented diversity in patient populations limiting potential biases. Third, the COVID-19 pandemic undoubtedly impacted all organizational capacities, limiting resources and availability of staff to participate. However, it was the catalyst for virtual recruitment. Finally, an independent evaluator was not budgeted nor required as part of the funding award and this may limit the objectivity of the findings.

Conclusion
This project demonstrates the complexity of hospital datasharing for QI activities in the context of building a learning collaborative using PRO data. Different organizational capacities are important for each implementation milestone. This evaluation presents a trajectory that outlines the progress needed to bring the implementation to its full maturity and helps clarify next steps towards meeting those targets.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.