Patient and Stakeholder Engagement in the PCORI Pilot Projects: Description and Lessons Learned

BACKGROUND Patients and healthcare stakeholders are increasingly becoming engaged in the planning and conduct of biomedical research. However, limited research characterizes this process or its impact. OBJECTIVE We aimed to characterize patient and stakeholder engagement in the 50 Pilot Projects funded by the Patient-Centered Outcomes Research Institute (PCORI), and identify early contributions and lessons learned. DESIGN A self-report instrument was completed by researchers between 6 and 12 months following project initiation. PARTICIPANTS Forty-seven principal investigators or their designees (94 % response rate) participated in the study. MAIN MEASURES Self-report of types of stakeholders engaged, stages and levels of engagement, facilitators and barriers to engagement, lessons learned, and contributions from engagement were measured. KEY RESULTS Most (83 %) reported engaging more than one stakeholder in their project. Among those, the most commonly reported groups were patients (90 %), clinicians (87 %), health system representatives (44 %), caregivers (41 %), and advocacy organizations (41 %). Stakeholders were commonly involved in topic solicitation, question development, study design, and data collection. Many projects engaged stakeholders in data analysis, results interpretation, and dissemination. Commonly reported contributions included changes to project methods, outcomes or goals; improvement of measurement tools; and interpretation of qualitative data. Investigators often identified communication and shared leadership strategies as “critically important” facilitators (53 and 44 % respectively); lack of stakeholder time was the most commonly reported challenge (46 %). Most challenges were only partially resolved. Early lessons learned included the importance of continuous and genuine partnerships, strategic selection of stakeholders, and accommodation of stakeholders’ practical needs. CONCLUSIONS PCORI Pilot Projects investigators report engaging a variety of stakeholders across many stages of research, with specific changes to their research attributed to engagement. This study identifies early lessons and barriers that should be addressed to facilitate engagement. While this research suggests potential impact of stakeholder engagement, systematic characterization and evaluation of engagement at multiple stages of research is needed to build the evidence base. Electronic supplementary material The online version of this article (doi:10.1007/s11606-015-3450-z) contains supplementary material, which is available to authorized users.


INTRODUCTION
Patients and other key healthcare stakeholders are increasingly becoming engaged in the planning and conduct of biomedical research. 1,2 Patients and stakeholders can be engaged across stages of research including identifying study topics, choosing hypotheses, analyzing data, and disseminating findings [3][4][5] (see Table 1 for illustrative examples). Levels of engagement range from consultation, to collaboration in bi-directional partnerships with researchers, to stakeholder-directed projects. 6 Participatory research approaches have been increasingly recognized as potentially beneficial in several countries, 7-10 and a robust community-based participatory research literature documents key issues in including communities in research. 11 In the US, growing support for engaged health research is demonstrated by patient engagement programs at several federal health agencies. [12][13][14][15] In 2010, the Patient-Centered Outcomes Research Institute (PCORI) was established to fund comparative clinical effectiveness research (CER) to assist patients, clinicians, and other healthcare stakeholders in making informed health decisions. 16 PCORI requires research engagement in all its funded CER studies. 17 Despite increased attention to engagement, descriptive information about engagement in health research is limited. Some reviews describe engagement in published studies and report impacts of engagement, 1,3,18-21 but the sparse detail on how engagement is implemented, 22,23 and the lack of systematic characterization and evaluation of engagement 5,24,25 highlight the need for additional approaches for obtaining information on engagement. Some earlier studies describe engagement in research in the U.K., 26,27 but no systematic measures exist to characterize or evaluate engagement. The literature provides insufficient information to permit identification of best practices and guidelines for replication by others. Researchers have acknowledged the need for a better understanding of optimal approaches to engagement, and the opportunity to learn more about engagement from PCORI's funded projects. 24 This paper presents our work to collect information about engagement of patients and other stakeholders in PCORI's first awardees, the 50 PCORI Pilot Projects.

Sample and Data Collection
In 2012, PCORI awarded contracts totaling $31 million for 50 PCORI Pilot Projects (http://www.pcori.org/assets/PCORI-Pilot-Projects-Funding-Announcement-Amendment-1-_v2_-09302011.pdf). The intent of this funding was to advance methods for patient-centered outcomes research. Unlike subsequent PCORI funding, these first projects were intended to have an explicit focus on methods (analytic methods, outcome measures, or other specific aspects of research methods). Projects were funded for a maximum of 2 years and $250,000 in direct costs. Engagement of patients and other stakeholders was one criterion by which PCORI evaluated the proposals for funding (as well as significance, investigators, innovation, approach, and environment). PCORI required engagement unless the applicant sufficiently explained why engagement was not pursued.
AcademyHealth, a national health services and policy research organization, was competitively selected to support active management of the PCORI Pilot Projects. PCORI and AcademyHealth collected data regarding engagement of patients and other stakeholders in the design, conduct, and dissemination of the projects (response was voluntary). MaGil IRB provided Institutional Review Board (IRB) approval for secondary analysis of these data.
All 50 PCORI Pilot Project principal investigators (PIs) or their designees were invited to complete an online questionnaire 28 in July 2013 (approximately 6 to 8 months following project initiation). We sent two reminders 3 and 6 weeks later and a final personalized reminder in early November.

Measures
The instrument (Appendix A, online) was developed by the PCORI and AcademyHealth team based on PCORI's conceptual model for patient-centered outcomes research, 29 key informant interviews with five Pilot Project PIs, qualitative analysis of Pilot Project research proposals, discussion with the PCORI Patient Engagement staff, and literature review, as noted below. In addition to capturing descriptive elements (who, when, and extent of engagement), the instrument content followed the conceptual model categories of ways to partner, communication, capture and use of perspectives, and perceived influence of partners. To inform future practice, the instrument also addressed challenges to and facilitators of engagement.
Stakeholders Engaged. Respondents reported whether they had engaged patients or stakeholders in ways other than as research subjects, the types of stakeholders engaged in their project, and the number of individuals engaged.
Level of Engagement. Respondents classified the extent to which each stakeholder type was engaged: 1) Consultation: researchers use stakeholder views to influence decisionmaking regarding research. Consultation allows researchers to obtain stakeholder views without necessarily being committed to act on them; 2) Collaboration: stakeholders have an ongoing partnership with researchers and greater ownership of the project; or 3) Stakeholder-led: stakeholders initiate, design, and undertake the research process. 6

Analysis
Descriptive statistical analyses (including frequencies, proportions) were performed using the survey platform. 28 The majority of items were created as closed-ended Likert-scaled ratings; we examined percent of sample reporting each response category and interpreted based on response scale. We analyzed responses from open-ended questions (motivations for engagement, establishment of relationships, challenges, lessons learned, and contributions of stakeholder partners) using a descriptive qualitative approach, with each question treated as its own deductive code. Responses to each question were grouped by sub-codes representing themes identified inductively by one member of the team (LEE), and reviewed by another member (LPF). Disagreements were resolved through consensus.

RESULTS
Researchers from 47 of the 50 projects completed questionnaires (94 %; 33 PIs and 13 other designated research staff). These projects were geographically distributed(Northeast=30 %, South=27 %, Midwest=12 %, West=30 %). Almost half (46 %) of the projects focused on four health care decision supports; while 30 % focused on research prioritization and analytic methods, 15 % were about developing outcomes instruments, and 13 % related to developing technology and infrastructure for patient-centered outcomes research.

Engagement of Patients and Other Stakeholders
Most respondents (83 %, N=39) reported engaging patients or other stakeholders in their project in ways other than as research subjects. Among those, investigators most commonly reported engaging patients/ consumers (90 %), clinicians (87 %), clinic/hospital or health system representatives (44 %), caregivers/family members (41 %), or patient or caregiver advocacy organizations (41 %) ( Fig. 1). Most projects (84 %) engaged at least two types of stakeholder; 22 % reported engaging four or five types of stakeholders. The primary motivation for engaging patients, their caregivers, and advocacy organizations was to understand their values and perspectives on items such as the impact of a disease, the most appropriate interventions, the importance of specific study outcomes, and research products that meet patients' needs. The primary motivations for engaging clinicians included their Bclinical and methodological expertise^and Binsights^into care delivery, obtaining clinician Bbuy-in,^and ensuring that the research outcomes were clinically meaningful and translatable.
A notable proportion of relationships between researchers and stakeholders existed for < 1 year (37 % of patients, 36 % of caregivers/family members, 19 % of clinicians, 33 % of patient or caregiver advocacy organizations, 27 % of health system representatives). However, some researcher-stakeholder relationships were more long-standing (> 3 years), particularly relationships between researchers and clinicians (50 % of clinicians, 36 % of health system representatives, 27 % of caregivers/family members, 23 % of patients, 22 % of patient or caregiver advocacy organizations). Researchers reported establishing new relationships with patient stakeholders through traditional recruitment in the clinical care setting (e.g., staff outreach, mailings, and flyers); clinician referrals; community outreach; or utilizing investigators' existing networks with colleagues, patients, and advocacy organizations.
Regarding engagement of patients specifically, 15 % of projects reported engaging one patient, while 56 % of projects reported engaging ≥ six patients. Most projects reported engaging patients as consultants (35 %) or collaborators (53 %) rather than as co-leaders (6 %). At the time of assessment, patients had engaged most commonly at the stages of research question development, proposal development, study design, data collection, topic solicitation, and results review (Fig. 2). Respondents engaging other stakeholder types gave similar responses regarding the level of engagement and research stages (Appendix B, online).

Challenges to and Facilitators for Engagement
The most commonly reported challenges to engagement were lack of stakeholder time (46 %), lack of research team time (35 %), lack of stakeholder training/ background (30 %), difficulty in finding the appropriate representatives to engage (27 %), lack of research team resources (24 %), and lack of research team training/ background (22 %). Respondents generally described these challenges as Bpartially resolved,^and shared some tangible strategies ( Table 2). Respondents also reported establishing trust and learning how to work together as challenges. Some solutions to these challenges included allowing ample opportunity for stakeholders to ask questions, communicating through multiple channels, having face-to-face meetings, making time to socialize, and training researchers on engagement.
Respondents most commonly endorsed communication strategies and shared leadership approaches as Bcritically important^facilitators (49 and 41 %, respectively). Examples of communication strategies included: trying different arrangements (big group, small group, individual) to see what works best; maintaining regular contact with stakeholders; interacting through multiple channels (online, teleconference, in-person); using non-technical language; and holding pre-meetings with stakeholders  to go over items to be discussed in larger meetings. Examples of shared leadership strategies included: adopting a perspective of shared responsibility for decision-making, training of research team members, and involving research partners in research team meetings. Respondents also endorsed remuneration, training of stakeholders, and training of researchers as Bimportant( 41 %, 38 %, and 30 %, respectively) (Fig. 3).

Early Lessons Learned about Engagement
Early lessons learned (Table 3) suggest the benefits of establishing meaningful partnerships with stakeholders rather than engaging stakeholders merely to fulfill a requirement. Researchers reported the importance of selecting stakeholders strategically to fit project needs, continuously involving stakeholders, adapting to the practical needs of stakeholders, and clearly defining roles and expectations for research partners.
Respondents also emphasized the importance of in-person contact for relationship building.

Contributions of Patients and Other Stakeholders
Twenty-one projects (57 %) planned to assess the influence of stakeholders on the research process. The themes that emerged regarding early tangible contributions to the research projects from patients and stakeholders included changes to the project outcomes or goals, changes to the project methods, enhanced access to study populations or study settings, modifications to the study interventions, and refinement of the study instruments or interview questions. Although still in the early stages, several projects identified contributions of patients and stakeholders to results interpretation and dissemination of findings (Table 3).

DISCUSSION
To address the need for more information about engagement in research, we created and fielded a data collection tool to obtain information about stakeholder engagement practices and strategies employed in PCORI's Pilot Projects. In the first 6 to 12 months of these projects, researchers most commonly engaged patients, and the majority of projects engaged multiple types of stakeholders. Compared to the published literature, which reflects lesser engagement of other types of stakeholders, 1,18 many PCORI Pilot Projects reported engaging clinicians (> 85 %), and a substantial number of projects reported engaging with caregivers, advocacy organizations, and health system representatives (>40 % for each). However, similar to previous studies, few investigators reported engaging purchasers, payers, policy makers, and the life sciences industry. This pattern may reflect the basic methods research focus of the funding; the stakeholders engaged by these awardees are those most proximal to the methods questions addressed. Subsequent PCORI funding that has a focus on comparative effectiveness research rather than methods research is expected to expand the universe of stakeholders engaged. Our findings highlight the need for a better understanding of how the views from multiple types of stakeholders are incorporated into a study and of the impact of relationship length and quality on the research process. While engagement was most frequently reported in developing research questions, consistent with other reports, 1,37 the extent of engagement in data collection, data analysis, interpretation of results, and dissemination is noteworthy. Engagement in these later phases is expected to increase as projects progress. While the challenges, facilitators, and lessons learned reported here are generally consistent with the existing literature, 3,5,19,20,38-40 this study is unique in capturing this information systematically and assessing the relative importance across a group of projects. Our findings collectively highlight the effort, attention, resources, and flexibility required for concerted engagement. Across these survey responses, for example, respondents emphasized meaningful and continuous partnerships, shared leadership strategies, communication through multiple strategies that fit specific stakeholder partners, and adaptation to the practical needs of stakeholders. Similarly, others highlighted the importance of sharing power for successful research engagement. 41 Lack of time-both for stakeholders and researchers-was the most frequently noted challenge and was identified as a top barrier to engagement in a recent survey of patients and clinicians. 39 Thus, it is critical to recognize resource needs and develop strategies that maximize stakeholder input while minimizing the time burden, and the responses provide support for elements in the conceptual model that helped to inform these items. 29 Most challenges experienced were only partially resolved. Challenges to engagement remain to be addressed in terms of individual research team process as well as through institutional and funding agency infrastructure. Some issues have tangible solutions (e.g., financial support of partnership in research budgets, matching researchers with stakeholder partners); others require change in social norms (e.g., developing meaningful partnerships). PCORI supports several efforts to address these issues, including funding opportunities for developing research partnerships (http://www.pcori.org/fundingopportunities/pipeline-to-proposal-awards/) and for developing training and building capacity for engagement (http:// www.pcori.org/funding-opportunities/eugene-washingtonpcori-engagement-awards/). This study also demonstrates the contributions that engaged stakeholders had already made in the first year of the projects. Researchers reported that patient and stakeholder input resulted in changes to study outcomes, goals, methods, interventions, and materials. These contributions may enhance the appropriateness of study materials, 19,21 and the relevance of the research questions and outcomes 1,3,18,21,22,[42][43][44] to the study population. Some projects reported stakeholder contributions to the interpretation of qualitative data, which may help researchers to contextualize findings for specific audiences. 3,21 Increased access to study populations or settings suggests that engagement may mitigate commonly encountered challenges with recruitment and retention. 3,18,22,43 These results are consistent with available evidence on positive impacts of engaged research on research process and dissemination of results. 1,6,18,21,22,[42][43][44][45] More research is needed to better understand how engagement impacts research quality, relevance, and dissemination.
While the findings represent significant progress towards systematically understanding engagement in research, several limitations should be considered. We obtained only researchers' perspectives, which may differ from the views of other stakeholders engaged. 26 This tool did not assess principles of engaged research (e.g., trust, respect) that have been identified as important to the success of engagement. While others have noted important negative impacts of engagement, 21 the current tool did not specifically ask about these effects and thus did not provide new insights on drawbacks. Obtaining information at later stages of research projects will permit evaluation of engagement outcomes. Additional information about construct validity of the measure is needed. Results obtained here should be compared to other forms of project reporting, including open-ended project status updates required of awardees and conversations between funding agency staff and awardees. Future data collection efforts should include appropriately timed repetition of key items. Additionally, further cognitive testing of items is warranted to ensure item intent is understood by respondents and that responses reflect respondent intent.
The generalizability of these findings may be limited because respondents were investigators on methods projects rather than health outcomes or CER. Because these respondents were the first PCORI awardees, they also may have more interest in and experience with engagement than the broader health research community. Thus, their knowledge and attitudes about engagement may be more positive. Finally, these self-report data may overestimate research engagement, particularly given the role of the research funder in data collection. Many of these limitations will be addressed in future data collection efforts with PCORI awardees.
Engagement in research is becoming more widespread as the value of end-user needs and perspectives is recognized, and as funding initiatives requiring or encouraging engagement proliferate. Since the PCORI Pilot Projects were funded, researchers have become more familiar with the concept of research engagement. PCORI has provided additional guidance to applicants regarding its funding requirements related to engagement in research and is continuing to collect information on engagement from PIs and their patient and stakeholder partners to inform the literature and PCORI's policies regarding research engagement. Seek genuine partnership Respondents emphasized that partnerships with stakeholders must be genuine. For example, one respondent commented, Btheir participation was enhanced because they quickly realized that their role was not symbolic in nature but was integral to the project's development in many ways.^Respondents sought to actively involve stakeholders in the decision-making process, which contributed to shared ownership of the project. Theme 2: Select stakeholders strategically Respondents recognized the need to strategically select stakeholders representing a variety of different communities or viewpoints and the need to identify the necessary viewpoints to match project specific goals. For example, one respondent described their work as follows: BThe three types of stakeholders are quite different and all have been essential to the project. We would not have written a proposal without the administrative stakeholders, the patients and providers are describing the issue of behavioral health within the clinic and identifying their preferences in patient-centered care, and the stakeholder group is deciding how they can feasibly change the way they administer services to meet these needs.T heme 3: Continuously involve stakeholders Respondents acknowledged the need to Bstart early^and Bask often^as patient and other stakeholders have unique knowledge and improvements to add to the research process along the entire research continuum. Theme 4: Adapt to the practical needs of stakeholders Respondents noted the benefits of addressing the practical needs of stakeholders to facilitate their participation (e.g., scheduling meetings outside of traditional working hours, providing transportation). Theme 5: Define expectations and roles Respondents identified establishing Bparameters and expectations for roles^, giving stakeholders guidance, and allowing time for stakeholders to Bget comfortable with their roles^as important tasks. Theme 6: Use in-person contact to build relationships To establish strong working relationships, respondents recommended having multiple meetings and underscored the importance of face-to-face contact. While technology and electronic interfaces were noted as facilitators of collaborations across distances, several respondents highlighted the importance of laying the foundation for partnerships through initial in-person meetings.

Contributions of Patients and Other Stakeholders
BUp to this point, what are the most significant contribution(s) made by these patients and other stakeholders?^(N=36) Theme 1: Changes to project outcomes or goals Researchers reported that engagement of patients and stakeholders led to a shift in the outcomes of interest to the project. As one respondent described, engaging patients helped the team realize that researchers' priorities Bdo not always match priorities of patients.^Another respondent noted BI can say with confidence that our project (the methods and even the project goals) have evolved, in some cases dramatically, based on our collaborations with stakeholders.T heme 2: Changes to project methods Respondents indicated that input through engagement affected their methods in a variety of ways. As one respondent noted, Bpatient/family collaborators' input was absolutely essential to developing our methods.^For example, patients and stakeholders helped researchers Bdevelop appropriate plans and processes for recruiting patients and interviewing them.^They also helped to improve the ease of data collection, which the respondent indicated may help to speed data collection. Theme 3: Enhanced access to populations or study settings Respondents reported that stakeholders facilitated access to clinical professionals and clinical settings in which studies can be conducted. Theme 4: Modifications to interventions Stakeholders' feedback was said to have influenced the development of study interventions, for example, by increasing their ease of use or the manner in which they were implemented. Theme 5: Refinement of instruments and interview questions Input into instrument development included pilot testing study instruments, giving Bstrategic adviceâ bout the length of surveys, and ensuring that the right questions are included for both quantitative and qualitative methods of data collection. As one respondent noted, Bour patient collaborators have fundamentally changed the way we ask important questions of patients and families.T heme 6: Interpretation and dissemination of results While most projects were still in the early stages, contributions of patients and stakeholders for interpreting and disseminating results were beginning to emerge. Respondents identified meaningful patient and stakeholder contributions to the interpretation of qualitative findings. One respondent also noted, Bthey have also changed the way we share study results with kids and families. I mean, they have SERIOUSLY changed things.^Multiple respondents indicated that they expect to engage stakeholders in the interpretation and dissemination of results once findings are available.
The evidence on engagement will grow if researchers systematically document more information about engagement and its impact on individual projects in publications, 1,23 if relevant medical subject headings (MESH) indexing terms for patient engagement are developed, 3 and if information on engagement is systematically collected from researchers and research partners. PCORI continues to collect and use learnings from engagement in funded projects. Future research should evaluate the impact of engagement at each stage of the research process, including the effects of engagement in interpretation of results and dissemination on the clinical uptake of findings. Studies should consider both positive impacts and potential undesirable consequences of engagement, and strategies for mitigating unintended consequences. Furthermore, future evaluations should identify which strategies for engagement-which stakeholders to involve, how to involve them, and at what stages of process-have the greatest positive impact on research quality and usefulness to patients and their caregivers and other end users of the research. Ultimately, evaluations demonstrating links between engagement and longer-term outcomes such as health decision making and health outcomes 29 are needed to develop evidence-based guidance about the best approaches to engagement in research.