Understanding the Influence and Impact of Stakeholder Engagement in Patient-centered Outcomes Research: a Qualitative Study

Background Engaging patients and other stakeholders as partners in research offers promise in improving the relevance and usefulness of research findings. Objective To explore the influence and impact of patient and other stakeholder engagement on the planning and conduct of comparative effectiveness research studies. Design Qualitative study with virtual, hour-long semi-structured interviews. Participants Fifty-eight researchers and fifty-one partners from a diverse purposeful sample of fifty-eight studies funded by the Patient-Centered Outcomes Research Institute (PCORI). Approach Content and thematic analysis of interview data. Key Results Described as an integral, long-term part of the research process, engagement influenced all aspects of the design and execution of studies. Partner influence was also dynamic and iterative, taking different forms over the course of the study. Across studies, we identified 387 discrete examples of influence and classified each as one of five types of influence, derived inductively from the interview data: co-producing, redirecting, refining, confirming, and limited. Most projects exhibited multiple types of influence, with 50 researchers and 41 partners reporting two or more types of influence within a project. Of the 387 examples of stakeholder influence, 306 had at least one reported impact on the study. Such impacts included changes to reflect the needs and preferences of patients or clinicians, as well as impacts on study feasibility, study quality, engagement scope or quality, and study relevance. Both researchers and partners identified multiple types of impact within projects, with 42 researchers and 38 partners reporting two or more types within a project. Because of these observable impacts, researchers and partners described engagement as worthwhile. Conclusions Findings provide insights for funders and institutions supporting engagement, measurement efforts, and clinical researchers aiming to conduct engaged research and observe similar influences and impacts in their own studies.

BACKGROUND: Engaging patients and other stakeholders as partners in research offers promise in improving the relevance and usefulness of research findings. OBJECTIVE: To explore the influence and impact of patient and other stakeholder engagement on the planning and conduct of comparative effectiveness research studies. DESIGN: Qualitative study with virtual, hour-long semistructured interviews. PARTICIPANTS: Fifty-eight researchers and fifty-one partners from a diverse purposeful sample of fifty-eight studies funded by the Patient-Centered Outcomes Research Institute (PCORI). APPROACH: Content and thematic analysis of interview data. KEY RESULTS: Described as an integral, long-term part of the research process, engagement influenced all aspects of the design and execution of studies. Partner influence was also dynamic and iterative, taking different forms over the course of the study. Across studies, we identified 387 discrete examples of influence and classified each as one of five types of influence, derived inductively from the interview data: co-producing, redirecting, refining, confirming, and limited. Most projects exhibited multiple types of influence, with 50 researchers and 41 partners reporting two or more types of influence within a project. Of the 387 examples of stakeholder influence, 306 had at least one reported impact on the study. Such impacts included changes to reflect the needs and preferences of patients or clinicians, as well as impacts on study feasibility, study quality, engagement scope or quality, and study relevance. Both researchers and partners identified multiple types of impact within projects, with 42 researchers and 38 partners reporting two or more types within a project. Because of these observable impacts, researchers and partners described engagement as worthwhile.
CONCLUSIONS: Findings provide insights for funders and institutions supporting engagement, measurement efforts, and clinical researchers aiming to conduct engaged research and observe similar influences and impacts in their own studies.

INTRODUCTION
A growing body of evidence is resolving early questions concerning the feasibility and value of engaging patients and other stakeholders in the clinical research process. Engaging stakeholders promotes inclusion and partnership with individuals who bring unique perspectives and hold a direct interest in research findings. Early studies demonstrate that stakeholders engaged as research partners have influenced study protocols and study enrollment rates. [1][2][3][4][5] These contributions by stakeholder partners have resulted in impacts to studies' acceptability, feasibility, rigor, and relevance. 6 Much of the initial discovery about how patients and other stakeholder partners impact research has been based on openended survey responses, study reports, or publications that included varying, researcher-generated, or nominal information about engagement. More detailed descriptions of how stakeholders influence the research process, the specific ways in which that influence affects the planning and conduct of a study, and the contributing factors and approaches to engagement that determine the nature of influence that stakeholders might have would strengthen this knowledge base. Such information can bring a deeper understanding of the specific benefits of patient and other stakeholder engagement in health research and can offer greater specificity to clinical researchers wishing to engage stakeholders.
To advance understanding of how engagement influences the research process, we report findings from in-depth interviews with researchers and partners. Research questions were as follows: (1) How did engagement influence the planning and conduct of studies? (2) What impacts on the study resulted from that influence? We defined influence as the contributions of partners in terms of behaviors, decisions, or events within individual studies. We defined impact as the results of those contributions on the study design and conduct. We conducted interviews with individuals from a diverse sample of studies from the Patient-Centered Outcomes Research Institute (PCORI) research portfolio. PCORI requires awardees to engage diverse stakeholders in comparative effectiveness research studies. Awardees have used a wide array of engagement practices, implemented with varied intensity ranging from input to consultation to collaboration to shared leadership. 4,6,7 Consequently, this shared context in PCORI-funded studies provides a rich opportunity to describe the influence and impact of engagement on research.

METHODS
We designed a systematic study using qualitative methods because of the need for in-depth information on influence and impact of engagement. To ensure that this work was fully informed by stakeholders, PCORI's Advisory Panel on Patient Engagement provided input throughout all stages of the study. The American Institutes for Research's Institutional Review Board (IRB) reviewed the protocol.

Sample
We used purposeful sampling to achieve heterogeneous representation of PCORI's portfolio in terms of study completion status (i.e., complete and active), funding announcement category, PCORI priority content area, study design, study populations, and health conditions. 8,9 Active studies were eligible if they had completed at least one contract year. To optimize the chance that interview participants could speak to engagement's influence, we prioritized studies whose principal investigators (PIs) reported in the PCORI-collected survey and administrative data that partner influence had occurred or that they had engaged different types of stakeholders across multiple study phases. From 301 eligible studies, we identified 80 for inclusion and emailed PIs requesting participation. We asked PIs to share contact information for study partners, whom we contacted separately to request participation.
The final sample comprised 58 studies. We conducted 109 interviews with 58 researchers (PIs and/or their designee from the research team) and 51 partners. Table 1 shows study and participant characteristics. Seven studies did not include a partner interview because of partner death, time since study conclusion, language preference, or lack of response to the request for contact information. Partner interviews represented a diversity of perspectives, with 21 (41%) identifying solely as patients or caregivers.

Data Collection
We developed semi-structured interview protocols for hourlong interviews conducted between June 2018 and January 2019 in English by phone or video. Interviews focused on eliciting examples of partners' influence during the study-including roles and responsibilities, behaviors, and contributions to decision making-and how engagement affected the design or conduct of the study. Interview protocols included questions about positive impacts to the study as well as unintended, undesirable, and negative impacts. Two teams of four experienced qualitative researchers interviewed researchers and partners, respectively. All were previously familiar with engagement in comparative effectiveness research. To prepare for interviews, interviewers studied available study publications and awardee reports to PCORI and then tailored interview protocols accordingly.

Analysis
To understand the data from different perspectives and triangulate findings, we used multiple types of analysis: coding, Coding and Memoing. All interviews were professionally transcribed and uploaded into NVivo 12 qualitative data analysis software. Five analysts coded the data using a codebook that initially included deductive codes based on the interview protocol and attributes of the interview participant and study (e.g., interview type, health condition). Analysts applied these deductive codes and added open codes to label examples of influence. Then, analysts wrote a structured memo for each study summarizing content related to the research questions and identified quotes from both researcher and partner interviews reflecting concrete influences and impacts of engagement.
Cataloging Examples of Influence and Associated Impacts.
We used content analysis techniques to identify concrete, discrete examples of influence in coded data and catalog them. 10 For each example, we developed a descriptive label reflecting the main idea (e.g., partners proposed an outcome measure, partners led recruitment activity), selected illustrative quotes, identified the study phase when it occurred, and what impact, if any, resulted. We then assigned each example to one of five types of influence stakeholders exerted, which were developed inductively from a preliminary analysis of the first 24 completed summary memos and refined using the remaining transcripts. After cataloging all examples, two analysts independently reviewed the set of examples to confirm or challenge the influence type assignments. When analysts did not agree, they met to refine definitions and reach agreement on the best fit. We also assigned each example to one or more types of impact on the study as applicable, starting with four previously identified types of impacts: study acceptability, feasibility, rigor, and relevance. 6 We used a similar process of assigning, reviewing, and coming to agreement on examples while also inductively revising the types of impact to better reflect the interview data.
Analysis and Synthesis. Analysts met weekly to discuss observations, identify thematic and content patterns in the interviews, and iteratively synthesize findings across code output, summary memos, and the catalog of examples to identify patterns of overlap and divergence, explore relationships among concepts, and identify cross-cutting themes.

RESULTS
For each research question, we identified three cross-cutting themes, which are described in the text with additional examples and illustrative quotes included in tables. Unless otherwise noted, themes were consistent across interview participant type (i.e., researchers, partners) and study characteristics (e.g., study design, PCORI content area, health condition focus).

How Did Engagement Influence the Planning and Conduct of PCORI-Funded Studies?
Across the 58 studies, we cataloged 387 discrete examples of influence-250 from researcher interviews and 137 from partner interviews. Researchers from all studies reported at least one example. Two partners could recall engagement activities generally but could not recall specific examples of influence.
Across the examples, interview participants described involvement by a variety of partners, including patients, caregivers, representatives from professional and patient and consumer advocacy organizations, clinicians, payers, healthcare administrators, and other staff at healthcare settings such as office managers or staff who work with electronic health records. The number and types of partners involved varied, with some studies including multiple types of partners and others including one or two types.
Described as an Integral, Long-term Part of the Research Process, Engagement Influenced All Aspects of the Design and Execution of Studies. Interview participants talked about engagement as part of the overall process that shaped the study design and conduct. The examples of influence spanned study phases and activities from designing the study to disseminating study findings. Table 2 highlights examples for each activity, such as co-writing the proposal, developing new survey items, enrolling study participants, and contributing to manuscripts. Partners influenced how researchers conceptualized the study, executed study tasks, and communicated about the study to both study participants and external audiences. Partners also addressed study-related challenges, such as issues with recruitment or low retention. In addition, different partners could be involved at different times during the study. For example, clinicians could be involved during data collection, while patients were involved during study design discussions or dissemination. & Limited or no influence: Influence was constrained or did not occur because researchers were not able to implement partner suggestions (e.g., due to institutional review board constraints); researchers encountered challenges balancing stakeholder influence with science best practices; researchers did not ask for or apply partner input; or partners felt they did not have much input to offer.
& Confirming: Partners validate existing plans or materials. Table 3 provides examples of each type of influence and the frequency of reports by interview participants. Table 5 highlights an illustrative quote for each type.
The number of types of influences per study ranged from one to five, with 50 researchers and 41 partners reporting two or more types of influence within a single study. Sometimes, these influences were iterative, driving a cycle of influence where redirecting could lead to later refining or confirming. For example, in one study, early redirection by stakeholders resulted in a well-planned data collection protocol that required minimal refinement prior to implementation.  Another said, "We originally anticipated approaching and trying to recruit patients in-person...But what we learned in talking with this clinic was that many of their patients are farmers that live two and three hours away." This teaching could result in any type of influence from redirecting to confirming. Researchers responded by making changes, such as to data collection procedures, even when partners were not directly involved in that part of the study. Researchers even mentioned longer-term changes, such as changing their approach to research and altering priorities for future studies.
What Impacts on the Study Resulted from that Influence?
Of the 387 influence examples, 306 had at least one identifiable impact on the study-208 from researcher interviews and 98 from partner interviews.

Researchers and Partners Reported that Engaging Patients and Other Stakeholders Had a Multi-faceted Impact on
Study Planning and Conduct. In one example, a researcher summarized the multiple impacts as follows: "The study got done. We exceeded expectations…We recruited more than we expected, we enrolled more than we expected. It was phenomenal from that frame." From the examples, we identified five types of impact, listed in order of prevalence:  Table 4 provides examples for each type and frequency of report by interview participants. Table 5 highlights an illustrative quote for each type of impact.
The number of reported impacts per project ranged from one to five, with 42 researchers and 38 partners specifying two or more types of impact within a project. Most reported impacts affected the study in ways that researchers and partners wanted. For example, partner input improved enrollment or the study's comprehensiveness. However, a few examples resulted in undesirable impacts such as increased costs or delays. In these instances, researchers described needing additional time or resources to incorporate partner suggestions.
Partners shared examples of when researchers failed to listen to or invite partner suggestions early, which resulted in the need for extensive revisions later that used up time and resources.
Not All Partner Influence Resulted in an Observable Impact on the Study. About 20% of the examples of influence did not have a reported, observable impact. These included examples where participants did not yet know the impact since the study was not complete, or partners did not know how their contributions had affected the study. Also, when influence was limited, it often resulted in no impact.

Because of Observable Impacts, Researchers and Partners
Described Engagement as Worthwhile. Researchers stated that, although engagement took time and effort, it yielded a valuable impact on the study. For example, one researcher said: "At first we viewed it as burdensome, but over time we really started to see the value in the way it was impacting the decisions we were making and how we were carrying out the study so that it would be more relevant to patients and providers on the front lines." As a result, researchers said they were more likely to seek stakeholder input on study proposals and implementation problems in the future. Partners also valued the opportunity to have an impact on studies. For example, one partner said, "This study has opened my eyes personally to how important my input is. I did not know that until I got involved with this study, how important a patient's voice is in studies."

DISCUSSION
These findings add a new level of understanding about both engagement and the dynamic nature of partnership in patientcentered research. Findings confirm earlier analyses of the benefits of patient and other stakeholder engagement in research and that active influence of partners most often improved studies' feasibility and acceptability. 2,6 Findings also offer new insights about how influence happens, show the diversity of influences that stakeholders have, and suggest additional impacts than what has been previously documented, including on engagement approaches. The ways in which partners exerted influence on studies-by redirecting, co-producing, refining, or confirming-were dynamic, non-linear, and iterative, driving a cycle of influence across the stages of a study. Although these types of influence show conceptual similarity to the continuum of engagement identified in other frameworks, the underlying concepts differ. 11 Because each study reported multiple types of influence, classifying an individual study to one type of influence-or a single place on the continuum of engagement-was not possible. Study teams could establish a "co-producing relationship" that could still have refining, redirecting, or confirming instances of influence. Similarly, study teams that initially engaged with partners as consultants could be transformed by early interactions, eventually amassing multiple examples of co-producing influence. In this way, engagement evolves throughout the lifecycle of study-weaving throughout the continuum of engagement in response to study needs. The discovery of iterative influence suggests that initial opportunities for stakeholder influence may generate future, though different, forms of influence, all of which increases the likelihood of desirable impacts on the study's overall success.
It is worth noting that not all instances of influence resulted in a desirable, observable, or known impact on the study. The few reported undesirable impacts related to unanticipated delays or increased costs. Sometimes, researchers unintentionally constrained influence by not offering an opportunity or clearly articulating a role for stakeholders. In other cases, researchers could not implement stakeholders' suggestions due to IRB constraints or other barriers. This finding adds weight to early conceptualizations of engagement that suggest that human and system factors can enable or disable partner contributions. 11-13 Institutional system adjustments and training for researchers and partners in engagement may be warranted. For example, researchers and partners may need additional support balancing input and scientific rigor in team discussions.
Many impacts identified by this study were proximal in nature. This focus is consistent with the fact that two-thirds of the studies in the sample were in progress and proximal impacts may be easiest to recognize and recall. It will be important, as a greater number of stakeholder-driven studies are completed and diffused, to also measure longer-term impacts, such as the relevance and usefulness of findings in healthcare decision making. Doing so requires having a comprehensive way of capturing proximal impacts as they are occurring. The taxonomy of types of influences and impacts from this study can be foundational to future research in this area.
Limitations and Strengths. We used a purposeful sampling approach with PCORI-funded studies and conducted interviews in English, which may limit generalizability of the findings. For each study, we interviewed one partner and one to two researchers; other people involved in the study may have had different experiences. However, across studies, we were able to identify diverse perspectives, including patients, clinicians, and other stakeholders. Because interviewers were outside of the funder, participants had greater assurance of confidentiality to share less socially biased examples.
At times, interview participants had difficulty recalling specific examples of influence and impact. Because the analysis rested on recalled examples, it may not capture the full breadth of influences or impacts. To enhance recall, interviewers prompted participants to think back to specific events or decision-making points mentioned in study progress reports and asked them to share what they remembered. Even with the limitations, these systematic, in-depth qualitative findings lay a critical foundation for continued examination of engagement's influence and impact.
Implications. For funders and institutions, these findings continue to strengthen the evidence for investing in and supporting engagement in research, including further research on how to engage effectively to achieve and amplify these types of influences and impacts. The evidence for engagement now stands at a critical juncture-moving from describing the feasibility of engagement and the possible ways engagement can influence and impact studies to recommending how stakeholders should be engaged. To do so, direction for future research necessitates understanding what engagement approaches work best for whom at different stages of the research process, as well as the longer-term impacts of engagement on the uptake of research findings and ultimately health and health care.
These findings also have implications for measuring engagement processes and outcomes. There is a lack of useful, validated measures. Early measures may be overly simplistic by classifying levels of engagement to entire studies or perceiving engagement as linear. This study suggests that measuring engagement is more complex and will need to capture the multiple types of influence occurring within individual studies and how these different types iteratively build on each other. Table 5 Illustrative quotes for types of engagement influence and impact   Type  Illustrative quote   Influence Redirecting "People had problems with [blood samples being sent away for analysis]. They didn't want to have their blood go out from their community. Based on the suggestion from …the advisory panel, we decided that we will work with the local health services where they will do the lab work." (Researcher) Co-producing "From the very first glimmer, I was involved. We sketched out what a trial would look like together and wrote the grant together." (Partner) Refining "We were missing eligible patients because of a [four-week screening] window. All these clinicians realized that and said, 'let's change the protocol.' And we did make that window bigger." (Researcher) Limited or no influence "We had an online platform that we wish we would have had a little more involvement with the development, because it was an issue with a lot of our patients using that platform." (Partner) Confirming "It was a complicated study. When we initially came up with the idea, we weren't sure that it was even going to be feasible. But after again engaging the stakeholders…we were reassured and encouraged to move forward with it." (Researcher) Impact User-centeredness and acceptability "Another challenge [the researchers had was] making this questionnaire user-friendly, not asking too invasive questions … we gave them suggestions from a patient standpoint." (Partner) Study feasibility "The survey questionnaires were coming in at a slower pace than what the researchers wanted… [After we gave our suggestions], they realized a huge difference. They mentioned something like 80, 90 percent [up] from 50 percent." (Partner) Study quality "The stakeholders gave us insights and suggestions that we probably wouldn't have come up with on our own that in the end improved the trial design, trial implementation, and, hopefully, will give us the most meaningful results." (Researcher) Engagement scope and quality "We worked together and created a patient partner guide and a handbook and a glossary … It's something that we will have to continue to refine and change to make it as user friendly as possible." (Partner) Relevance "We published an article that documented the results of our focus groups and key informant interviews with patients and caregivers, and through their [partners] assistance, we ended up getting an Altmetric score that placed us in the top 5% of all medical articles published….I attribute it to what they did." (Researcher) Finally, for clinical researchers wishing to realize influences and impacts in their own studies, these findings also offer insights. For engagement to be successful, previous research has highlighted the importance of clarifying roles and expectations for stakeholder partners and ensuring that partners understand the value of their efforts to the project. 14 These findings offer additional insights to assess the progress and success of engagement, including whether engagement is yielding the desired influences and impacts on the study. For example, if partners are only confirming or refining existing plans or materials, studies may not be realizing the full potential of engagement. Researchers may need to build in opportunities for, pay attention to, and respond to stakeholders' redirection; extend an invitation to co-execute study activities; or create opportunities for partners to lead and innovate on study tasks. Taking time to assess stakeholders' interest in influencing the study and shaping opportunities accordingly engenders trust and authentic partnership.
Taken as a whole, the findings suggest that the actions of funders and institutions to support engagement, efforts to improve measurement, and the formation of active partnerships between researchers and stakeholders are necessary to see the impacts we observed in this study in health research more broadly and ultimately move towards a culture of engagement.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.