Development and Application of a Patient Group Engagement Prioritization Tool for Use in Medical Product Development

Introduction Patient group engagement is increasingly used to inform the design, conduct, and dissemination of clinical trials and other medical research activities. However, the priorities of industry sponsors and patient groups differ, and there is currently no framework to help these groups identify mutually beneficial engagement activities. Methods We conducted 28 qualitative, semi-structured interviews with representatives from research sponsor organizations (n = 14) and patient groups (n = 14) to determine: (1) how representatives define benefits and investments of patient group engagement in medical product development, and (2) to refine a list of 31 predefined patient group engagement activities. Results Patient group and sponsor representatives described similar benefits: engagement activities can enhance the quality and efficiency of clinical trials by improving patient recruitment and retention, reduce costs, and help trials meet expectations of regulators and payers. All representatives indicated that investments include both dedicated staff time and expertise, and financial resources. Factors to consider when evaluating benefits and investments were also identified as were suggestions for clarifying the list of engagement activities. Discussion Using these findings, we refined the 31 engagement activities to 24 unique activities across the medical product development lifecycle. We also developed a web-based prioritization tool (https://prioritizationtool.ctti-clinicaltrials.org/) to help clinical research sponsors and patient groups identify high-priority engagement activities. Use of this tool can help sponsors and patient groups identify the engagement activities that they believe will provide the most benefit for the least investment and may lead to more meaningful and mutually beneficial partnerships in medical product development. Electronic supplementary material The online version of this article (10.1007/s43441-020-00217-0) contains supplementary material, which is available to authorized users.


Introduction
Over the past decade, patients have collaborated with researchers, funders, academia, and sponsors to inform research priorities, funding decisions, health services research, the selection of outcomes, clinical trial protocol designs, and recruitment and retention [1,2]. Because patient group engagement has significant potential to improve the clinical trial enterprise, the Clinical Trials Transformation Initiative (CTTI)-a public-private partnership co-founded by Duke University and the US Food and Drug Administration (FDA) whose members include representatives from across the clinical trials ecosystem-developed the Patient Groups and Clinical Trials project in 2013 to foster this collaboration. Initially the project sought to better understand stakeholders' perceptions regarding the importance and value of engaging patient groups and the various clinical trial services that patient groups provide. This led to the development of recommendations, best practices, and a list of specific activities for engaging patient groups throughout the clinical trial process [3]. To further advance mutually beneficial patient group engagement, the project developed a financial model to better articulate the impact that patient engagement may have on key business drivers and to demonstrate that return on investment should support broader adoption [4].
This work, along with emerging best practice resources on patient engagement in clinical trials, such as those from the Patient-Focused Medicines Development (PFMD) [5] and FasterCures [6], is helping to accelerate patient group engagement. While there is still much to understand, engaging patient groups in clinical trials is also gaining broader acceptance: for example, the Patient-Centered Outcomes Research Institute (PCORI) requires patient engagement in any of their funded clinical trials [7]; the National Academy of Medicine (NAM) recommends including patients as partners in research [8], and the US FDA has acknowledged the importance of patient involvement through a range of initiatives and guidance documents [9][10][11][12].
However, despite resources to help stakeholders understand the breadth of potential patient engagement activities and promising practices [13], there is no widely used framework or method to facilitate identifying fit-for-purpose activities that are mutually beneficial for the sponsor and for the patient group or the patient community they represent. To support this need, CTTI has developed a framework and a prioritization tool to aid both sponsors and patient groups in determining, from their perspective, (1) the benefit that patient group engagement can bring to their organizations and the clinical trial process, (2) the investment that such engagement would require, and (3) those engagements that are of highest priority to each organization. The tool supports users-both patient groups and sponsors-in identifying relevant engagement opportunities for a specific study, subjectively assessing the benefits and investments of each (low, moderate, high), and visualizing and discussing the output together as partners.
This manuscript describes the process CTTI used to gather evidence to develop the tool, describes the tool itself, and describes how the tool can be used by sponsors and patient groups to guide decisions on priority patient group engagement activities.

Evidence Gathering
Working from the CTTI Patient Group Organizational Expertise and Assets evaluation tool, we developed a list of 31 patient group engagement activities in medical product development [14]. We conducted 28 qualitative, semistructured interviews (SSIs) with representatives involved in engaging patients in medical product development from research sponsor organizations (n = 14) and patient groups (n = 14), from January 26, 2017, to April 18, 2017 (Table 1). Representatives were purposively selected [15] based on their knowledge of the types of patient group engagement activities their organization has participated in and whether their organization is actively engaged in medical product development. In addition, we purposively recruited representatives from organizations of varying sizes (e.g., based on annual budget) and organizations involved in medical product development across the clinical trial continuum (i.e., pre-discovery through post-approval). Representatives were asked to review each of the 31 CTTI patient group engagement activities [14] and consider the relative benefit of each activity. They were instructed to categorize each engagement activity as either providing a high, moderate, low, or no benefit to their constituents or company using an interactive online pile sorting platform created for this study. They were then asked to describe their rationale for their ratings. Following the "benefits" questions, representatives individually reviewed the same 31 engagement activities again, considering whether the activity would require a relatively high, moderate, low, or no investment to perform. After classifying all of the activities, participants were asked to explain their rationale for determining the investment category for the activities. We also asked the participants if any of the 31 patient group engagement activities were unclear and if so, how to refine the description of the activity.
All interviews were audio recorded and transcribed verbatim. We used applied thematic analysis to analyze the data [16]. NVivo 11 software was used to organize and code the transcripts [17]. Three analysts initially coded each of the transcripts using an apriori coding structure based on questions in the interview guide. Inter-coder reliability was assessed on 10% of the transcripts. Any discrepancies in how these codes were applied were resolved through group discussion and edits were made to the codebook to aid in future application of the codes. Next, all coded text related to the initial coding structure was reviewed for information that revealed representatives' beliefs about the benefits of and investments required for engaging patients in medical product development and also to refine the 31 activities. This information was coded and thematically organized by two trained qualitative analysts using a process of constantly comparing new information to information previously identified and coded. The data organized within the emergent thematic groups were verified by a third analyst. Finally, coding frequencies and matrices were reviewed to identify themes that were common across patient group and sponsor representatives, as well as those that were differentially expressed by certain groups, or possibly idiosyncratic. Themes were described in analytical memos, which were used to present the results below.

Benefits of Patient Group Engagement
Patient group and sponsor representatives described similar potential benefits of patient group engagement ( Table 2). Both groups suggested that patient group engagement can enhance the quality and efficiency of clinical trials by improving patient recruitment and retention, by reducing costs, and by making trials more able to meet expectations of regulators and payers. Other benefits suggested by the representatives include reducing the burden of participation by optimizing trial design and conduct, and amplifying the patient voice in medical product development, thereby improving the product's ability to more directly address patient needs. In addition, sponsor representatives indicated that patient group engagement in clinical research motivates research staff, patient groups, and ultimately trial participants (if patient groups remain engaged throughout the trial period), which helps ensure that the trial is conducted well. Respondents also noted that patient group involvement in clinical research can strengthen grassroots advocacy of clinical trials and enhance the reputation of the sponsor, trial, and product in the public sphere.

Considerations Made When Evaluating Benefits
Representatives reported that the level of benefit offered by each of the 31 CTTI patient group engagement activities was determined by subjectively assessing one or more of the following factors: Motivating Patients and Researchers to Get More Involved The benefit is that the more motivated our organization becomes with our participants, the more motivated the researchers become in their work For our own colleagues, when they've engaged with patients they feel more motivated, they feel more inspired. We believe [the effect is positive] in terms of retention of colleagues, productivity of colleagues and just the general effect on the workforce Strengthen Grassroots Advocacy of Clinical Trials I think that what we've been able to do is mobilize a grassroots effort over time that is sophisticated and knowledgeable about drug development. We've empowered tiny family foundations and we gave them the tools to advocate We bring our community volunteers from across the country to advocate for a DOD appropriation for cancer research annually That benefits our community by making sure that the FDA understands the unmet needs in our community and is kind of primed for understanding what would be the main benefits of a drug Deeper relationships with the advocacy community helps individuals feel like they've been a part of the development of a new medicine, and are therefore willing and able to champion it to ensure that it gets to people who need it Better relationship with the community has far-reaching impacts that you wouldn't necessarily anticipate. It's not just about learning how we can make existing medications better, but also potentially sparking ideas for unmet needs we weren't aware of, helping to build greater bridges between the community and how the industry is viewed as a whole I think it really makes a difference for how the patient community views the pharmaceutical industry … and that we truly are partners in disease management and not what tends to be recorded on the front page of the Wall Street Journal Enhancing the Reputation of the Sponsor, Trial No comments I think we think that engaging with patient advocacy groups brings a positive reputational benefit to [company's name]. We feel it's the right thing to do, and by putting our money where our mouth is and actually doing it, we do feel it probably helps us with reputation • The extent of the effect of the activity on the patient population or organization. For example, some patient group engagement activities could affect a large segment of the patient population or could affect several future trials. • The necessity of patient group involvement to conduct the activity. • The necessity of the activity to advance medical product development. • The ease of accomplishing the activity in the short term. • Reputational benefits gained by conducting the activity.
For example, some patient group engagement activities might be perceived by patients and other stakeholders in the community as "the right thing to do."

Investments in Patient Group Engagement
All representatives indicated that the investments required for successful patient group engagement include dedicated staff time and expertise, as well as financial resources, all of which can be impacted by the scope and longevity of the specific engagement activity (Table 3). Investments also could include the creation of new infrastructure, processes, and organizational policies to facilitate the activity. Representatives noted that some engagement activities may require additional time, effort, or burden placed directly on patients, which may be a cost that some groups are unable or unwilling to afford. Finally, a patient group representative reported that organizations may need to consider whether or not engaging in a particular activity, or associating themselves with a particular research partner, will cost them their reputation or ethical principles.

Considerations Made When Evaluating Investments
Representatives indicated that the level of investment for each engagement activity was determined by subjectively assessing one or more of the following factors: • The amount of financial resources needed to conduct the activity. For example, some patient group engagement activities might be longer-term and require continual financial investment. • The level of staff time and expertise required across the lifespan of the activity. • The amount of organizational commitment needed, given existing infrastructure. For example, some patient group engagement activities might demand a great deal of commitment from the organization to establish necessary infrastructure and processes. • The amount of direct patient involvement and potential patient burden. For example, some patient group engagement activities might necessitate interacting directly with patient populations and require a great deal of patients' time and effort. • Reputational risks posed by engaging in the activity. For example, some patient group engagement activities might pose a potentially serious risk to the reputation of the sponsor or patient group if not done well or if the partnership is perceived to violate the ethos of the organization.

Modifications to the 31 Patient Group Engagement Activities
Representatives also suggested ways to refine the original list of 31 patient group engagement activities, such as clarifying any unclear descriptions of engagement methods, combining methods that were similar, and identifying any other engagement methods they felt were missing from the original list. Suggested modifications were compiled and used to condense the list of patient group engagement activities to 24 unique activities across the medical product development lifecycle ( Fig. 1; see supplemental material for further description of each engagement activity).

Discussion
Although best practices and research for assessing patient group engagement are still evolving, this type of collaboration is recognized as having the potential to significantly improve the clinical trial enterprise [2,18]. Ensuring that collaboration is focused on areas where the greatest benefit can be achieved for everyone involved, given limited resources, is an important step in the development of strong partnerships to improve the relevance of information gathered from clinical trials.

CTTI Prioritization Tool for Sponsors and Patient Groups
We used the findings in these interviews to develop a webbased tool to help clinical research sponsors and patient groups, both individually and jointly, identify high-priority patient group engagement activities that will be most relevant to their clinical research interests and needs. The resulting "prioritization tool" supports users in identifying engagement activities that are most relevant to their situation (e.g., a particular clinical trial, or a collaboration across a development program), and provides a framework for transparent and intentional decision-making. The tool is available on the CTTI website: https ://prior itiza tiont ool. ctti-clini caltr ials.org/. The tool seeks to assist users in identifying: 1. Relevant engagement activities that would be of most value (high benefit and low investment) to pursue on their own or in partnership 2. Engagement activities that would be beneficial for their constituents but that may be too costly to invest in (high benefit and high investment activities) 3. Engagement activities that provide little direct benefit or cost to their constituents (low benefit and low investment activities) but could potentially be valuable to other strategic research partners 4. Engagement activities that are unlikely to be worth pursuing (low benefit, high investment activities).
By identifying these specific engagement activities, the user will be able to better choose which activities they would seek to gain in a new research partnership, as well as what they may have to offer potential partners. Then the partners can allocate resources to those projects that are of the most value jointly to both organizations.

Application of the Prioritization Tool
The tool walks users through completing the following 3-step decision-making process: Step one (Fig. 2) involves patient groups and research sponsors-either working together or independently-identifying relevant engagement activities. Users of the tool are provided examples of each of the 24 patient engagement activities identified by CTTI (Fig. 1) and can also choose to add their own fit-for-purpose activities.
Step two (Fig. 3) involves evaluating the relative benefits and investments associated with each activity that was identified as relevant in step one. For each engagement activity, users are instructed to assess the expected level of benefit the activities will provide to their organization or constituents and the expected level of investment it would take their organization to accomplish the activity. To help evaluate the potential level of benefit offered and investment required by a particular engagement activity, the tool suggests that users consider the factors described above. These assessments are intentionally subjective, as detailed financial or strategic modeling is often unrealistic for projects at this stage. At this time users are encouraged to add more details about how they plan to implement each engagement activity and the rationale behind their benefit and investment ratings. This information is stored for future reference and may be used when sharing the results of the prioritization tool with colleagues and potential partners.
After the evaluations are made, the results are visualized in a priority matrix, where each activity is mapped onto a 3 × 3 grid consisting of rows pertaining to the level of benefit

Reputation or Ethical Principles
We've tried not to sell our soul, and that's a very important issue for us. I'll give you an example. Years ago, we brought our board together, and the issue was a clinical trial for a drug… and the clinical trial might have placebo. We met for a day. I remember it was a very unusual board meeting. We had board members in tears as they agonized over the following question.
We had come to the conclusion that we felt that the placebo was not necessary, and that we thought that the placebo was not very good science. We didn't want to know whether their drug was better than nothing. We also felt that we did not want to put patients who have the activity is expected to achieve and columns for the level of investment required to perform the activity (Fig. 4). If desired, the user can still adjust its rating (and thus the position of the engagement activity in the matrix) either by going back to the earlier ratings or by placing the particular activity in a different cell in the matrix.
Step three involves identifying mutually beneficial activities. Research sponsors and patient groups interested in working together can compare and discuss priorities to arrive at activities that are of high value for each.
This project and the application of the tool have some limitations. First, we used purposive sampling to select participants who could provide expert experiential knowledge into the various ways patient groups are engaged in medical product development. Their opinions may be different from other patient group and sponsor representatives. Second, this tool does not provide guidance on all factors that influence how or why industry sponsors and patient groups may collaborate in medical product development. There may be a multitude of other activities or benefits or investments that could be considered that are not included in the tool or list of activities. To account for this in the tool, we have provided ways for users to enter their own list of engagement activities. In addition, the benefits and investments listed in the tool are only provided as aspects to consider as the user evaluates the value of the activities. Users are free to evaluate the relative "benefit" or "investment" based on their own understanding of these terms. Third, while revising the list of activities, we were guided by the participants' feedback on the wording and thoroughness of the list but made our own subjective judgements as to what to revise for the final list. The final list of 24 activities was not re-evaluated using a consensus-seeking process. Future research could explore the breadth and clarity of the final list of engagement activities. Finally, we have not assessed the acceptability or feasibility of the final tool, and have no knowledge of users' experience with the tool. Future research can be conducted to evaluate users' willingness to implement the tool, as well as their experience with, and the perceived helpfulness of, the tool when engaging patient groups in medical product development.

Conclusion
In summary, CTTI has previously developed a foundational set of recommendations for patient group engagement [19,20]. The recommendations address perceived barriers, including common legal and regulatory concerns, and encourage sponsors, investigators, and other stakeholders to engage with patient groups early and often for better and more efficient clinical trials and to develop meaningful partnerships and demonstrate mutual benefits [19]. The new tool helps implement these recommendations: it allows for up-front and continued collaboration by having both sponsors and patient groups define the level of expected benefit and investment when making decisions on which activities to prioritize. Important next steps may include demonstrating the usefulness of using this tool in fostering meaningful collaborations. Future work could focus on providing example cases where representatives from industry sponsors and patient groups use this tool Step 1-Identify Fit-for-Purpose Patient Engagement Activities.

3
collectively as a pair to identify and prioritize valuebased engagement activities. These case studies may provide useful real-world examples of how the tool can be implemented as well as reveal the impact of intentional industry and patient group partnerships. It is our belief that by examining the comparative value of engagement activities and deciding which activities provide the most benefit for the least investment, meaningful partnerships may be developed that will naturally foster discussions regarding expectations, goals, and specific roles in the design, conduct, and dissemination of research. Future research can evaluate if meaning partnerships do in fact result from using this tool. Ultimately, the impact of meaningful engagement will and should be measured by the resulting usefulness of the information provided by the clinical trial [1].

Funding
Funding for this work was made possible, in part, by the Food and Drug Administration through cooperative Agreement U18FD005292 and Grant R18FD005292. Views expressed in written materials or publications and by speakers and moderators do not necessarily reflect the official policies of the Department of Health and Human Services, nor does any mention of trade names, commercial practices, or organization imply endorsement by the United States Government. Partial funding was also provided by pooled membership fees from the Clinical Trials Transformation Initiative's (CTTI) member organizations.

Conflict of interest
Dr. Levitan is an employee of Janssen Research and Development, LLC. He is a stockholder in Johnson & Johnson and in a portfolio that at times includes other pharmaceutical and health care-related companies. The other co-author(s) have no conflicts of interest do disclose with respect to the research, authorship, and/or publication of this article.

Open Access
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/.