FormalPara Key Points for Decision Makers

Health preference studies are increasingly used to inform decisions across the medical product life cycle.

Research related to understanding different preference elicitation methods and method selection along with using previous research in new decision contexts were highly prioritized topics.

Researchers with a primary academic affiliation tended to prioritize methodological and/or less studied topics, whereas stakeholders from industry, consultancy, health technology assessment/regulatory, and patient organizations tended to prioritize applied research topics relating to consistency of practice.

A large majority of respondents said that an attribute reference library would be a useful contribution to the field.

1 Introduction

The value of the patient perspective in the medical product life cycle has never been more appreciated than it is at the current moment. Patients and patient advocacy groups, regulatory and heath technology assessment (HTA) bodies, and industry leaders are increasingly advocating for the use of information collected from patients to inform product and trial designs, market access, and reimbursement schedules [1,2,3,4]. The US Food and Drug Administration has approved guidance on the use of health preferences in marketing authorization [5]. The European Medicines Agency gave a favorable opinion to a framework on planning and conducting health preference studies [6], the National Institute for Health and Care Excellence has published a perspective on the use of preference data in HTA decision making [1], and the CIOMS working group XI published a report emphasizing the importance of including patient perspectives in medical product decision making [7]. This has resulted in an ever-growing field of researchers who study health preferences and an enormous growth in studies assessing what patients value in their healthcare [8]. With this interest in health preference assessment, there have also been calls to ensure that the studies are methodologically sound and produce reliable and valid information [9].

To address these issues, the IMI-PREFER project, a 6-year European public-private partnership, was launched in 2016 to inform on the use of health preference studies for decision making throughout the medical product life cycle [10]. In 2018, the IMI-PREFER project conducted a survey to identify research priorities based on expert consensus, early literature reviews, stakeholder interviews, and a ranking exercise of research topics and questions [11]. The most important research priorities identified were related to four high-level concepts: evidentiary standards, assessment of preference heterogeneity, means to minimize patient burden, and means to maximize patient understanding of concepts presented in preference studies. These were used to guide the research questions addressed in ten PREFER case studies that provided evidence to support recommendations on when and how to execute health preference studies [12,13,14,15,16,17,18,19,20,21,22].

Since this initial research call was conducted, the field of preference research has evolved as studies were published related to these topics [16, 23,24,25,26,27,28,29,30,31,32,33,34,35,36]. Thus, the objective of this exploratory study was to reassess what members of the preference research community currently think are research priorities.

2 Methods

2.1 Participants and Recruitment

Preference research stakeholders from industry, academia, consultancy, HTA/regulatory, and four patient organizations were invited to participate in a one-time online survey. Invitations were sent through e-mail distribution lists of major health preference research groups, including the PREFER Consortium (N = 134), PREFER External Advisors (N = 87), the International Society for Pharmacoeconomics and Outcomes Research-Health Preference Research Special Interest Group (N = 260), and (4) International Academy of Health Preference Research (N = 143). Participants were invited to participate in a web-based survey and agreed to provide their personal opinion. There was no renumeration for participation. Data were analyzed using a respondent id number and there was no attempt to identify individuals based on individual characteristics provided. A subgroup analysis was conducted between those who identified as primarily academic and other stakeholder groups pooled together. Respondents were informed of the nature of the survey, the expected burden, and the voluntary nature of responding. They provided consent to participate prior to being asked any questions.

Recruitment was from 1 April to 16 May, 2022. Potential respondents were sent an initial e-mail followed by two reminders. The number of participants was not capped and a minimum sample of at least 50 participants was deemed sufficient to allow for the exploration of heterogeneity based on prior research [37, 38].

2.2 Objects: Research Topic Identification

Objects used in the prioritization tasks were research topics that could increase the confidence in and acceptance of health preference research in decision making throughout the medical product life cycle by organizations and groups such as government regulators (European Medicines Agency, Food and Drug Administration), reimbursement agencies, patient groups, and industry. The research priorities in the ranking exercise were developed during the writing of the PREFER recommendations. These objects were identified in line with good research practices [39]. This process started by compiling a long list of topics from previous research agendas [11, 35, 36]. Members of the PREFER Consortium and the scientific advisory board were sent the list and asked to add any topics reflecting research topics they found relevant to the field or that were identified during the lifetime of the PREFER project. Additional topics were identified for inclusion by regulatory authorities during review of the final chapter of the PREFER recommendations discussing future research areas. Meetings were then held with PREFER Consortium members to review this full list, combine similar research topics, and remove topics that were no longer relevant in the continuously changing health preference research sphere. The final outcome was a list of 19 objects reflecting current research topics for health preference research (Table 1). Each object was given a short name followed by a more detailed description to ensure the research topic was understood uniformly by all participants. Objects were reviewed by the co-authors of the study for clarity and by nine preference researchers outside of the research team including five that were not involved in PREFER (see Acknowledgments).

Table 1 Research topics and example questions assessed in exercises
Full size table

2.3 Survey Instrument

The survey began with background questions related to the participants’ professional affiliation, familiarity with preference research, and geographic location. Respondents were then presented with the 19 methodological research topics and asked to complete two prioritization tasks. The first task consisted of classifying the 19 topics into four importance categories (‘Important question to study in future’; ‘Important but studied adequately already’; ‘Important but too complicated or impossible to study’; ‘Not important to study in future’). The second task was a best-worst scaling (BWS) 1 exercise in which participants were asked to select best (most important) and worst (least important) topics for future studies that would “increase acceptance of preference methods and their results by decision makers”. Respondents were initially presented with an example BWS-1 choice task and then asked to complete 19 BWS-1 tasks. The BWS-1 design (see Electronic Supplementary Material [ESM]) used a pre-specified balanced incomplete block design [40], and each BWS-1 task presented four objects to the participant in random order. During the choice tasks, the objects presented both the topic in bold as well as the description.

As medical products in preference tasks are often described using attributes that are applicable to multiple different treatments and disease areas, respondents were asked questions about the utility of an attribute library for reference in health preference research following the prioritization tasks (“Do you think an attribute library would be a useful contribution to the field?”). Finally, an open-text question was asked in which respondents could comment on research priorities and additional research topics. Respondents were able to navigate forward and backward within the survey.

The survey was pre-tested online by N = 9 colleagues, and other PREFER Consortium members to remove software bugs and make any final wording adjustments to the survey or instructions to improve clarity (see Acknowledgments). The final survey used can be found in the ESM. Analysis of the results was performed in R [41].

2.4 Data Analysis

As the aim of this study was exploratory and not meant to test hypotheses, only descriptive statistics were generated for participant characteristics and prioritization tasks. The initial classification task about whether a research topic was important to study in the future was analyzed by comparing the counts and proportions in each response category (not important, important, important but studied adequately already). The analysis of BWS-1 consisted of three different tabulations of choice frequencies for each research topic: the number of best selections, the number of worst selections, and the best-worst score calculated as the difference between the number of best and the number of worst selections, which incorporates differences in opinion into topic prioritization [42]. These analyses were also conducted to compare subgroup priorities between respondents who reported as being academically affiliated versus all other stakeholders. No formal significance tests were conducted to test for differences between groups as the aim of this study was exploratory.

3 Results

3.1 Participant Characteristics

Of the N = 107 participants who accessed the survey, n = 101 completed the demographics, n = 76 completed the initial ranking exercise, and n = 66 participants completed the BWS-1 choice task (Table 2). Response rates could not be calculated because individuals may have received multiple invites if they belonged to more than one professional network. The n = 34 respondents that dropped out before the BWS-1 included participants from all stakeholder groups including n = 6 from industry, n = 15 from academia, n = 3 from consultancy, n = 2 from HTA, n = 7 from patient organizations, and n = 1 other. Median completion time for survey completers was approximately 20 minutes.

Table 2 Respondent demographics (%)
Full size table

3.2 Direct Categorization of Research Topics

The results of the first categorization task are presented in Fig. 1. In all cases, the topics presented were identified as being important for future studies by a majority of respondents. The topics categorized the most as ‘important questions to study in future’ were Synthesis of Preferences Across Studies and Expressing Uncertainty in Patient Preference Studies. The topics that were most categorized as ‘important but studied adequately already’ were related to Internal Validity/Data Quality, Attribute Presentation & Framing, and Method Selection Guidance. Three topics (Transferability Across Populations or Related Diseases, Changes in Preferences Over Time, and using Psychological Constructs to Explain Preferences Across Methods) received the most categorizations of being ‘important but too complicated or impossible to study’. Using Psychological Constructs to Explain Preferences Across Methods also received the most categorizations of ‘not important to study in future’ along with Revealed Preferences - Role in Decision Making and the impact of Changing Number of Attributes.

Fig. 1
figure 1

Categorizations for research topics and questions (n = 76)

Full size image

3.3 BWS-1 Ranking of Research Topics

The results of the BWS-1 are presented in Fig. 2 and the ESM. The topics that were selected as ‘best’ the most were Transferability Across Populations or Related Diseases, Comparing Methods, Synthesis of Preferences Across Studies, and Method Selection Guidance. The topics that received the most ‘worst’ selections were Revealed Preferences - Role in Decision Making, Psychological Constructs - Explain Preferences Across Methods, Educational Materials - Digital v. Text Formats, and Changing Number of Attributes. The topics that had the highest ratio of best-worst selections were Method Selection Guidance, Changes in Preferences Over Time, Synthesis of Preferences Across Studies, Transferability Across Populations or Related Diseases, and Internal Validity/Data Quality.

Fig. 2
figure 2

Best-worst score for each research topic (n = 66). Note: Number of times selected as ‘best’ topic (blue) or ‘worst’ topic (orange) shown in bar. Topics are sorted from the highest best-worst difference to the lowest best-worst difference. The light blue line indicates an even number of ‘best’ and ‘worst’ selections

Full size image

3.4 Comparison of Respondents with an Academic Affiliation Versus Other Stakeholders

Different priorities were found in the two tasks between respondents with primary academic affiliations versus other affiliations. In the categorization exercise, other stakeholders were more likely to label a topic or question as too difficult or impossible to research compared with those working inside academia (Future Patient Preference Research: What to Prioritize Now and the ESM).

In the BWS-1, respondents working within academia tended to prioritize more methodological or less studied topics, such as Transferability and External Validity, while those working outside academia tended to prioritize more applied research topics relating to improving consistency of practice, such as Methods Selection Guidance, Internal Validity, and Synthesis of Preferences Across Studies (Fig. 3). No difference was found in the lowest five priority topics between those working in academia and other stakeholder groups (Revealed Preferences - Role in Decision Making, Educational Materials - Which Materials to Enhance, Education Materials - Digital v. Text Formats, and Psychological Constructs - Explain Preferences Across Methods, in the Lowest Five Priority Topics).

Fig. 3
figure 3

Best-worst scaling type 1 ranking by academia (left, n = 37) versus other stakeholders (right, n = 29). Note: the arrows in this figure are only intended to highlight differences in the rankings and are not intended to represent statistical testing of differences between the two groups. DCE discrete choice experiment

Full size image

3.5 Preference Attribute Libraries

Sixty-six (62%) respondents responded to the question regarding the usefulness of an attribute library. Of those, n = 52 (79%) responded ‘yes’, n = 4 (6%) responded ‘no’, and n = 10 (15%) responded ‘not sure’. N = 30 (48%) considered the attribute library as equally important or more important than the 19 research topics and questions presented in the prioritization tasks. The therapeutic areas most often given where an attribute library would be beneficial were oncology (n = 15), rare diseases and cardiovascular diseases (n = 4 each), and vaccination or infectious diseases and chronic diseases (n = 2 each). Respondents found the possibility of an attribute library important for specific benefits and risks related to commonly used endpoints or those that have been established as “gold-standard” endpoints or concepts (e.g., mortality, survival, and progression-free survival in oncology). However, respondents raised concerns about attributes being context dependent and not necessarily re-usable, or raised concerns about feasibility (e.g., how to standardize attributes and maintain the library). Two respondents referred to existing frameworks that could cover or contribute to the creation of an attribute library (a disease-specific core outcome set and EuroQoL).

3.6 Free Text Comments about Research Priorities and the Survey

N = 20 respondents answered the open-ended questions “Is there anything else you want to share about future research priorities in health preference research to increase acceptance of these methods?” N = 7 participants commented positively and found research topics included in the ranking exercise comprehensive. Respondents reported the following topics as more or equally important as the topics in the list: Account for preference heterogeneity using patient’s personal aspects, context, and other social determinants of health (five mentions); neutral entities to perform patient preference studies, which avoids potential biases and may ensure methodological rigor (two mentions); use preferences to guide endpoint selection in clinical trials (one mention); and include under-represented populations in patient preference studies (one mention). Additional remarks raised by respondents can be summarized into two main areas: establishing responsible entities for performing patient preference studies that give confidence in the robustness/validity of methods (mentioned three times); need for guidelines and best practice in patient preference studies (mentioned five times).

4 Discussion

Over the past decades, there has been increased interest in measuring health preferences to aid decision making during drug development. This has generated questions about how to assess health preferences reliably from a variety of different stakeholders (including academic researchers, industry members, consultancies, health authorities, and patient groups). This prioritization exercise was conducted to prioritize research topics for the health preference research community with the goal of increasing acceptance of health preference methods and their results by decision makers in the medical product life cycle. We identified 19 important research topics for future study that would increase acceptance of preference methods and their results by decision makers. Within these 19 topics, there were clear priorities for specific topics.

While all the research topics presented were considered important to study by a majority of the respondents, the most important research topics related to a mix of methodological and applied research topics. Two of the highest priority topics were related to the use of health preference research outside of the individual study population: either for use in other populations or for use in a meta-analysis and predicting preferences. Both of these topics were identified as important in both ranking tasks and were not listed as having been previously studied. Conducting a preference study can be a time- and resource-intensive undertaking, thus the reuse of previous health preferences to inform new or future decision making can help ensure that patient values are considered when a new study is not possible or necessary.

Five topics (Internal Validity/Data Quality, Attribute Presentation & Framing, Method Selection Guidance, Changing Number of Attributes, and Educational Materials - Which Material to Enhance?) were listed as important for future research by over half of the respondents, though over 20% of respondents felt that these topics had been researched enough previously. The topic of Method Selection Guidance was highly rated in the BWS-1 task, indicating that many respondents think it is a top priority despite 34% of respondents thinking it had already been researched adequately. The disparity between ranking these as important topics and thinking that while important it has previously been studied enough may simply be a difference of opinion. However, it may also reflect a lack of awareness of previous work in this area. Recent publications have highlighted decision criteria that can be used to guide method selection [23, 24], and previous research has been published on internal validity tests and patient comprehension [25,26,27], attribute presentation and framing [28,29,30,31], the number of attributes [32], and educational materials [16, 33, 34] thus this finding may reflect a variability in awareness of the contribution of this previous work. As the amount of methodological research available increases, there will be a need to provide consolidated and updated dissemination resources. Examples of these types of resources could be online courses and webinars, seminars, trainings hosted by professional organizations, or catalogs and repositories of published studies.

Survey participants did express an interest in a library of previously developed attributes for targeted areas, for example, oncology outcomes and outcomes frequently seen across diseases. Challenges with an attribute library include reaching a consensus on which attributes to include and the most appropriate attribute definitions, sufficient uptake, and long-term sustainability. One possible model towards an attribute library could be to follow the example of OMERACT [43], an independent organization that strives to improve endpoint outcomes through a data-driven, iterative consensus process involving relevant stakeholder groups. In addition, an intermediate step towards an attribute library could be the registration of most preference studies in a standardized manner, for example, through the Health Preference Study Technology Registry [44].

If we compare this study’s results to the previous PREFER prioritization exercise, some trends can be observed [11]. In the previous prioritization exercise, transferability of preference results both within a patient population and to other populations was highly prioritized. These topics overlap with the current study topics of Synthesis of Preferences Across Studies and Transferability Across Populations or Related Diseases, which were more highly prioritized than in the original study, indicating that they remain a topic of interest to stakeholders. The transferability of preferences information is often discussed in research articles [45,46,47,48] and is relevant to the transferability of other health economic analyses such as those conducted by HTA organizations [49]. However, there is no published guidance on how to assess the transferability of preferences or what evidence is needed to support the application of health preference information to new contexts. Additionally, the comparison of different methods for preference elicitation, study of non-discrete choice experiment methods, and the consistency of preference outcomes from different methods were highly prioritized topics in the previous exercise. In the current study, this topic remained an important topic with indications that while additional research has been done in these areas since the first prioritization exercise [23, 50,51,52,53], more research would help to understand which method to choose when conducting a preference study. Interestingly, the topic of stability of preferences over time was previously ranked as least important, but in this updated exercise it was considered the second most important research topic.

Not unexpectedly, there were differences in prioritization of health preference research based on stakeholder affiliation. Academic researchers tended to prioritize methodological and/or less studied topics, including transferability and external validity. For other stakeholders, most of them are likely to use preferences for decision making, prioritized applied research topics relating to consistency of practice, including methods selection guidance, internal validity, and synthesis of preferences across studies. Differences in prioritization of research topics most likely reflect the different needs and experience of different stakeholders. Academic researchers prioritized studying the transferability of preferences more than respondents whose primary preference work was done outside of academia. One possible explanation for this could be that academic researchers have more experience organizing, designing, or managing patient preference studies as 76% of respondents identifying as academic reported having this experience compared with 54% of other stakeholders. Another possible explanation is that while it may be more imperative in industry to keep costs for preference studies lower and reduce barriers to their use, industry-sponsored preference studies are frequently conducted to inform specific decisions, and therefore transferability might seem less relevant. In addition, regulators are an important audience for industry-sponsored preference studies, and regulatory agencies have stressed the importance of fit-for-purpose preference studies to inform decision making [54, 55]. As more evidence is generated around this topic, it may become more prioritized by other stakeholders. To ensure that the needs of a variety of stakeholders are met and to encourage a diversity of perspectives, it remains important to continue cross-sector collaborations.

Lower priority topics were similar across the groups, including questions on revealed preferences, educational materials, and psychological constructs. Lower prioritization of these topics may reflect the perception that these questions are difficult, if not impossible, to answer (Revealed Preferences) or that there are lower levels of familiarity with topics by members of the preference research community (Educational Materials and Psychological Constructs).

A strength of this study was the use of two instruments to rank the topics, allowing for a richer understanding of respondent opinions. By combining the results, we were able to understand why some topics may have been important to some while not being prioritized by others. Another strength of this study was the inclusion of the broader health preference community, including professional society preference research groups and scientific advisors beyond PREFER researchers. This differs from previous prioritization exercises in that it included a broader community. However, this study did have some limitations. One limitation of this study was that the study topics were identified based on experiences with PREFER by researchers involved with PREFER. While participants representing academia, industry, and regulatory agencies provided input, it is possible that a different sample of stakeholders may have identified somewhat different research topics. Further, the survey was sent to professional societies and mailing lists that had existing professional relationships with the study authors. The survey was not publicized outside of these networks and did not attempt to recruit preference researchers not affiliated with these professional societies. The respondents therefore reflect a convenience sample. Another limitation is the experience that the stakeholders reported in generating and using preference information. While none of the stakeholder respondents indicated that they had no experience or were unaware of what health preferences studies were before this survey, 37% of respondents indicated that they had never organized, designed, or managed health preference studies and 46% said they had not yet used the results of preference studies in their work. However, the impact of health preference research are being felt more broadly than by only those who directly conduct and apply this information, so a broader sample likely reflects the actual stakeholder community and the reach of health preference research.

5 Conclusions

There is great promise in the use of patient preference studies to inform decisions across the medical product life cycle, but more research is needed to bolster confidence in the use of these methods. Our study identified prioritized topics for future research to increase the acceptance of preference methods and their results by decision makers. We encourage preference researchers to continue contributing toward research needs as prioritized with this study and to increasing the confidence in both the robustness of preference methods and preference study results when applied to decision making across the medicine development life cycle.