FormalPara Key Points for Decision Makers

A conventional value assessment may not capture the full spectrum of value for patients.

This paper describes a practical method to engage patients in eliciting and prioritizing patient experience value elements.

A rare neurological disease, neuromyelitis optica spectrum disorder, was used as a case study.

1 Introduction

The increasing public pressure about rising drug prices highlights the importance of assessing the value of health technologies. Two main questions are raised regarding the value of new healthcare technologies: (1) what value elements are important for different stakeholders, including patients, caregivers, healthcare professionals, and payers/insurance? (2) What is a fair price premium for the added value of new technologies? A common critique of conventional value assessment is the lack of formal representation of additional value elements beyond regulatory approval endpoints such as patient-centered and societal value elements [1]. Several alternative methods have been suggested to broaden a conventional value assessment in capturing a more complete perspective on value. In recent years, multiple value frameworks have been published or updated to capture a broader list of potential value elements [2,3,4,5]. Methods to include these additional value elements in value assessment applications are emerging and include an augmented cost-effectiveness analysis and a multi-criteria decision analysis (MCDA) [6,7,8,9,10,11].

Guidance documents suggest that multiple stakeholders, including payers/decision makers, healthcare providers, academic researchers, and patients and informal caregivers should be involved to elicit the relative importance of value elements as various stakeholders have different perspectives and decision contexts [12,13,14]. Different decision contexts may exacerbate information asymmetry, for example, decision makers may not be aware of patient experiences (PEx) related to different health technologies. According to the guidance of the US Food and Drug Administration, PEx data may be collected by any persons and are intended to provide information about PEx with a disease, or the related treatment [15]. While conventional value elements derived largely from regulatory approval studies (e.g., survival, safety) are included in value assessment applications and weighed by a multi-stakeholder group (where patients and their family caregivers are also represented) [16], the relative importance of subdomains within patient-centered value should be determined by patients and their families [17].

The dearth of PEx value elements in value assessment applications is even more apparent for therapies focused on treating patients with rare diseases. Many novel technologies for rare disease, such as cell and gene therapies, are approved based on limited evidence (e.g., single-arm studies, small sample sizes) at the regulatory approval stage compared to non-rare diseases [18, 19]. Further, novel treatments can raise significant affordability concerns, even in high-income countries. The combination of data gaps and affordability concerns can lead to significant uncertainty for both patients and payers. An example case of a rare disease where the recent approval of breakthrough treatments has limited evidence and may benefit from further data on patient insights is neuromyelitis optica spectrum disorder (NMOSD) [20,21,22]. Neuromyelitis optica spectrum disorder is a rare recurrent inflammatory disorder of the central nervous system that commonly presents with recurrent attacks of optic neuritis and transverse myelitis and affects approximately 0.5–4/100,000 patients globally, making the recruitment of patients for both pivotal clinical trials and post-marketing research a challenge [23]. Neurologic injury in NMOSD is often severe, leading to blindness and/or paralysis. As a result, its impact on patients and their families, as well as society, is substantial. However, the impact of NMOSD on patients and their families is not well documented and understood. To fill gaps in the NMOSD literature on PEx, we used common MCDA weighting methods to prioritize the multiple criteria that drive treatment decisions in NMOSD and compare the relative value of PEx value elements. Specifically, the objective of this research was to implement a combined qualitative and quantitative approach to elicit and prioritize the most important PEx value elements for patients with rare disease, with NMOSD as a case study.

2 Materials and Methods

Our method for eliciting and prioritizing PEx value elements involves a three-step process: (1) collecting potential PEx value elements from existing literature sources identified in a targeted literature review, and preparing an initial list of value elements through deliberation by the multi-stakeholder research team; (2) a pre-workshop webinar and surveys to identify additional PEx value elements; and (3) a workshop to prioritize the value elements using the swing weighting method commonly used in MCDA.

2.1 Patient Experience Value Element List Development

The sources of the initial PEx value element list are shown in Fig. 1. Identification of the initial list of value elements was based on two foundation works of US value frameworks: the International Society for Pharmacoeconomics and Outcomes Research Value Flower [2, 4] and the Institute for Clinical and Economic Review Value Assessment Framework [3]. These were extended with the results of two previously published studies: a systematic literature review that aimed to collect and analyze PEx value elements in published value frameworks [24] and an original research article applying mixed methods [5]. In the paper by Inotai et al. [24], based on the systematic literature review, a multidisciplinary research team developed five potential PEx value elements that were challenged, discussed, and approved by a panel of international payer experts. The original research article by dosReis et al. [5] aimed to identify patient-informed value elements that can be used to make value assessments more patient centered. This mixed-method study, including one-on-one discussions with patients from a diverse set of disease areas, identified 42 value elements organized into five domains: short-term and long-term effects of treatment, treatment access, cost, life impact, and social impact [5].

Fig. 1
figure 1

Sources and selection of patient experience value elements. ICER Institute for Clinical and Economic Review, ISPOR International Society for Pharmacoeconomics and Outcomes Research

Full size image

Initial selection and merging overlapping value elements from the four included papers to minimize redundancy was conducted iteratively through deliberations by a multi-stakeholder research team. The team included an NMOSD clinical expert (USA), a European Patients’ Academy on Therapeutic Innovation-trained [25] non-NMOSD patient expert (Hungary), and four academic researchers (from the USA and Hungary) experienced in value framework and MCDA development, in compliance with the International Society for Pharmacoeconomics and Outcomes Research MCDA Emerging Good Practices Task Force report’s principles [26]. Value elements were then flagged as (1) conventional value elements, (2) additional PEx value elements, and (3) additional societal value elements. Description and categorization of value elements were challenged by members of the research’s Steering Committee including internationally recognized health economists. Domains of PEx value elements (as described in this paper) were ranked and weighted by patients with NMOSD in a workshop, societal value elements will be ranked and weighted by a multi-stakeholder group in a future workshop.

2.2 Case Study Population

Patients with NMOSD (aged ≥ 18 years) who were positive for serum aquaporin-4 autoantibodies and fluent in English were recruited at the University of Colorado Anschutz Medical Campus (Protocol #21-3409). Participants received a gift card for their time and effort in the research.

2.3 Pre-workshop Webinar and Online Survey

The pre-workshop webinar, embedded to an anonymous online survey (using SurveyMonkey [27]), provided a concise lay-language audio-visual summary of the rationale of value assessment of health technologies and the study objective. The main research question—what do patients consider to be valuable when selecting a treatment for NMOSD?—was made explicit to participants of the case study at every stage of the process. The introduction was followed by short audio-visual explanations for each of the nine value elements, backed up with illustrations and real-life examples to ease understanding. After watching the short video for each value element, participants were asked to (1) rate the level of understanding of the value element on a five-level discrete rating scale (where 1 is “Very poor” and 5 is “Excellent”), with the opportunity to provide suggestions or comments for the concept or wording in a free text format, (2) rate the level of importance of each value element for themselves in their treatment decisions (where 1 is “Not important at all” and 5 is “Extremely important”), with the opportunity to add any personal experience on why it is, or why it is not important for people with NMOSD and their families. Finally, (3) participants were asked whether they think the list of value elements presented cover all important aspects of a new treatment for patients with NMOSD and their families, and if not, what might be missing. This step ensured that participants had the opportunity to propose additional value elements to extend findings from the literature. During the online survey, participants had the opportunity to pause and re-watch the prerecorded short videos on the value elements and navigate forward and backward among the questions. The pre-workshop webinar and online survey were pilot tested with two patients with NMOSD (who did not participate in the research) who were then interviewed to capture their impressions and suggestions. The voting exercises of the workshop were tested with university students for technical difficulties and feasibility.

2.4 Workshop

Participants were asked to contribute to the subsequent workshop only if they watched the pre-workshop webinar and completed the anonymous online survey. The workshop was held in a hybrid form, i.e., participation was possible face-to-face (onsite) or virtually (Zoom). The first part of the workshop was a group discussion, moderated by a trained facilitator, aimed to discuss (1) the objective of the research, (2) each value element (with a special focus on those items with a fair or worse understanding in the online survey), and (3) NMOSD-specific experience of participants gathered from the survey results and the comprehensiveness of the list of value elements.

The second part of the workshop was the ranking and weighting exercise, aiming to elicit participants’ preferences on the relative importance of the PEx value elements by anonymous voting through Mentimeter [28]. Mentimeter is an interactive polling software that allows users to develop advanced ranking and weighting exercises. To test participants’ understanding on voting questions and to make sure their Mentimeter was operating properly, both ranking and weighting exercises were preceded with similar questions on an everyday topic (i.e., important value elements of a good night’s sleep). To reduce the cognitive burden of the ranking exercise, participants were asked to select the three most important value elements in the first voting. These items (ranked by the entire group) were then removed from the list and participants were asked to repeat the voting and again select the next three most important value elements from the remainder of six. Finally, in a third voting, they were asked to rank the remainder value elements. Draws were acceptable voting outcomes.

The swing weighting method conducted is a commonly used preference elicitation technique in MCDA development to weight ranked value elements [29]. Participants were asked to indicate how much more a value element is important to them compared to the one ranked just below. First, a value element ranked #8 regarding its relative importance was compared to a value element ranked #9; then a value element ranked #7 was rated relative to #8, until the value element considered the most important #1 was compared to #2. In their votes for relative importance between each value element pairs, responders could select a value on a continuous scale between 0% (i.e., equally important) and + 50% (i.e., huge difference). Relative importance (mean of differences in percent) between element pairs were then converted to weights, summing up 100% for the total of nine value elements [9]. The final weights were then presented to the participants and discussed together with the moderator of the workshop.

3 Results

3.1 Participants

Eleven patients with NMOSD agreed to participate in the case study. Ten patients completed the pre-workshop survey and watched the webinar between 9 July and 19 July, 2021, and eight patients participated in the workshop on 22 July, 2021, allowing 3–13 days to consider the value elements. One participant could not take part in the ranking and weighting part of the workshop because of the accessibility issues of the voting platform (not accessible for the severely visually impaired). Out of eight participants who completed all steps of the case study, 50% were non-white and 25% of Hispanic origin. Over 35% had a time from diagnosis greater than 15 years, and 75% identified as female. The full description of the demographics and disease background of participants is provided in Table 1.

Table 1 Descriptive characteristics of patients with neuromyelitis optica spectrum disorder involved
Full size table

3.2 List of Value Elements

Table 2 describes the short descriptions and the Appendix in the Electronic Supplementary Material (ESM) shows the additional illustrations of the nine value elements presented to research participants both throughout the online survey and the workshop. In the pilot and during the online survey and the workshop, all responders confirmed the completeness of the list of value elements (i.e., no additional elements were proposed, and no recommendations were received regarding textual phrasing).

Table 2 Description of patient experience value elements (see Appendix in the ESM for graphical illustrations designed for participants)
Full size table

3.3 Understandability and Importance of Value Elements (Quantitative Results)

In the online survey, responders found the value elements understandable (range 4.3–4.7 out of a maximum 5), which was also reflected in their minimum scores not less than 3 (“Fair”) for any of the value elements. The online survey discovered the responders’ individual perception on the importance of each element on a 5-point scale, while the workshop revealed the relative importance of the value elements (i.e., the group’s mean percent difference) in the ranking and weighting exercise according to the group’s preference. Quantitative results are summarized in Table 3.

Table 3 Individual scores on understandability and importance, and group ranking and weighting of value elements
Full size table

3.4 Participants’ Quotes on the Value Elements (Qualitative Results)

Participants were asked to reflect on each value element and their importance also in a free text format: first in the online survey, then in the group discussion part of the workshop. Table 4 contains the most illustrative participant quotes (edited for grammar, clarity, and brevity while maintaining intended content) associated with each value element from the online survey (S) and the workshop (W), respectively.

Table 4 Illustrative quotes from patients on the value elements from the online survey (S) and the workshop (W)
Full size table

3.5 Additional Themes from the Workshop Discussion

In the online survey, workshop participants addressed some additional themes not directly related to the value elements discussed. Two participants stated their preferences for value elements are different at the time of the workshop as compared with earlier stages of the disease: (1) “Those were my three most important. Maybe it’ll change years down the line.” (2) “I’m looking at it through a filter of where I’m at today in terms of my disease progression and if there’s any clinical or subclinical concerns. Whereas prior, I would have considered the filter of where my values were at the point, where I was in an acute state or recently navigating the diagnosis. I don’t know but the value shift is very dynamic.” Participants also reflected on the significance of the relative importance of the value elements: “I think this has been great. Thinking back to attributes of each element and the magnitude of difference between them. I think it's always a false choice to choose one versus the other, but I don’t think I've ever thought about the space between my top choices.” For some patients, participating in the workshop offered new connections to other patients with NMOSD and emphasized the value of personal connections with peers who are having similar PEx.

4 Discussion

4.1 Ensuring Completeness of Value Elements

We applied multiple steps to ensure the completeness of the final list of value elements, including robust literature sources, multiple iterations with stakeholders (steering committee, pilot), and providing multiple opportunities for participants to revise the list of value elements or propose new elements while (1) completing the online survey (individual feedback) and (2) sharing their experiences in the workshop (group interaction). Consequently, in the online survey, we observed high scores on understandability (Table 3), relevant NMOSD-specific examples and, during the workshop, no emergence of missing value concepts.

4.2 Consistency of Quantitative Results in the Online Survey and the Workshop

Individual preferences regarding the importance of value elements in the pre-workshop online survey and group ranking during the workshop were overall consistent. For example, the two value elements considered to be the most important by responders (Patient’s financial burden, 4.6; and Uncertainty about long-term benefits and safety of the treatment, 4.5) were also ranked in the top 3 during the workshop by the group, where participants had to make trade-off decisions during a priority setting. Interestingly, in Ranking Vote 1, 2 and 3, although value elements changed their rankings, they tended to maintain their rank groupings (i.e., being ranked within #1–3, 4–6 or 7–9).

4.3 Importance of Qualitative Data

In research on PEx, collecting quantitative evidence is desirable but may not show the full picture without supplementary qualitative data. Patients, even with the same disease area and stage, might have completely different experiences with their condition and how it is affecting their own life. Individual patient quotes may reveal additional aspects of value otherwise hidden in numbers. We therefore recommend that regardless of the decision tool (i.e., conventional value assessment, MCDA), the collection of both qualitative and quantitative data is essential.

4.4 Conducting Patient-Centered Research

A major part of the framework was ensuring that patients are engaged in a truly meaningful manner through multiple steps: (1) patient-centric design of research through the involvement of a trained patient expert; (2) piloted each step with patients with NMOSD; (3) invested time to translate terminology into lay language and to create helpful visual tools for patients; (4) allocated fair compensation for participants for their time; and (5) considered the maximum time spent with tasks requiring concentration (which may depend on the nature of the disease). Methodology ensured all participants have a chance to (1) think about the value elements thoroughly, (2) interact with others to facilitate discussion and create consensus, and (3) share an informed view on PEx value elements. We commend the National Health Council on their recommendations and confirm the use of the Patient Engagement Rubric [30] helped enrich our study.

4.5 Cognitive Burden of the Study

Concerns have been raised on the cognitive burden of certain preference elicitation methods and its effect on the reliability of results [31]. We addressed this limitation by dividing the ranking into three consecutive votes, asking only for the top three of the 9, 6 and 3 value elements, respectively. (Notably, this did not only reduce the cognitive burden of participants, but also enabled them to change their priorities reflecting the group dynamics).

4.6 Relation to Other Research on Patient Value

The specific role of patient preferences in value assessment applications is still emerging [13, 31Previous research continues to test ways of incorporating patient preferences (e.g., discrete choice experiments) into decision contexts such as early-phase clinical development, benefit-risk assessment (regulatory specific), and health technology assessments [32]. For example, Bouvy et al. suggested presenting patient preference research as a stand-alone and distinct evidence source alongside a conventional cost-effectiveness analysis, instead of attempting to incorporate those preferences into modeling efforts [33]. Overall, our exercises are quite similar in nature to such efforts in the field, with some differences that highlight the nuance of the PEx through qualitative discussions, rankings, and weighting all in one session.

4.7 Applicability in Value Assessment

Value elements proposed in this framework could be used to support deliberative decision making by standardizing and prioritizing the list of value attributes to consider if aiming to understand the impact of health technologies on patients in a holistic manner. Alternatively, value elements can be integrated within existing value assessment frameworks through an augmented cost-effectiveness analysis, by including patient-centric domains in the calculation of incremental costs or effectiveness, or using them as cost-effectiveness threshold modifiers. Finally, value elements could be incorporated into existing or future MCDA tools with an explicit decision rule to guide decision making [24]. For developers, the inclusion of these value elements in decision making would provide a signal for additional evidence generation in a holistic manner and, if benefit claims are supported by adequate scientific evidence, those efforts will be rewarded [9, 12].

4.8 Limitations

There are some limitations of this research. First, eliciting PEx elements from existing frameworks as an initial step may introduce a selection bias toward what constitutes value. Second, while sample size is challenging in rare disease, the sample size of the current case study can be considered small. The dropout of the patient with visual difficulties highlights a potential selection bias with a possible impact on results (patients’ priorities may vary with progressing disease; however, the proposed methodology may be applicable primarily in mild or moderate disease stage), as well as a lesson to be learned for future studies including patients with severe symptoms. Nevertheless, there was another patient with NMOSD participating in the program who was blind and was still able to fully take part remotely even in the voting, using assistive devices at home. Where possible in future research, increased efforts should be made to account for recruitment and accessibility issues especially in rare diseases. It should be noted, however, that in a combined qualitative and quantitative session, larger sample sizes may increase the complexity and lose the nuance of the discussion. In addition, because NMOSD is a rare disease with treatments that were approved based on approximately 40 patients per study arm, we made every attempt to ensure a broader representation in descriptive characteristics (i.e., across age, race, and disease stage); however, a small sample size did not allow assessment of the heterogeneity of quantitative data. In terms of sample size, our case study is consistent with other patient involvement efforts such as the study by dosReis et al. where the analysis included 14 patients (representing various disease areas) [5]. Regardless, we plan to expand our analysis to at least two more rare disease areas to compare the importance of value elements across rare diseases.

Third, although this study represents engagement rather than a conventional research study that seeks to extract quantitative data alone, there were limitations to the extent patients were engaged throughout the design of the case study. One patient expert has been involved from the very beginning, but ideally, a panel of patients with NMOSD could have served as an advisory board for the research team during the full process. The pilot with two patients with NMOSD could partly counterbalance this limitation.

Fourth, the discussion during the workshop did not blind participants to the ranking results between the three consecutive cycles of voting. However, this is considered a minor source of bias, as part of the objective of the moderated discussion and voting exercises was to facilitate a consensus on the worth of PEx value elements at a group or averaged level.

Finally, NMOSD has a large representation of Spanish-speaking patients in the USA. We plan to update our framework with Spanish language materials to ensure we represent all communities not only with NMOSD but in other rare diseases moving forward.

4.9 Next Steps

The research will continue aiming to test the hypothesis if patients with different rare diseases (with variations across the nature of disease and age at onset) have different preferences towards PEx value elements. Additionally, the relative importance of conventional value elements, including clinical outcomes (i.e., survival, safety, quality of life) and cost, and the aggregate weight of additional PEx (where relative importance of PEx elements are estimated per disease basis, following the method proposed in this study) and societal value elements will be determined by a multi-stakeholder group (involving patients and their family caregivers, but also payers/decision makers, healthcare providers, academic researchers). The-long term objective is to complete a value framework for health technologies for rare diseases inclusive of PEx, societal, and conventional value elements.

We demonstrated that a combined qualitative and quantitative method is feasible and efficient, even in a rare disease such as NMOSD. For example, prioritized PEx value elements may inform value assessment applications through improving deliberations on “Other Benefits and Contextual Considerations” [3]. Specifically, instead of relying on deliberation panels to perceive the additional benefits of PEx value elements, patients can prioritize and weight those PEx value elements themselves as a separate exercise during the appraisal process. Continued applications of the proposed process will produce efficiencies over time and reduce current study duration from 6 months to 2–3 months. Furthermore, the PEx value elements need not be confined to deliberation. The findings may also flow to an “impact inventory” of all PEx value elements important to patients, providing a valuable resource for future researchers in value assessments [34,35,36]. Disease-specific impact inventory tables may inform an augmented cost-effectiveness analysis and MCDA applications in addition to overall evidence generation for future health technologies.

5 Conclusions

This research provides a framework for prioritizing the most important PEx value elements with a case study in rare disease. Using the method described in this study, we have demonstrated this method to be both feasible, efficient, and acceptable by patients. The process can inform value framework applications and may facilitate PEx evidence generation in early phases of health technology development to ultimately improve PEx with relevance specifically to rare diseases.