FormalPara Key Points for Decision Makers

This study will engage patients, family caregivers, and stakeholders throughout the value assessment using multicriteria decision analysis.

This study will assess the values of multiple sclerosis treatments based on multiple criteria that are important to patients and family caregivers.

1 Introduction

While cost-effectiveness analysis (CEA) with quality-adjusted life-year (QALY) has been widely used for assessing the value of healthcare worldwide, strictly adhering to CEA with QALY would likely prove unacceptable in the United States (US) context [1]. For instance, in their revised guidance for the Medicare Drug Price Negotiation Program, starting from 2026, the Centers for Medicare & Medicaid Services (CMS) clarified that they would not consider CEA with QALYs for initial price applicability when it lowers the value of extending the life of elderly, disabled, or terminally ill individuals [2].

It has been acknowledged that value assessments using CEA with QALY may fail to capture other important value elements [3]. Furthermore, previous CEA with QALY studies rarely considered patient preferences or engaged patients and family caregivers, who increasingly play an active role in US healthcare [4, 5]. Literature indicated that patient preferences directly impacted the outcomes of a treatment or intervention through psychological factors or indirectly via patient adherence rates [6]. Therefore, patient preferences have been increasingly investigated to inform value assessment [7]. While patient preference studies are often motivated by efforts to promote patient centricity [7], which refers to patients involved or engaged in the design and conduct of studies, eliciting patient preferences is only a form of indirect patient involvement [8]. Similarly, literature indicated that while family caregivers provided 34 billion hours of care across the US in 2017, equating to $470 billion [9] in care delivered, including family caregivers’ burden in value assessments is often scarce or oversimplified [10]. A recent article indicated that family caregivers’ voice or engagement is essential in value assessment, especially for health conditions with a greater demand for caregiving [11].

The Patient-Centered Outcomes Research Institute (PCORI) defines engagement in research as the meaningful involvement of patients, family caregivers, clinicians, and other healthcare stakeholders throughout the research process—from topic selection through design and conduct of research to dissemination of results [12]. The literature reports that patient engagement provides valuable contributions to research feasibility, acceptability, rigor, and relevance [13]. This approach supports more relevant research that better aligns with patients’ and clinicians’ real-world unmet needs and concerns [13]. However, the literature also indicates that current value assessment methods do not effectively include the patient’s perspective in the evaluation process or are not truly patient-centered value assessments, which require quantifying and integrating patient-centered outcomes and preferences along with a full spectrum of patient costs to calculate value [14].

The engagement of patients and their family caregivers in value assessment is even more critical for costly treatments with disparities in health outcomes, such as disease-modifying therapies (DMTs) for multiple sclerosis (MS), that require increased effort to better understand the role of social determinants of health (SDOH) in racial and ethnic health disparities [15]. African American individuals had a higher risk of developing MS (a 47% increased risk), and their MS tended to progress faster than White individuals [15,16,17]. They often experienced a more aggressive disease course, including more frequent relapses, worse recovery from relapses, and faster transition from relapsing MS to secondary progressive MS. Literature also found more significant disparities existed among people who suffered from disadvantaged SDOH, e.g., lower income and education [15]. Accordingly, MS affects patients living in the Deep South states, e.g., Alabama, Louisiana, and Mississippi, uniquely since these states have large African American populations (approximately 26–38% of the total population), [18],have a high proportion of people with disadvantaged SDOH (e.g., poverty, low educational attainment, substandard housing, high levels of crime and unemployment rate), and experience most significant disparities in the US in terms of healthcare access and quality (e.g., Alabama, Louisiana, and Mississippi were among the lowest five states in the US for overall health ranking) [19]. Additionally, a literature review indicated that the family caregivers of patients with MS were significantly affected by MS since they provided most of the assistance with daily activities for the patients [20]. Another literature review showed that MS influenced family caregivers’ well-being due to physical burden and psychological stress on caregivers [21]. More than half of family caregivers missed approximately 7 working days per year caring for a patient with MS [22]. Furthermore, those with disadvantaged SDOH were less likely to have paid sick leave and were more prone to financial impact due to missing work [23]. Due to the nature and impacts of MS on patients and their family caregivers, a patient-centered value assessment has emerged as an essential area of research.

Multicriteria decision analysis (MCDA) is a systematic approach to compare competing options based on key shared attributes [24]. Primarily, it involves various steps, including defining the decision problem, selecting and structuring criteria, measuring performance, scoring alternatives, weighting criteria, calculating aggregate scores, dealing with uncertainty, and reporting and examining findings [25]. MCDA has emerged as an alternative or complementary value assessment framework to address the limitations of CEA with QALY for various reasons [26, 27]. First, MCDA can account for patient preferences and various types of values, including traditional (e.g., cost and QALY) and novel (e.g., equity, innovation) value elements. Second, MCDA has a transparent decision-making process and systematic reporting format [28]. Third, decision makers can understand technology value and data gaps during the MCDA process. Fourth, MCDA outputs are easy to understand. Fifth, the underlying link, named ‘evidence-informed deliberative process’ [29], closely connects the MCDA approach with the accountability for reasonableness principle [30]. Last, the Professional Society for Health Economics and Outcomes Research (ISPOR) Value Assessment Framework Special Task Force suggested MCDA as an approach to aggregate the different dimensions of value [31]. Therefore, MCDA has been applied in various healthcare decisions, including value assessment of healthcare [32].

However, only 8% of the published MCDA studies involved patients or family caregivers as decision makers [33]. Among these studies, two recent US studies involved patient-engaged MCDA or patient-centered MCDA [34, 35]. The first study developed the patient-engaged health technology assessment (HTA) based on goal attainment scaling (GAS) principles to identify health outcomes suitable for MCDA [34]. Another study used the patient-centered MCDA roundtable convened by the National Health Council (NHC) and National Pharmaceutical Council (NPC) to conduct an educational session before MCDA was used to illustrate a pilot study to determine the value perceptions of hypothetical treatment profiles from a multistakeholder panel [35]. To our knowledge, none of the previous MCDA studies fully engaged patients and family caregivers throughout the MCDA process to assess the value of healthcare. As a result, these studies might fail to capture important health outcomes or attributes for patients (e.g., mental health burden associated with MS [36]) and their family caregivers (e.g., caregiver quality of life [37]).

The proposed study will uniquely combine patient-centered principles [38] with the MCDA framework [25, 26] for valuing MS treatments. This approach is different from some potential patient-focused methods, such as discrete choice experiment (DCE), where researchers may have predetermined treatment attributes at hand. Specifically, our final study scope will be determined by patients and family caregivers under the patient-centered engagement structure. Notably, while we acknowledge that using ‘person-centered’ may convey a more holistic view of individuals instead of confining people to a specific role (e.g., a patient, a caregiver), we follow the PCORI guidance [38] and use ‘patient-centered’ throughout. The objective of this study will be to determine the values of DMTs for MS using a patient-centered MCDA from the perspectives of patients with MS and their family caregivers living in Alabama, Louisiana, and Mississippi.

2 Methods

This study will engage patients with MS and their family caregivers throughout MCDA to assess the values of DMTs. The engagement approaches will follow the recommendations from the patient engagement best practices resource document [39], the PCORI Engagement Rubric [40], and the PCORI Engagement Awards Evaluation Reporting Tool [41]. The project also follows two ISPOR MCDA Emerging Good Practices Task Force Reports to design and complete patient-centered value assessment [25, 26]. The study proposal will be submitted for approval from Auburn University and the University of Alabama at Birmingham Institutional Review Boards.

2.1 Study Samples

This project will mainly focus on the population impacted by relapsing-remitting MS (RRMS), the most prevalent MS phenotype [42]. Specifically, we will include patients living with RRMS and family caregivers with experience caring for patients with RRMS. Patients and their family caregivers living in Alabama and Mississippi from the patient-led MS research community established in our previous PCORI-funded engagement project [43] will be invited to join this study. They were trained and had experiences to engage effectively in patient-centered outcomes research/comparative effectiveness research (PCOR/CER). We will recruit additional patients and their family caregivers from Louisiana. For patients and their family caregivers from Louisiana, we will provide additional intensive training workshops (i.e., other than the workshop mentioned later) to ensure they can also engage in patient-centered research. These patients are aged ≥ 18 years and diagnosed with RRMS. Similarly, the family caregivers are ≥ 18 years of age and are affected by a family member with RRMS. At least two-thirds of participating patients are expected to be African American or people with disadvantaged SDOH, e.g., low incomes (less than median household income in each state) and education backgrounds (lower than a college degree). Generally, the appropriate sample size in MCDA is arbitrary. A recent systematic review indicated that the number of stakeholders who participated in scoring alternatives in previous MCDA studies varied widely [33]. The median numbers of individuals were 9, 10, and 16 for regulatory decisions, clinical decision making, and priority-setting studies, respectively. This project aims to include a total of 20 patients and 20 family caregivers across the three states.

2.2 Study Design

This study will be a cross-sectional study. Figure 1 shows the overarching workflow of this study. The study team includes a patient partner from the Alabama-Mississippi-Louisiana Chapter of the National Multiple Sclerosis Society (NMSS), healthcare provider partners (i.e., a neurologist, a nurse practitioner who specializes in MS, and also a qualitative researcher), and a value assessment researcher who is a pharmacist and has MCDA experience. The study team co-created an engagement plan when developing this protocol. The plan describes how patients and other stakeholders should be engaged throughout the study, including planning, conducting, and disseminating the study results.

Fig. 1
figure 1

Overarching workflow and participants. The project team will compensate patients and family caregivers for their time, based on the rate suggested by the advisory committee throughout the project. MCDA multicriteria decision analysis, NMSS National Multiple Sclerosis Society, AL Alabama, MS Mississippi, LA Louisiana

Full size image

The study will also establish an advisory committee to guide the project and ensure high patient and family member engagement. The committee comprises three patient representatives from the MS patient-centered research community at the national level (iConquerMSTM), a healthcare provider, two healthcare payers, and an industry representative. We intend to include more patient representatives than other types of representatives to ensure a high level of patient engagement in this study. The study team will meet with the Advisory Committee every other month to present the study progressions and plans and ask for their suggestions. The study team will address or respond to the suggestions accordingly. Additionally, the study team will reinforce six PCORI Engagement Principles, including reciprocal relationships, co-learning, partnerships, transparency, honesty, and trust, when we work together and with the Advisory Committee throughout the project. This study will compensate committee members for their time, based on agreeable rates. Patients and family caregivers will also be compensated for their time, based on the rate suggested by our advisory committee.

Primarily, this study is composed of two major phases, including (1) capacity building for value assessment focusing on MCDA, and (2) value assessment of DMTs for MS using MCDA.

2.2.1 Phase 1: Capacity Building for Value Assessment Focusing on Multicriteria Decision Analysis (MCDA)

The study team will use various resources, e.g., ISPOR value assessment frameworks [4], NHC value classroom [44], and ISPOR MCDA emerging good practices [25], to develop training materials, which will include multiple value assessment topics (see examples in Table 1). MCDA for healthcare decision making will be emphasized. The study team will develop the learning outcomes to ensure that the training materials will be for the introductory level of value assessment focusing on MCDA, which is the goal of the training workshops. After we develop the training materials, we will seek inputs from the advisory committee for two rounds. For the first round, we will ask the advisory committee to review the materials independently. Each committee member will be asked to comment on the content, appropriate language used in these materials, effective delivery format, and engagement needs. The study team will consolidate the comments and use them to adjust the materials. For the second round, we will share the adjusted materials with the committee again. We will conduct a group discussion with the advisory committee during the committee’s meetings. The project team will collect the comments from the discussion to improve and finalize the training materials.

Table 1 Examples of training topics
Full size table

We will use the training materials to conduct six workshops on six different days in six months. The workshops will allow us to interact with the patients and their family caregivers, and they will be familiarized with the core concepts of value assessment. We mainly aim to introduce the overarching goal and terminology used in value assessment practices instead of training them as experts. We expect the workshops will help them recognize the importance of value assessment and understand the critical roles they can play. The format of these workshops will be similar to our previous PCORI-funded project workshops that were implemented successfully with this community [43]. For instance, we will conduct these workshops virtually. We will loan a tablet, e.g., GrandPad®, with internet access for people without a capable electronic device. We will ask the patients and family caregivers to review some materials, including short readings or videos, before they come to each workshop. Each workshop will last 2 h to prevent any burnout, since one of the major MS symptoms is fatigue. We will use the experts from the study team to deliver the workshop materials and train patients with MS and their family caregivers. We will enlist experts for specific topics and work with them to plan training workshops if necessary. We will use plain language during the workshops. We will provide many examples, including the Deep South context, which includes health disparities among African American patients with MS and patients with disadvantaged SDOH [45]. We will use a learner-centered teaching style, which actively engages participants through various activities, e.g., discussion and brainstorming. At the end of every workshop, we will evaluate their learning outcomes and ask them to evaluate the workshops. This evaluation will include reflection, an audience response system, and case discussion to field questions during the workshops.

2.2.2 Phase 2: Value Assessment of Disease-Modifying Therapies (DMTs) for Multiple Sclerosis (MS) Using MCDA

Primarily, this phase comprises three major steps: (1) instrument development (defining the decision problem, selecting and structuring criteria, measuring performance); (2) data collection and analysis (scoring and weighing criteria, calculating aggregate scores, testing uncertainty); and (3) interpretation of findings.

2.2.2.1 Instrument Development

The instrument development includes defining the decision problem, selecting and structuring criteria, and measuring performance. While describing the following processes for the patients, we will repeat these processes for the family caregivers.

Defining the decision problem: The scope of the problem will center on patient perspectives on the values of DMTs approved by the US FDA for MS. These values can be used to guide either reimbursement decisions or shared decision making between patients and healthcare providers. We will conduct a 2-hour virtual meeting to engage MS patients and seek their input in refining the project’s scope and the decision problem. We will adjust the scope and decision problem based on patient feedback accordingly.

Selecting and structuring criteria: We will use the framework for developing disease-specific patient-centered core impact sets (PC-CIS) developed by the NHC multistakeholder advisory group to guide criteria selection [46]. We will review the literature, including patient preference and disparity studies [47, 48], to generate a list of criteria and their performance ranges. These criteria will guide our semi-structured focus group interviews with MS patients. Specifically, the list of criteria will help the study team develop interview questions to guide the group discussions. Essentially, the interview questions will be open-ended, e.g., “What value components for MS treatments are important to you?” However, we may use close-ended questions, e.g., “What do you think about ‘X (a potential criterion)’ in terms of the DMT treatments?”, to confirm some criteria that were identified from the literature but not mentioned while using the open-ended questions. We anticipate 2 h for each interview. Each interview will include approximately six patients. The interview data will be transcribed and analyzed along with the findings from the literature review. We will conduct the interviews until the data become saturated. At the end of the interviews, we will ask patients to rank the importance of the identified criteria. After gathering all criteria along with ranking from patients, the project team will ask the Advisory Committee, which also includes patient representatives and other stakeholders, to review and choose the criteria based on their relevance and simplicity of assessment, understandability for patients, and applicability in the data collection process. We anticipate that these criteria will include not only the benefits and risks of DMTs but also some novel value elements, e.g., equity. We will structure these criteria in a visual manner to show the composition of the overall value of DMTs.

Measuring performance: Primarily, we will use evidence from systematic literature reviews and network meta-analyses to measure the performance of the criteria. If no evidence exists for any criterion, we will ask clinical experts from our study team and the Advisory Committee to provide their opinions. We will construct a performance matrix, showing the performance of DMTs against the criteria. This matrix will include estimates of average performance, variances of these estimates, and the data sources. At the end of this step, we will conduct a 2-hour virtual meeting to share the criteria and their performance matrix and obtain the patients’ input. We will adjust the criteria and their performance matrix based on their input.

2.2.2.2 Data Collection and Analysis

Figure 2 shows the data collection and analysis, which include scoring alternatives and weighing criteria, calculating aggregate scores, and testing uncertainty. We will conduct two in-person meetings for this step—one for the patients and another for the family caregivers. Similar to the previous step, we will describe the following processes for the patients and repeat the processes for the family caregivers. There are several approaches for scoring and weighing in MCDA. These approaches include compositional and de-compositional methods [25, 26]. While the composition methods generate separate estimates of scores and weights, the de-compositional methods derive scores and weights simultaneously. No method is clearly superior [26]. We propose using the compositional method in this study because it will likely be less complex for the patients and family caregivers; however, the Advisory Committee will be engaged in this decision. All analyses will be performed using Microsoft Excel® (Microsoft Corporation, Redmond, WA, USA).

Fig. 2
figure 2

Data collection and analysis process. DMT disease-modifying therapy

Full size image

Scoring treatment performance: We will capture patients’ priorities or preferences for changes in performance within each criterion when scoring DMT alternatives. To incorporate performance estimates into MCDA, we will use partial value functions to convert them into scores, defined on a scale of 0–100 points. Therefore, all criteria will be measured on comparable scales. We will also consider whether highor low performance is better when establishing the partial value functions. To choose the partial value function, we will ask whether each patient's value performance is linear between the two extreme ends (0 and 100). If not, we will use a bisection and difference method to develop a nonlinear partial value function. That is, we will ask each patient to identify the performance level of each criterion that is worth 50 points on the scale and then identify the score on the 0–100 scale for the midpoint on the range of performance [26]. We will repeat this process to define the shape of the value function. We will share all value functions with every patient and discuss the differences. After the discussion, each patient can adjust their functions, but we will not force consensus. We will use each patient’s value function to score DMT performance on each criterion.

Weighing criteria: There are various weighting methods with different resource requirements, the chance of bias, and complexity [49]. We proposed using a swing weighting method. The swing weight method is a relatively simple approach and requires a low level of resources. It was also used by the European Medicines Agency (EMA). For the swing weighting method, we will apply a hierarchical technique used in a previous study to explicitly account for the positive and negative impacts of the criteria [50]. We will first categorize all criteria, e.g., benefits and risks. Next, we generate weights within each category of criteria and then between categories by comparing the highest weighted criterion from each category. To do so, we will present the performance range (i.e., the swing) of each criterion to the patients and ask them individually which range is the most important to them. We will assign a weight of 100 points to that criterion and ask the patients to assign 0–100 points to other criteria to reflect the relative importance compared with the most important criterion. To validate these weights, we will ask questions such as “Would a change of a to a’ on criterion A be valued x times as much as a change from c to c’ on criterion C?” We will share individual weights and discuss the differences. After the discussion, each patient can update their proposed weights, but we will not force consensus. We will normalize the weights of positive categories (e.g., benefits) to sum to one and adjust the weights of negative categories (e.g., risks) to maintain the relative importance with the criteria of the positive categories.

Calculating aggregate scores: We will use an additive model assuming the selected criteria are independent since patients will be asked to consider redundancy or overlapping during the selection of related criteria in instrument development, and they will be asked to score treatment performance independently. We will estimate each DMT’s weighted score, which is the sum of the products of each criterion’s (normalized) mean weight and mean score. We will subtract the weighted sum of the negative criterion (e.g., risks) scores from the weighted sum of the positive criterion (e.g., benefits) scores.

Testing uncertainty: We will test for parameter uncertainty, e.g., uncertainty in the performance ranges of the criteria, by conducting both one-way and probabilistic sensitivity analyses. We will vary one value of the criteria performance at a time for the one-way sensitivity analysis. We will also perform 5000 iterations for the probabilistic sensitivity analysis. In each iteration, we will randomly select one of the patients’ value functions and weights. We will conduct scenario analyses to test for structural uncertainty (e.g., selected criteria) using different sets of criteria. We will also examine the heterogeneity in preferences among subgroups identified by our advisory committee.

2.2.2.3 Interpretation of Findings

We will present the findings of this project in tabular or graphical forms to our advisory committee and allow them to discuss how to interpret and disseminate the findings before we conduct a 2-hour virtual meeting to present and discuss the findings with the patients and family caregivers.

3 Discussion

The common approach for developing patient engagement in the value assessment framework is to briefly inform or prepare patients prior to the value assessment process. This approach has met with only limited success since it is not a true patient engagement. This protocol proposed a plan for conducting a patient-centered MCDA to assess the values of DMTs for MS in the US. Our approach is different from the common approach since we will engage patients and other stakeholders throughout the value assessment study. To achieve a high level of engagement, we include patients and other stakeholders in not only the study team but also the Advisory Committee of the study. We will comprehensively train patients and family caregivers who are the study participants with a series of value assessment workshops before we effectively engage them throughout the MCDA process. To our knowledge, they will be the first group of patients with MS and family caregivers who have opportunities to familiarize themselves with a full range of MS outcomes, including economic outcomes. After the completion of this project, they can also engage in the future patient-centered value assessment. Especially living in the Deep South states, they are more likely to experience health disparities due to various reasons, including disadvantaged SDOH and the historical mistrust of the public health system. We expect the study results and their capacity build-up can be expanded to other states, leading to equitable access to healthcare in the US.

The Institute for Clinical and Economic Review (ICER) has used CEA with QALYs to evaluate the values of DMTs for several years. Furthermore, there are more DMTs in the research pipeline. However, the perspectives of the patient and family caregivers were not well incorporated. This study aims to demonstrate whether the patient-centered MCDA is a feasible approach to assess the values of DMTs for MS treatments. We expect that this study will likely lead to improved approaches for engaging patients and family caregivers in value assessment. This approach may also capture value elements beyond QALY, e.g., equity, described by the ISPOR Special Task Force [3]. It should allow decision makers to systematically account for patients’ and family caregivers’ voices, leading to shared decision making and improved patients’ and family caregivers’ quality of life. However, our proposed approach does not intend to serve as a comprehensive value assessment tool or replace other value assessment methods, such as CEA with QALY. Instead, further research focusing on incorporating the patient-centered MCDA with other value assessment methods (such as CEA with QALY) is warranted.

There are two known limitations. First, this study includes a relatively small number of patients with MS and their family caregivers living in Alabama, Louisiana, and Mississippi since we intend to use a patient-centered MCDA, which requires a lot of resources to engage the patients and their family caregivers in the study. Therefore, the generalizability of the project findings may be limited. However, we anticipate that our participants’ characteristics, needs, and preferences should be similar to those of other patients and their family caregivers living in the Deep South states. Another known limitation is that this study includes only a limited number of criteria that are important to patients with MS and their family caregivers to control time and cognitive demands from the project participants. However, we will carefully select the criteria and fully engage the study participants in this process to minimize the impact of this limitation.