FormalPara Key Summary Points

Why carry out this study?

The extent to which clinical outcome assessments (COAs) commonly used in Alzheimer’s disease (AD) clinical trials measure the concepts identified as those that matter to people with AD and their care partners has not previously been reported.

What did the study ask?

The study assessed the concordance between 42 important treatment-related outcomes identified in the What Matters Most study and items from 20 COAs commonly used in AD trials to assess their concordance.

What was learned from the study?

The overlap between concepts that matter to people affected by AD and individual COAs used in AD trials is somewhat limited.

Certain widely used COA items map to multiple items that matter to people affected by AD, suggesting that significant movement on these COAs reflects clinically meaningful outcomes from the perspective of the patient.

Future research should explore whether clinical trials in AD include an appropriate set of COAs reflective of the concepts that matter to individuals with AD and their care partners by stage of disease severity.

Introduction

While great strides have been made in recent years in measuring biological aspects of Alzheimer’s disease (AD), less attention has been given to measuring the clinical outcomes that matter to people living with AD and their care partners. The What Matters Most (WMM) study series is an initiative to assess and better understand treatment-related needs, preferences, and priorities of individuals with or at risk for AD and of care partners for individuals with AD.

The first phase of the WMM study sought to identify the treatment-related outcomes that are important and meaningful to people with or at risk for AD and their care partners, through qualitative research with people affected by AD [1]. The study population comprised 5 groups across the disease spectrum: individuals with unimpaired cognition and evidence of AD pathology (Group 1); individuals with mild cognitive impairment (MCI) and evidence of AD pathology (Group 2); individuals with a diagnosis of mild AD (Group 3); the care partners of individuals with a diagnosis of moderate AD, along with their care recipients, in dyads (Group 4); and the care partners of individuals with a diagnosis of severe AD (Group 5).

During the second phase of WMM, 274 people identified by clinicians as living with or at risk for AD, or those with self-reported confirmed genetic risk profile, and their care partners completed an online survey to rate the importance, on a scale of 1–5, of the 42 treatment-related outcomes identified in phase 1, thereby further supporting and confirming the qualitative findings [2]. This quantitative evaluation confirmed that all WMM items were at least moderately important to all patient groups (minimum importance rating, 3.4; maximum rating, 4.6), and most were at least moderately important to both care partner groups (minimum importance rating, 2.1; maximum rating, 4.4). WMM findings are intended to assist AD researchers in designing effective clinical outcome assessment (COA) strategies to inform patient-centric endpoints and clinical trial design, identify or develop measures to incorporate patient and care partner preferences and priorities, and generate evidence to support regulatory and reimbursement evaluations of new treatments. Phases 1 and 2 research results were strikingly consistent, supporting the rigor of the WMM findings.

To explore how well COAs commonly used in AD clinical trials reflect the treatment-related concepts of importance to patients and caregivers elicited in Phase 1 and quantified in Phase 2 of WMM, this technical assessment study was initiated. Numerous COAs are available to evaluate treatment effects in AD clinical trials. However, whether these COAs capture outcomes that patients and caregivers perceive as important has not been well studied. This is especially important in the earliest phases of the disease, when shifts in cognitive change may be less pronounced and the outcomes that matter may be more subtle and related more directly to maintenance of activities of daily life and of a sense of well-being. In conducting this study, we concur with Jutten et al. that “A crucial aspect of any clinical trial is using the right outcome measure to assess treatment efficacy. Compared to the rapidly evolved understanding and measurement of pathophysiology in preclinical and early symptomatic stages of Alzheimer’s disease (AD), relatively less progress has been made in the evolution of clinical outcome assessments (COAs) for those stages” [3]. Our research is a step in meeting this challenge.

Patient-focused drug development encourages clinical researchers to bring forward the patient voice and develop patient-centric clinical trial endpoints [4,4,6]. The objective of this study was to identify commonly used COAs and to determine the extent to which these COAs assess the concepts of greatest importance to individuals at risk for or with AD or their care partners, thereby generating evidence to support future COA strategies and incorporate patient-centric, meaningful clinical trial endpoints into future AD trials.

Methods

The technical assessment was conducted in 2 parts. First, COAs commonly used in target populations across the entire AD spectrum (preclinical, MCI, mild AD, moderate AD, severe AD) were identified through a review of the literature and ClinicalTrials.gov. For a select subset of 20 COAs used across the spectrum of AD, concepts from phase 2 of WMM were mapped to these COAs.

Identification of Existing Measures

Measures across all categories of COAs were considered, including patient-reported outcomes (PROs), clinician-reported outcomes (ClinROs), observer-reported outcomes (ObsROs), and performance outcomes (PerfOs). Existing instruments were identified in the literature through PubMed and Embase database searches conducted using a search strategy for studies published in English during the period 2014–2019 (Tables A-1 and A-2, Appendix A, Supplementary Material). The citation and abstract for each study retrieved from the searches were screened for relevance against prespecified inclusion and exclusion criteria (Table A-3, Appendix A, Supplementary Material). Full-text articles meeting inclusion and exclusion criteria were selected for a second level of review. Seminal articles published before this period, if identified from the reference lists of studies identified in the primary searches, were also included.

Independent searches were conducted in ClinicalTrials.gov to identify COA measures used in registries and interventional studies within the past 5 years. Search terms were designed to identify ongoing trials, trials that were no longer enrolling, completed or terminated trials, and trials with 2014–2019 primary completion dates (including those with and without results).

Included records from the searches of the literature databases and ClinicalTrials.gov were reviewed for information regarding study type (i.e., clinical trial or real-world observational or registry study); population (preclinical AD, MCI, mild AD, moderate AD, or severe AD) and the reporter (patient, caregiver, or clinician); and COAs used as outcome measures.

COA Selection and Concept Mapping

A subset of COAs commonly reported in the literature or deemed relevant based on team interest were selected to compare with the 42 WMM concepts. COAs for the mapping exercise were selected based on frequency and context of use in the literature to represent relevant populations across the AD spectrum, as reflected by the 5 groups participating in WMM (e.g., preclinical, MCI, mild AD); various types of measures (e.g., PerfO, ClinRO, PRO); and various domains, aspects of functioning, and contexts of use (e.g., neurocognitive assessments, memory tests, functional activities assessments, composite/functional assessments, neuropsychiatric symptom measures, and well-being measures).

Each WMM concept was tabulated and compared with items in the individual COAs by trained qualitative researchers, including subject matter experts using copies of the instrument or its manual and publicly available information (e.g., seminal papers describing the development or psychometric evaluation of the COA). Matches between WMM items and content on an individual measure were noted and coded (in green) by two researchers to document an exact match. Instances where the content in the COA [e.g., Quality of Life in Alzheimer's Disease Scale (QoL-AD) item asking “How about your living situation? How do you feel about the place you live now? Would you say it’s poor, fair, good, or excellent?”] did not adequately reflect the spirit of the WMM content (is able to live on your/their own), but assessed similar abilities/functioning were noted and coded accordingly (in yellow); COA concepts that were not included in the WMM list were also noted. Specifically, matches in content mapping from individual measures to WMM concepts were determined by how closely the COA content addressed the specific WMM concept, as identified in the in-depth interviews with 60 individuals [1] and quantified in the second phase of the WMM study with 274 individuals [2]. Multiple checks of mapping were performed by two researchers, one of whom was the primary interviewer for the WMM phase 1 study (D.D.); discrepancies were resolved in discussion between these reviewers, and overall mapping findings were adjudicated by the study team. Finally, an additional technical review was conducted by a senior team member with significant experience in AD observational and clinical trial research.

Compliance with Ethics Guidelines

RTI International’s institutional review board determined that Phase 2 of the WMM study met the criteria for exemption from full review on 30 April 2019 (RTI International’s Institutional Review Board Identification No. STUDY00020627). All respondents were informed of the study objectives, and all provided online informed consent to participate in the survey and have their responses published in summary form.

Results

Identification of Existing Measures

More than 900 records were retrieved for initial review from searches of PubMed, Embase, and ClinicalTrials.gov databases. After removal of duplicates and excluded studies, 109 records representing 107 unique studies were retained for further review. This total included clinical trials (n = 99) and real-world observational and registry studies (n = 10); 22 studies were identified in the database searches [7,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,28] (see Table B-1, Appendix B, Supplementary Material) and 87 were identified in the searches of ClinicalTrials.gov (see Table B-2).

Initial reviews of the literature and ClinicalTrials.gov yielded nearly 100 COAs across 107 unique studies in AD (Table 1). PerfO measures were the most common type of COA identified in the combined searches, followed by ClinRO and ObsRO measures. The most frequently used COAs identified in literature and ClinicalTrials.gov searches were the Alzheimer’s Disease Assessment Scale–Cognitive Subscale (ADAS-Cog), Mini-Mental State Examination (MMSE), Neuropsychiatric Inventory (NPI), Alzheimer’s Disease Cooperative Study–Activities of Daily Living Inventory (ADCS-ADL), Clinical Dementia Rating Scale (CDR) (including CDR Sum of Boxes [CDR-SB]), and Alzheimer’s Disease Cooperative Study–Clinical Global Impression of Change (ADCS-CGIC) (Table 2). Of these, only three COA measures were used across the full spectrum of disease in AD (MMSE, ADCS-ADL, and NPI).

Table 1 Summary of COAs reported in the literature (n = 109 studies)
Full size table
Table 2 Most frequently reported COAs
Full size table

Concept Mapping

The 20 COAs selected for mapping the 42 WMM concepts are used across the AD spectrum and commonly reflected in the AD literature (Table 3).

Table 3 COAs identified for concept mapping
Full size table

Alignment of COAs and WMM Concepts

Table 4 depicts a high-level overview of the mapping exercise results. Cells highlighted in green are those in which the COA item closely matches the WMM concept. The cells in yellow show where the content in the COA did not adequately reflect the spirit of the WMM concept (based on the qualitative work) but assessed similar abilities/functioning. The blank cells are those in which the COA did not contain an item measuring the WMM concept. As seen in Table 4, 10 of the 42 WMM concepts matched (coded green) to 6 or more COAs, and 7 of the 42 WMM concepts only mapped indirectly to the COAs reviewed. Detailed concept mapping results are included in Appendix C, Supplementary Material.

Table 4 Color-coded mapping of the WMM concepts by clinical outcomes assessment
Full size table

As shown in Table 4, every WMM concept is represented in one or more of the 20 COAs reviewed, even if not an exact match (coded yellow). The WMM concepts that most closely matched items in the COAs reviewed, as defined by 6 or more green codes, included Not having difficulty doing hobbies or leisure activities and Learns new information, tasks, or procedures (10 COAs each); Completing basic household chores and Remembers things on a list or reminder (8 COAs each); Remembers what someone just told you/them, Orientation to date and time, Manages money or pays bills correctly, Keeps an interest in doing things you/they enjoy, and Socializing with family and friends (7 COAs each); and Remembers words or names of familiar objects (6 COAs).

Many COAs included items not directly measuring the WMM concept per se but measuring similar abilities/functioning related to the WMM concept. As a result, 7 of the WMM concepts matched indirectly to any COAs, indicated by the concepts with yellow codes only in Table 4. Remembering why you/they walked into a room, Not repeating yourself/themselves frequently, Remembering names of people you/they have known for a long time, Recognize people you/they have known for a long time, Not putting things in obviously wrong places (e.g., a shoe in the refrigerator), Ability to stay safe (e.g., remember to turn off appliances or running water, not wandering, not being taken advantage of), and Ability to live on one’s own. The COAs that directly mapped to the most WMM concepts (defined as 10 or more) included the iADRS (20 concepts), ADCS-ADL (17 concepts), ADCS-ADL-MCI (14 concepts), ADCOMS (13 concepts), and CDR/CDR-SB (12 concepts) (Table 4). Among the candidate COAs, 2 (ADCS-CGIC and ZBI) did not directly map to any WMM concepts.

Multiple WMM concepts were represented in a more limited number of COAs (e.g., having 1–3 exact matches). Three concepts (Being able to stay safe, Remembering names of people known for a long time, Recognizing people known for a long time) had no exact matches to any COA.

Discussion

This work begins to address one of the most pressing concerns in AD research today: Do clinical trial endpoints reflect outcomes that are meaningful to patients and care partners over the continuum of AD? The What Matters Most study series includes work conducted over 4 years and was designed to begin to address this important question. In phase 1 of WMM, we conducted qualitative research with individuals at risk for or living with mild AD and care partners of individuals with more advanced AD to identify a comprehensive set of 42 AD symptoms, impacts, and treatment-related outcomes that are meaningful to individuals across the AD continuum [1]. Building on this qualitative study, Hauber et al. [2] evaluated the concepts that are meaningful to people affected by AD and quantified the importance of symptoms, impacts, and outcomes of AD to people at risk for or with AD and care partners of people with AD. As a third stage in this integrated and iterative research process, this study examined the relationship between concepts of importance identified by patients and care partners in the What Matters Most study series [2] and concepts assessed in frequently used COAs.

Among the 42 WMM concepts, 10 concepts matched to items on 6 or more COAs, and every WMM concept was represented in 1 or more of the 20 COAs reviewed, even if not always an exact or perfect conceptual equivalent. The COAs that mapped to the most WMM concepts included the iADRS (a composite score that includes items from the ADCS-ADL/ADCS-ADL-MCI identified as instrumental activities of daily living and the ADAS-Cog), ADCS-ADL, ADCS-ADL-MCI, ADCOMS (also a composite score pulling items from other existing COAs), and CDR/CDR-SB. These findings provide important and relevant content validity for this group of COAs that correlate most robustly with the WMM concepts. Notably, concepts of emotional or behavioral well-being were rated among the WMM items of highest importance [2], but were not always included in the COAs reviewed. When examining COAs that measure concepts relating to emotional or behavioral well-being, the NPI mapped most closely to WMM items.

Two COAs did not map directly to any WMM concept: the ADCS-CGIC and ZBI. Although the concepts included in the ADCS-CGIC are relevant to functioning in an AD population [e.g., memory, orientation, behavior/mood, basic and complete (instrumental) functional ability, social function], the items are broad and encompass a large range of activities/functioning in a single item score, potentially reducing the ability to show precisely how the concepts map to the items. Such broad assessments may not adequately capture subtle changes in patient functioning. Additionally, the ZBI, as a measure of the impact of AD on care partners, assesses multiple concepts that may be very important to care partners, but has a different scope from the WMM survey evaluating care partners’ perceptions of the impact on the person with AD. It is notable, however, that, in the first qualitative phase of WMM, when asked to describe the impact of their role as care partners on themselves (as opposed to the care recipient), a majority of participants in groups 4 and 5 cited impacts to their social lives (79%), leisure activities (75%), and physical health (68%) [1]—all concepts captured on the ZBI. Moreover, some of the items in the ADCS-ADL, one of the assessments with the most overlap with WMM concepts, included many concepts not included in the WMM concept list. While these concepts were not identified in the WMM studies, as with other commonly assessed items like constructional praxis, their potential clinical relevance should be further evaluated to achieve the best balance of clinical and meaningful measurement.

The numerous COAs used in AD, many of which map to WMM concepts of importance to patients and care partners, pose a challenge to researchers in selecting best-fit measures. Although the content included in the 20 COAs evaluated in this study assesses many domains important to the evaluation of effects of treatments, all COAs have strengths and weaknesses that must be considered within their context of use and with the target population and type of intervention in mind. Selected COAs must be fit for purpose within each specific context of use, and mapping across WMM is just one component of this determination. Additionally, exact concordance between COAs and concepts that matter to individuals living with AD in the community setting cannot be expected, given that outcomes evaluated in trials are driven in part by practicalities such as measurability, ability to detect change, and relevance to regulatory authorities and other decision-makers. Importantly, while the COAs evaluated in this study may include items that map directly to WMM items, these items may be embedded in a domain or total score in which other items may not link to WMM concepts, potentially limiting the sensitivity of these measures in detecting changes in WMM items. More work is needed to understand whether and how existing COAs can appropriately measure concepts that matter and consequently capture changes in these concepts across the disease spectrum. Use of companion tools and/or selection of core item sets may more comprehensively capture disease and treatment-related impacts/benefits that are most meaningful to those living with AD and their care partners. Future research is planned to explore how to group WMM concepts into subjectively determined domains, evaluate how these domains are reflected in COAs as applied and reported in AD trials, and develop a list of prioritized core outcome assessments informed by WMM findings.

Broadly, the WMM findings and the results of this technical assessment provide more pointed context for drug developers and regulators to examine the relationship between proposed clinical trial endpoints and the interests of those living with AD. Use of COAs that map most robustly to the concepts of interest will further the continued development of truly patient-centric clinical trial endpoints, and further research will continue to elucidate WMM as well as the availability and need for fit-for-purpose COAs. This research underscores the importance of including patient-centric COAs in AD clinical trials, potentially in combinations that cover most of the WMM concepts of importance. Use of specific and sensitive measures may capture subtle changes—for example, in those experienced by individuals in the early stages of AD (presymptomatic, MCI, and mild AD)—in a patient’s sense of well-being, ability to maintain a full range of activities, and ability to remain independent. Future research should explore whether clinical trials in AD, and especially those focused on individuals in the earliest stages of AD where changes in cognition or functioning may be difficult to capture, include an appropriate set of COAs. Patient-centric COA strategies should evaluate concepts that matter to individuals with AD and their care partners, including measures of depression, preservation of a sense of well-being, and maintenance of functions that support independence. While it is not anticipated that a single measure could or should incorporate all 42 concepts, it is significant that there are clear gaps between WMM and what is measured most.

To our knowledge, this is the first evaluation to rigorously compare results of qualitative and quantitative findings across the spectrum of AD to the content of the most widely used COAs. A key strength of this study is the extensive direct patient and care partner input that is reflected in the findings. Defining and reaching consensus on what constitutes a clinically meaningful treatment benefit has been recognized as one of the most important initiatives in AD research [29, 30]. Establishing with patient and care partner input the content validity of concepts being measured in AD trials is a foundational step in understanding how to achieve clinically meaningful advances in treatment [31]. Our findings complement other efforts to improve COA assessment in AD, including a recently developed framework for outcome assessment in early AD trials [3].

There are several limitations to the current study. First, the qualitative interviews that informed the WMM concepts were cross-sectional and therefore explicitly looked only at concepts that people with and at risk for AD and their care partners said mattered to them at the time of the interview and with an indefinite time horizon. The scope of this research was not to identify or classify concepts that might be most sensitive to differing phases of the AD spectrum, particularly useful in differential diagnosis, or predictive of disease progression. Additionally, although the identified studies and subsequent instrument selection were based on stringent and well-described inclusion criteria, instrument selection for concept mapping was based on frequency of inclusion in the literature. The historical trajectory of AD research overweights articles on later stages of disease, which means instruments that measure the earliest stages may be underrepresented. Searches conducted throughout the project were targeted and not intended to be a systematic review of the entire AD literature. Mapping was conducted with versions of the COAs that were available at the time of the study or, when full versions of the assessments were not available, with available papers describing the COAs. Given the ever-expanding field, new data are frequently published that could impact findings presented here. Finally, while the concept mapping was rigorously conducted, it was not completely objective, and some subjectivity was included by reviewers in the mapping decisions. Matches in content mapping from individual measures to WMM concepts were determined by how closely the COA content addressed the specific concept as identified in the in-depth interviews with 60 individuals [1] and quantified in the second phase of the WMM study with 274 individuals [2]. Determinations on whether an item was an exact conceptual match and inherently captured a WMM concept (even if the language differed) was reached by consensus among the mapping team, but subject to bias by the researcher team’s knowledge of the WMM concept intent. For example, the CDR/CDR-SB instrument documents the clinician’s report on the “memory” of the patient, which may be based on the clinician’s interview with the patient or an informant. The CDR/CDR-SB worksheets do not inquire about all aspects of memory, such as “remembers words or names of familiar objects”; therefore, this concept was marked as not an exact (but close) match for the CDR/CDR-SB. It is possible that other researchers would have reached different decisions.

Conclusions

These findings demonstrate the extent to which concepts reflected in COAs used in AD trials measure what matters to individuals and caregivers of people affected by AD, as identified in the WMM study [2]. This research, together with other WMM research to follow in its path, informs our understanding of clinical meaningfulness of future patient-centric endpoints from the perspective of the patient. It further informs our understanding of the comprehensive nature of COAs included in ongoing AD clinical trials. Findings to date from the WMM series offer an opportunity to support the selection of best-fit COAs to support the development of study endpoints to demonstrate meaningful disease- and treatment-related impacts and benefits across a range of concepts.