Introduction

The traditional model of evaluating treatments based primarily on primary clinical outcome measures has stumbled in its application to rare disease medical product development. The challenge of heterogeneous populations makes it difficult to select a single outcome measure that will be sensitive to changes over the short term across the study population, especially given the additional challenge that standard trial designs may not be optimal for small numbers of patients [1,2,3]. In addition, the practice of prioritizing clinical endpoints familiar to regulators means that endpoints may be based on other diseases and may not be as relevant or sensitive to the rare disease population being studied.

An estimated 30 million people in the USA and 350 million people worldwide are living with a rare disease [4, 5]. There are currently 7000–8000 known rare diseases, and since fewer than 5% of them have a treatment [5, 6], there is a considerable need for the development of effective drugs. This unmet need combined with rare disease study design challenges make it important that rare disease clinical trials consider including a creative and pragmatic mechanism to supplement traditional clinical outcome assessment tools for capturing rigorous information about potential treatment effects from the perspective of the individual patient [3].

Measuring treatment effect in rare disease presents many methodological challenges [1, 7]. As a result of the small number of patients and the nature of rare diseases, study design often involves a tension between properly powering the study and minimizing the heterogeneity of the study population [8,9,10]. Broadening the inclusion criteria to power the study can result in increased heterogeneity, which then makes it difficult to select a single outcome measure with sensitivity across the study population [3]. Furthermore, the diversity of symptoms in any given rare disease, and their tendency to affect many organ systems, may result in a wide variety of manifestations and progression [1]. This diversity, combined with the novelty of the drug being assessed, may mean that it is not always possible to predict the different ways treatment benefit might manifest across patients, further complicating the selection of a single primary outcome measure [11]. Lastly, there is often a lack of validated, disease-specific outcome measures, which leads to borrowing outcome measures that were validated for other diseases. In addition to the potential lack of sensitivity, borrowed outcome measures may not reflect the most relevant or important patient experiences during the trial [12, 13] or take into account other comorbidities that can confound the assessment of the disease [14, 15]. These methodological challenges in rare disease clinical trials can lead to uncertainty about whether some trials fail as a result of a failed treatment or failed measurement of the drug effect. Type 2 errors, when a beneficial effect is not identified when in fact the effect is valid, are not usually addressed in regulatory approval decision-making [16].

While the “Patient-Focused Drug Development” section of the 21st Century Cures Act is an important first step that emphasizes the need to consider patient experiences in drug reviews [17], current regulatory requirements for drug approval do not require that patient experience data be considered as part of the evidence. The US Food and Drug Administration (FDA) enforces strict requirements for the quantum of clinical trial-derived evidence to establish that new treatments are safe and effective [18]. For a marketing application to support approval, the FDA must determine that there is substantial evidence of effectiveness, typically interpreted as at least two adequate and well-controlled trials [19]. Sufficient evidence of effectiveness is typically defined as statistical significance (at a 0.05 significance level) on a single or pair of primary endpoints, which means researchers may miss clinically important information about the impact of treatment on patients not captured using those measures. Particularly in rare diseases, clinically meaningful findings may be more important to patients and caregivers than statistically significant findings. Recognizing this over the decades, the FDA has consistently demonstrated extraordinarily reasonable flexibility in its review of certain applications for orphan drugs [20, 21]. Yet, with a recent series of rare disease study failures, some rare disease advocates have been critical of outcome measure selection that does not adequately take patient and family experience into account [22, 23]. While regulators continue to take steps to recognize the importance of incorporating patient experience data into their decision-making processes, there is a lack of transparency on the factors that influence orphan drug reimbursement and how they are weighted [24,25,26,27,28,29,30].

Particularly in rare diseases, regulatory approval does not align with payer drug reimbursement or patient access [31, 32]. In addition, the structure of both the public and private payer review system does not allow for the input of patients and caregivers. US Medicaid programs determine medication coverage through Drug Utilization Review (DUR) boards, where public comment is the only mechanism for patients to provide input [33], and some states only allow the public to reference published, peer-reviewed literature during their comments [34]. There is no requirement for weight to be given to public comments in the coverage decision-making process [35]. Since only formal evidence is reviewed, it is important to collect patient experience data and publish it as part of the trial. Private payers make decisions on a patient-by-patient basis, and appeals are conducted solely with input from the prescribing physician and not the patient. Patients should have direct involvement in coverage decision-making and in the development of the coverage criteria [26, 30]. The European Patients’ Academy (EUPATI) recently released guidance that emphasizes the importance of including patient experience data in Health Technology Assessment (HTA) and summarizing how it was used in reaching the final recommendation [36]. In addition to coverage decision-making, patient experience data may educate healthcare providers on the range of the patient population that benefits from a treatment. Providing a mechanism for rigorously captured and detailed data on the patient experience can help both payers and prescribers acquire the specialized expertise required to make informed decisions on prescription and coverage for rare disease treatments.

Incorporating rigorously captured data on patient experience from the individual patient’s perspective as part of a clinical trial program will help clinical trials to measure perceptions of change that inform the meaningfulness of trial outcomes and relevance to patients’ daily lives. The 21st Century Cures Act emphasized the need for patient-focused drug development (PFDD) and required the FDA to release a series of new guidance documents on the collection and use of patient experience data [17]. The FDA, EUPATI, and the Clinical Trials Transformation Initiative (CTTI) have released guidance to encourage patient involvement throughout the entire drug research and development process [37,38,39]. The FDA guidance includes a recommendation to use mixed methods in clinical trials to capture the patient experience qualitatively as well as quantitatively, and it provides researchers with recommendations for identifying what is important to patients [38, 40,41,42]. Engaging patients in drug development may lead to improved measurement of appropriate and meaningful outcomes. At a PFDD meeting for idiopathic pulmonary fibrosis, patients highlighted coughing as central to the patient experience, and yet traditional outcome measures for trials in idiopathic pulmonary fibrosis do not measure coughing, and it was not recognized as an important outcome in a Delphi panel of 254 international medical experts [11, 43, 44]. This insight from patients, and the lack of its inclusion into recognized outcome measures, underscores the importance of incorporating the patient voice in drug development.

Negative or ambiguous clinical trial results may leave regulators wondering whether the failure was due to the drug’s lack of efficacy or to methodological flaws. Qualitative interview data may help to explain the trial results and document patients’ experiences during the trial that may have been missed by traditional endpoints. The FDA was reluctant to approve eteplirsen, a drug developed for Duchenne muscular dystrophy, a rare neuromuscular disease, based on the small sample size of the clinical trial, use of historical control data, and ambiguous results [45, 46]. After the completion of the trial, The Jett Foundation, a not-for-profit patient advocacy organization, conducted a qualitative study with a convenience sample of eteplirsen trial participants and their caregivers. Patients and caregivers reported the positive impact of eteplirsen on their function and daily life, and The Jett Foundation presented the results at the FDA Peripheral and Central Nervous System Drug Advisory Committee meeting [47, 48]. Although the committee did not recommend the drug be approved, the patient experience data presentation may have contributed to accelerated approval. Since the qualitative data was not captured prospectively or as part of the trial protocol conducted by the sponsor, it was challenging to include it in the evidence submitted to regulatory authorities for consideration. Collecting qualitative interview data as part of a trial and with rigor allows for its inclusion in the package of evidence that regulatory decision-makers review.

In a phase 2 clinical trial of telotristat etiprate, a drug developed for patients with carcinoid syndrome, researchers conducted retrospective interviews with the clinical trial participants to identify meaningful changes they experienced in their symptoms and to define the degree of clinically meaningful change [49]. Patients identified reducing bowel movement frequency as their highest priority, and results supported a 30% reduction in bowel movements as a clinically meaningful change. The results of the qualitative interviews were used in conversations with the FDA about clinically meaningful change and selection of a primary endpoint for the phase 3 trial [49]. The phase 3 clinical trial of telotristat ethyl (formulated as telotristat etiprate) among patients with carcinoid syndrome showed that the drug significantly reduced bowel movement frequency [50], and exit interviews with patients at the end of the trial revealed that patients taking the drug experienced clinically meaningful reductions in bowel movement frequency [51]. While the quantitative results demonstrated a statistically significant reduction in bowel movement frequency, it was important for the researchers and regulators to understand whether the reduction in bowel movement frequency was clinically meaningful. Telotristat ethyl was ultimately approved by the FDA [52].

Understanding all meaningful changes that patients experience, including those only experienced by a small subset of the population, helps develop a fuller picture of the treatment effects on a rare disease population, can inform the development of future outcome measures, and broadens the population that may benefit from effective treatments. The objective of this paper is to describe the use of the Patient and Caregiver Perception of Change (PPC and CPC) Assessments, a methodology utilizing patient and caregiver video interviews to complement the data captured by traditional endpoints in rare disease clinical trials.

Methods

Study Design

PPC and CPC Assessment methodology would optimally include qualitative interviews conducted before, during, and after a clinical trial [53]. To inform the interview guide questions that will be used in the patient and caregiver video interviews, preliminary unstructured qualitative interviews are conducted with patients and caregivers in a rare disease population before initiating a study, which aids in the selection and development of appropriate outcome measures and interview guide questions. Including only patients and caregivers who will not be enrolling in the clinical trial where the video interviews will be implemented is important to avoid biasing the results of the clinical trial.

The goal of preliminary unstructured interviews before a study is to understand the symptoms of the condition, natural history, impact on patient function and quality of life, and patient input on the outcomes that are important and relevant to them. When recruiting patients and caregivers for the preliminary unstructured interviews, it is important to aim for diversity in disease severity, age, and any other factors that may be relevant. The interviewer starts with interviews of advocacy group leaders and asks advocacy groups to help recruit patients and caregivers for the preliminary interviews. The interviewer continues to conduct interviews until saturation is reached. Maximizing the quality of the information obtained through the qualitative patient and caregiver video interviews conducted via mobile application requires an in-depth understanding of how patients experience their condition.

Conducting qualitative patient and caregiver video interviews at baseline, during, and after a clinical trial gathers data on patient experiences with the drug and control. These interviews can inform future studies in addition to the primary goal of capturing any changes in symptoms, disease trajectory, patient function, and quality of life that patients experienced during the trial, especially those that were most noteworthy from the patient or caregiver perspective. Optimally, these interviews would be conducted prospectively, implemented both at baseline and throughout the study to explore patient experiences before, during, and after the trial, but retrospective interviews conducted at the end of the study can still offer valuable information on the patient experience.

When implemented prospectively, the baseline interview covers the manifestation of symptoms for that particular patient as well as captures information about the patient’s current function at the start of the study. Subsequent interviews are conducted to capture any potential change in function as well as to establish the potential durability of the response. If the interviews are added retrospectively after the study’s initiation, it is important to select the timing of the interview to capture any upcoming changes in patient status based on the study design (crossover, washout period, open label extension, etc.). As with other assessments of treatment effect, informing selection of both prospective and retrospective timepoints with pharmacokinetic studies helps to ensure that patients have been receiving the potential therapy for a clinically relevant amount of time.

Depending on the sponsor’s current progress in study design and startup, qualitative interviews can be integrated into the clinical trial or be completed as part of a separate but parallel study only open to participants of the original study. Conducting interviews as a separate study offers the operational simplicity of adding an interview component after the main study’s initiation, but risks a lower participation rate because the interviews are optional. Conducting this research as a separate study requires support for the project from study staff. Integrating qualitative research into the original study guarantees full participation, but requires that the interview study documents are ready early in the study planning process and are incorporated into the clinical trial’s protocol and implementation plan.

Video Capture

Qualitative interviews can be done in the clinic, over the phone, or with a web conferencing system, but the use of video capture to record qualitative interviews allows patients to visually demonstrate a treatment impact. For the purposes of this paper, we will refer to video capture through a mobile application, but video interviews can be captured through other methods. The use of a mobile application that records interviews with a full audit trail allows patients to record interviews at their own convenience but still within a set schedule, which may be an optimal solution for patients who are already burdened by clinical trial participation. Conducting interviews in the home environment roots patients in their daily lives, unlike interviews conducted in the clinic, which can be biased by encounters with other patients, clinic or travel fatigue, or performance on other trial measures.

Interview Guide Development

The process of building the interview guides required to assess PPC and CPC begins with a comprehensive disease literature review to gain a deeper understanding of what is known about the relevant symptoms and disease progression. After the disease review, researchers conduct preliminary unstructured interviews with patients and caregivers who are not part of the clinical program to key in on both the breadth and relative impact of symptoms that patients experience, the nuances of the symptoms that are disease-specific, and the ways patients and caregivers track the progression of the disease over the short term. The interviewer, trained in qualitative research, listens for aspects of the disease that may not be captured or may be only partially captured by the outcome measures being used in the clinical studies. The same interviewer should be used to conduct all interviews within a study to ensure uniformity. When studying novel treatments, seeking out any available patients in the community who have experienced spontaneous recovery or are on alternative treatments allows the researcher to gain insight about how treatment benefit may appear in clinical trial subjects and how patients evaluate that benefit.

The combination of the disease review and the preliminary unstructured interviews informs the development of the semi-structured video interview guide questions. After the guide is developed, the same patient advocacy collaborator, clinicians, patients, and/or caregivers that were interviewed during the preliminary unstructured interviews can review the guide to ensure that the questions are being asked clearly and that they cover the relevant domains. Figure 1 displays the inputs that inform the measurement of patient experience during a trial and the impact of that information after the trial. The semi-structured interview guide focuses on meaningful changes that occurred during the study, what specifically made the patient or caregiver notice those changes, and how those changes impacted daily life.

Fig. 1
figure 1

Inputs into and impacts from Patient and Caregiver Perception of Change Assessments during a clinical trial

Full size image

Many patient-reported outcome (PRO) questionnaires break down patient experiences into categories like “somewhat” and “frequently” or “moderately severe” and “severe.” These categories can present a problem for patients because they may experience change but not enough change to move from one category to the next, missing the level of change within each category that the patient may view as clinically meaningful or the context around why a patient chose a particular category. In addition, a patient’s sense of what a certain level of frequency or severity means may change over the timeframe of the study, leading to answers that may not reflect true change in disease progression, or it can be influenced by outside factors like comparisons to other patients in the clinic. Questionnaires can induce respondent fatigue in a way that can be avoided by the PPC Assessment’s approach of engaging patients in telling their story [54, 55]. The interview guide provides structure to that story and can include questions that add more context to the traditional endpoint data captured in the clinical trial, such as confounding factors that could have influenced functioning on certain performance measures or self-rated assessments of symptoms during the trial.

Study Population

The study population for the PPC Assessments includes patients in the trial, and when applicable, can be expanded to capture CPC and Observer Perception of Change (OPC) Assessments. These assessments should be added to studies where patients have people close enough to be able to comment on the patient’s day-to-day experiences. CPC and OPC Assessments are included because caregivers and observers may notice pertinent details from their perspective that a patient may not. As a result of their age or manifestations of their condition or a comorbidity (e.g., cognitive disability), patients may be unable to participate fully or at all in the video interviews; in such situations, caregivers are the primary or sole interview source. For the purposes of this paper, we will refer to the caregiver/observer interviews as “caregiver interviews” for simplicity, but depending on the disease, these interviews could be conducted with spouses, children, teachers, healthcare professionals, or any other person in the patient’s life that might have relevant observations.

Recruitment and Screening

If the interviews are integrated into the clinical trial, recruitment and screening is integrated into the study protocol. The interview study team can ask trial participants if they have a caregiver that could be invited to complete an interview as well. If the interviews are being conducted as an ancillary study, the clinical trial sites help the interview study to recruit the trial participants. The clinical trial study staff obtain patient consent to share their contact information securely with the interview study team so that they can learn more about participating in an interview. Recruitment can also be done through advocacy groups to supplement the recruitment conducted by the trial sites. When advocacy groups highlight the importance of including the patient and caregiver voice in a clinical trial, their support can be instrumental in maximizing patient and caregiver interest in the study. Advocacy groups can reach out to the general population of patients and caregivers by advertising the study on their website, group email lists, and/or social media.

The interview study team can use the contact information provided to them to contact interested patients and caregivers, which often results in greater participation than relying on patients to contact the interview staff. Once the interview study team explains the study to the patients and caregivers, and answers any questions they may have, they are taken through the informed consent process with the interview study staff over the phone. Verbal consent is obtained before determining their eligibility and inviting them to register for the study mobile application. In addition, electronic consent is obtained during the study mobile application registration process. Figure 2 provides an overview of the process of conducting the interviews through the mobile application.

Fig. 2
figure 2

Process of conducting the interviews using the secure mobile application

Full size image

Interview Procedures

In order for subjects in the clinical trial to use the study mobile application to record the interviews, study staff first send an email to invite patients and caregivers to register and download the study mobile application. Patients and caregivers click on a link, choose a password, and complete the electronic consent as part of their registration. Study staff also email the interview guide at this time. Study staff can work individually with any participant that needs assistance with the technology.

During study design, the interview study team determines when the interviews will be completed and how long the patients will have to complete each interview, which is determined on the basis of the number of questions in the interview guide and the specific burden of disease. The mobile application automatically tracks when the data window opens and sends patients and caregivers automated reminders, both through email and with push notifications on the phone. Participants may either select an interview partner to read the questions and record the interview, or record the interview themselves by using the smartphone’s self-facing camera (i.e., selfie mode). Participants can do their interview from anywhere at any time.

The mobile application includes the ability to present an orientation video that introduces the study and explains the interview recording procedures. The video capture mechanism is not unlocked until the patient and caregiver, if applicable, have watched the orientation video. Dividing the interview into modules that consist of no more than five or six questions each minimizes any difficulty with file upload and reduces respondent fatigue. The patients and caregivers complete the interviews independently to avoid influencing one another, and a separate video is recorded for each interview topic module.

After each module is recorded, the individual who recorded the video is prompted to review and submit the video in the mobile application through a secure platform. Interview study team staff review each video after it is submitted to ensure that it has been captured correctly (e.g., audible, follows interview guide). If videos need to be rerecorded, participants receive a notification in the mobile application that tells them that they need to rerecord a video. Study coordinators contact patients by email or phone with specifics on what aspects of the video submission did not meet quality criteria. Once study staff have verified that all interview submissions have met quality criteria, study coordinators send the participant a confirmation email to let him or her know that the submissions are complete. If incentives are being provided for the study, they are sent out after the interview completion.

If any adverse events (AE) are reported during the interviews, the interview study team should direct the patient or caregiver to report the AE to the main study. The interview study team should document when they found out about the AE, what was reported, whether the patient or caregiver reported it to the main study team, and the current status of the AE.

Data Analysis

The audio from each interview is transcribed by a medical transcription company under a confidentiality agreement. Interview transcripts are then analyzed by two coders trained in qualitative research who conduct a thematic analysis using open coding. Both coders read through all of the transcripts independently, identify salient themes, and add themes to a code list using open coding. Researchers compile the two code lists to create a master codebook of themes and subthemes, and the two coders discuss the codebook and come to an agreement about each code. If agreement cannot be reached, a third coder can be brought into the discussion to resolve any discrepancies.

The coders divide the transcripts into two sets, and each coder independently primary codes a transcript set using a qualitative data analysis software package, such as Dedoose Version 7.0.23 (Los Angeles, CA). After primary coding one set, the coders switch transcript sets and secondary code the other set. After secondary coding, the coders discuss all coding discrepancies and resolve all conflicts in coding. After finalizing the coding, the two coders summarize the key themes in the data overall and by individual patient using tables, figures, and representative quotes.

In addition to the qualitative data analysis, comparing the qualitative results to the traditional endpoint data captured in the clinical studies allows for the assessment of the concordance of the data [10], adds context to the study data, and provides insight into the degree of change that is meaningful to patients and caregivers, and why such change was meaningful to them.

This article does not contain any studies with human participants or animals performed by any of the authors.

Discussion

Including PPC and CPC Assessments during a clinical trial helps to provide a fuller picture of the patients’ lives before and during the treatment period and allows these patients to report on specific aspects of their daily lives that may have changed. Adding video interviews into a clinical trial helps to address some of the challenges of capturing clinically meaningful treatment effect in rare diseases and allows sponsors to collect information about the patient experience with the study drug outside of more traditional endpoints, which may capture meaningful effects that are only experienced by a small subset of the population. By taking the totality of the data into consideration, sponsors, regulators, payers, and prescribers may be able to better understand how treatment benefit manifests across small but heterogeneous patient populations.

PPC and CPC Assessment methodology builds on the use of the Patient Global Impression of Change (PGI-C) scale, which is employed routinely in clinical trials and has been a key endpoint in supporting FDA approval decisions by providing evidence of clinical meaningfulness in conjunction with more objective, traditional endpoints [41, 56,57,58,59,60,61,62,63,64,65]. This scale allows patients to retrospectively rate change from baseline in a clinical trial. The video interviews similarly involve the assessment of changes during a clinical trial by patients and caregivers, but they may be more informative than a rating scale, since the interview participants can explain the changes they experience and why those changes are meaningful.

The interview guide is designed to improve understanding of the real-world, nuanced, and potentially varied impact of a potential therapy, and patient and caregiver responses may help to answer the question of if and how the drug being tested works. When there are differences in disease presentation, disease progression, and treatment benefit from patient to patient, qualitative interviews can take into account the patient’s baseline severity and disease trajectory [8]. The qualitative interviews provide a valuable method to collect information on treatment effects that might benefit a subset of the study population, which can be valuable to healthcare providers as they make prescribing decisions. If the quantitative data is inconclusive, the qualitative data can be used to explain potential reasons why and what treatment impact, if any, patients and caregivers experienced [66]. Identifying changes that patients experienced in a thematic way and finding agreement between the qualitative and quantitative data can provide compelling evidence of a treatment effect. The qualitative data may also provide needed information about context during the clinical trial, which could help to interpret why an individual patient did or did not respond to the drug.

In addition to better understanding if a potential therapy works, qualitative interviews help to interpret the degree of change that is meaningful to patients. The extent of treatment benefit that patients consider meaningful often changes dramatically over the course of a disease. What patients with advanced-stage disease consider meaningful is typically of much smaller magnitude than patients earlier in their disease progression. For example, measuring hand function is crucial to understand the quality of life of patients with advanced-stage Duchenne muscular dystrophy who are non-ambulatory, but would not be relevant to patients earlier in their disease progression who are ambulatory with only a limited mobility deficit. Understanding the degree of change that is meaningful for different segments of the population informs the development of outcome measures for future studies and broadens the population that can gain access to and benefit from effective treatments.

Incorporating qualitative interview data into clinical trials allows researchers to learn from and build on each phase of development. Qualitative interview data on patient experience from a phase 2 clinical trial is crucial in guiding conversations with regulators and selecting a key endpoint for the phase 3 clinical trial [49]. Including the interview data in the dialogue with regulators after a phase 3 clinical trial provides a fuller picture from the patient and caregiver perspectives of the clinical meaningfulness of any changes and may support the clinical trial results [51].

Seeing the patients and caregivers on video and hearing directly from them about their experiences can be compelling and impactful to regulators, payers, and prescribers, particularly in instances of a small subset of responders. Including video interview data in conversations with regulators and payers may encourage them to consider an impact on patient quality of life that may not have been documented in the primary or secondary outcome measures. If clinical trials collect PPC and potentially CPC Assessment data, it can be provided as “patient experience data” that is part of the package of evidence that helps regulators, payers, and prescribers in their decision-making processes [17].

Conclusion

The need for effective treatments combined with rare disease methodological challenges makes it imperative that rare disease clinical trials include a mechanism for more systematically capturing information about potential treatment effects from the perspectives of the patients and caregivers beyond traditional clinical outcome assessments. Capturing the data with sufficient rigor allows it to contribute meaningfully to the clinical trial evidence utilized in regulator, payer, and prescriber decision-making. Adding patient and caregiver video interviews to rare disease clinical trials is an innovative and potentially powerful way to give patients and their families a voice in the drug development process. PPC and CPC Assessments more fully maximize each patient’s contribution to the scientific data, providing a breadth and depth of insight into their experience in the clinical trial that allows researchers, regulators, and payers to have a more complete understanding of the clinical trial results.