Abstract
Real-world evidence (RWE) is increasingly used to complement clinical trial data for regulatory decision-making and in certain cases utilized to establish the clinical effectiveness of a therapy. However, the use of RWE is not applicable for all regulatory submissions, and it can be challenging to identify appropriate use cases. An interactive tool was developed (“Decision Aid,” https://sn.pub/TpDjZx) to assist researchers, industry, and other stakeholders in identifying regulatory situations that can benefit from leveraging RWE by organizing precedent cases based on a given regulatory objective (new product approval, labeling expansion for new indication or additional clinical data, post-marketing requirement) and type of RWE study design (external control, observational study, pragmatic trial). Key success factors ensuring fit-for-purpose data and rigorous methods (e.g., clear endpoints, minimizing bias, data completeness) are also described. The tool allows the user to navigate through the precedent cases by selecting certain regulatory objectives and/or study designs. The Decision Aid supports regulatory activities in the RWE space and encourages further use of RWE in regulatory decision-making.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
The aim of this paper is to describe an interactive Decision Aid designed to: |
Serve as a tool for researchers, industry, and other stakeholders to identify areas of opportunity for real-world evidence (RWE) to support regulatory strategies and clinical development plans. |
Illustrate key uses of RWE for regulatory purposes (by utilizing precedent cases where RWE was successfully used to support regulatory decisions), corresponding RWE study designs, and underlying key success factors. |
Encourage further use of RWE and activity in this space to add to the growing evidence and establishment of RWE in regulatory submissions. |
This manuscript and associated Decision Aid are intended to help researchers identify potential use cases in which RWE might be considered to inform regulatory decision-making. |
Interactive Decision Aid
This article contains an interactive Decision Aid to help researchers, industry, and other stakeholders in identifying regulatory situations that can benefit from leveraging real-world evidence. To view it, click here: https://sn.pub/TpDjZx. The interactive Decision Aid can also be accessed through the supplementary material. All files must be extracted for full functionality.
Introduction
Real-world evidence (RWE) has traditionally played a relatively limited role in the regulatory process, for example, addressing unmet need and satisfying post-marketing safety monitoring requirements. In the current evolving healthcare landscape, however, there has been a shift towards utilizing RWE in a more central and fit-for-purpose role through identifying situations in which RWE can help establish the clinical effectiveness of a therapy to support regulatory decision-making rather than only to supplement insights from clinical trials [1].
In the US, the Framework for the Food and Drug Administration’s (FDA’s) RWE Program was released in December 2018 and outlines several important RWE-related efforts the Agency is undertaking to evaluate the potential of RWE to support changes to labeling, such as adding or modifying an indication, a change in dose, dose regimen, or route of administration; adding a new population; or adding comparative effectiveness or safety information [2]. In Europe, the European Medicines Agency’s (EMA’s) Regulatory Science Strategy 2025 initiative includes the promotion of high-quality RWE in decision-making to generate complementary evidence across the product lifecycle as one of its five strategic goals [2]. RWE initiatives are being conducted by other global health authorities around the world as well, including in Canada [3], China [4], and Japan [5], to encourage and optimize the use of RWE for regulatory decisions, while focusing on the reliability and quality of evidence. Given the increasing interest in leveraging RWE for regulatory decision-making, there have been many recent publications on insights and recommendations on ways to ensure fit-for-purpose RWE [6,7,8,9]. In addition, the currently ongoing COVID-19 pandemic has further reinforced and accelerated Health Authorities’ interest in the potential utility of RWE [10,11,12].
Despite the growing interest in leveraging RWE by industry and evaluation of RWE by Health Authorities for regulatory decision-making, it is important to note that RWE is not applicable to all situations and should not be considered a replacement for clinical trials [13]. Regulatory acceptance of RWE to support data from clinical trials is subject to the respective Health Authority’s approaches and practices. The authors of this manuscript developed an interactive tool (“Decision Aid”) to assist in more easily identifying circumstances in which the use of RWE informed regulatory decision-making (Figure 1; https://sn.pub/TpDjZx). To date, RWE has been used in certain situations to support new product approvals, label expansions for new indications or additional clinical data, and fulfillment of post-marketing requirements. In such cases, various real-world (RW) or RW-clinical hybrid designs have been utilized, including external controls (external benchmarks or external comparators), observational studies, and pragmatic clinical trials. There are various factors, such as the rarity of the disease, magnitude of treatment effect, availability of quality RWD, and others, that can help inform the appropriateness of a RWE approach for regulatory decision-making. These factors can be better understood by reviewing precedent cases in this space.
The aim of this paper is to describe an interactive Decision Aid designed to:
-
Serve as a tool for researchers, industry, and other stakeholders to identify areas of opportunity for RWE to support regulatory strategies and clinical development plans.
-
Illustrate key uses of RWE for regulatory purposes (by utilizing precedent cases where RWE was successfully used to support regulatory decisions), corresponding RWE study designs, and underlying key success factors.
-
Encourage further use of RWE and activity in this space to add to the growing evidence and establishment of RWE in regulatory submissions.
This article and associated Decision Aid are intended to help researchers identify potential use cases in which RWE might be considered to inform regulatory decision-making.
Compliance with Ethics Guidelines
This article is based on previously conducted studies and does not contain any new studies with human participants or animals.
Methods
Several case examples in which RWE was leveraged to support successful FDA and EMA regulatory submissions were selected and described based on a systematic review of regulatory labels, published literature, and regulatory assessment reports. Each case study was matched with the regulatory objective obtained (new product approval, labeling expansion for new indication or additional clinical data, post-marketing requirement) as well as the type of RWE study design used (external control, observational study, pragmatic trial). Based on this matching, the Decision Aid was organized accordingly by regulatory objective and RWE study design type.
Case studies were assessed for key success factors that have been described in the context of ensuring fit-for-purpose data and rigorous methods, which contributed to a positive regulatory opinion. The assessment was based on a checklist to ensure regulatory-grade data quality (completeness, transparency, generalizability, timeliness, scalability) [14], supplemented by additional considerations around clear endpoint definitions and strategies for minimizing bias for a careful assessment of data quality, reliability, and relevance [2, 15, 16]. Providing clear definitions of primary and secondary endpoints is important to avoid variable documentation of key events potentially influenced by subjectivity. RWD often encompasses a broader spectrum of patients which leads to better generalizability than clinical trials, but potential biases need to be considered and mitigated to allow for adequate interpretation of results. Pre-defined data abstraction, harmonization, and quality monitoring are key markers of quality, integrity, and completeness. Being transparent with Health Authorities around the study design deployed, whether in protocols, analysis plans, or reporting of study results, is another key success factor to allow Health Authorities to rigorously assess the RWE submitted. Leveraging recent and timely RWD ensures the data provide relevant insights around clinical decision-making and adequately reflects the dynamic nature of routine medical practice and treatment use. For data that were intended for scaling, data curation processes, applicability to multiple contexts, and variable definitions were also assessed.
The Decision Aid includes case studies on pharmaceutical products from multiple manufacturers and includes case studies that were available at the time this manuscript was drafted. The authors intend to submit annual updates as the regulatory landscape, frameworks, and case studies change.
Regulatory Use Decision Aid
The Decision Aid can be accessed at https://sn.pub/TpDjZx.
Users can navigate through the Decision Aid by using sequential multi-step navigation as the tool matches the regulatory objective being evaluated with the corresponding key success factors, the most applicable RWE study types, and real-life precedent cases for reference.
-
When opening the tool, the general RWE success factors are described, key terms are defined, and an overview of the case studies is provided.
-
When clicking on a particular regulatory objective (new product approval, label expansion, and post-marketing requirement), the most applicable corresponding RWE study types are displayed, including external control, observational study, or pragmatic clinical trial.
-
After selecting a specific RWE study type, the tool will display corresponding real-life precedent case studies for that specific combination of regulatory objective and study type, covering multiple therapeutic areas.
-
o
Individual case examples include key information, such as narrative and relevant success factors.
-
o
Each case study also includes hyperlinks to regulatory assessment reports to facilitate further reading.
-
o
-
The user can navigate forward and backward throughout the tool to access all information.
The list of case studies included in the Decision Aid is not exhaustive, as new case studies potentially became available after the tool was developed.
Discussion
The Decision Aid will help guide pharmaceutical companies and clinical researchers in assessing general appropriateness of a potential RWE approach for use in clinical development within a regulatory context. The Decision Aid does not aim to replace Health Authority guidance or the critical importance of engaging Health Authorities on RWE proposals early on to gather their feedback on the proposed approach, but the tool can help identify potential opportunities for leveraging RWE to inform certain regulatory objectives. It is this identification step that the Decision Aid aims to focus on. Further comprehensive assessment based on the particulars of a situation may be necessary to confirm whether a real-world evidence-based approach for a particular scenario is appropriate.
Conclusion
RWE has been increasingly used in later stages of drug development. Importantly, the intent of RWE is not to replace clinical trials but to apply RWE where appropriate to support drug development and ultimately regulatory submissions. Through identifying and illustrating key aspects of situations where RWE has been utilized for regulatory decision-making, the Decision Aid helps pharmaceutical companies and clinical researchers understand the type of potential evidence necessary for regulatory submissions, adds to the knowledge of available RWE, and encourages further regulatory activity using RWE to pave the way for broader acceptance by clinical researchers, pharmaceutical companies, and Health Authorities. To maximize the chances for success, any research proposal that intends to be used for regulatory decision-making should be discussed with the relevant Health Authority before the research is started.
References
Cave A, Kurz X, Arlett P. Real-world data for regulatory decision making: challenges and possible solutions for Europe. Clin Pharmacol Ther. 2019;106(1):36–9.
US Food and Drug Administration. Framework for FDA’s real-world evidence program. 2018. https://www.fda.gov/media/120060/download. Accessed Jul 2020.
European Medicines Agency. Glanville, D. Regulatory Science to 2025. 2018. https://www.ema.europa.eu/en/about-us/how-we-work/regulatory-science-strategy. Accessed Jul 2020.
Government of Canada. Optimizing the use of real world evidence to inform regulatory decision-making. 2019. https://www.canada.ca/en/health-canada/services/drugs-health-products/drug-products/announcements/optimizing-real-world-evidence-regulatory-decisions.html. Accessed Jul 2020.
China Center for Drug Evaluation. Key considerations in using real-world evidence to support drug development. 2019. https://www.chcuk.co.uk/china-key-considerations-in-using-real-world-evidence-to-support-drug-development/. Accessed Jul 2020.
Pharmaceuticals and Medical Devices Agency (Japan). PMDA Updates. 2019. https://www.pmda.go.jp/files/000231353.pdf. Accessed Jul 2020.
Dreyer NA. Advancing a framework for regulatory use of real-world evidence: when real is reliable. Ther Innov Regul Sci. 2018;52(3):362–8.
Mack C, Christian J, Brinkley E, Warren EJ, Hall M, Dreyer N. When context is hard to come by: external comparators and how to use them. Ther Innov Regul Sci. 2019. https://doi.org/10.1177/2168479019878672.
Baumfeld Andre E, Reynolds R, Caubel P, Azoulay L, Dreyer NA. Trial designs using real-world data: the changing landscape of the regulatory approval process. Pharmacoepidemiol Drug Saf. 2020;29(10):1201–12.
Beaulieu-Jones BK, Finlayson SG, Yuan W, Altman RB, Kohane IS, Prasad V, et al. Examining the use of real-world evidence in the regulatory process. Clin Pharmacol Ther. 2020;107(4):843–52.
US Food and Drug Administration. Coronavirus (COVID-19) Update: FDA Collaborations Promote Rigorous Analyses Of Real-World Data To Inform Pandemic Response. 2020. https://www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-collaborations-promote-rigorous-analyses-real-world-data-inform. Accessed Sep 2020.
US Food and Drug Administration. Coronavirus (COVID-19) Update: FDA takes additional action to harness real-world data to inform COVID-19 response efforts. 2020. https://www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-takes-additional-action-harness-real-world-data-inform-covid-19. Accessed Sep 2020.
Donnell JC, Kim-Le T, Dobrin R, Higashi M, Pereira A, Wagner S, Yang A, Hukkelhoven M. Evolving use of real-world evidence in the regulatory process: a focus on immuno-oncology treatment and outcomes. Future Oncol. 2020;17:333–47.
Miksad RA, Abernethy AP. Harnessing the power of real-world evidence (RWE): a checklist to ensure regulatory-grade data quality. Clin Pharmacol Ther. 2018;103(2):202–5.
Duke-Margolis Center for Health Policy. Determinign real-world data's fitness for use and the role of reliability. 2019. https://healthpolicy.duke.edu/publications/determining-real-world-datas-fitness-use-and-role-reliability. Accessed Jan 2021.
Kahn MG, Callahan TJ, Barnard J, Bauck AE, Brown J, Davidson BN, et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC). 2016;4(1):1244.
Arone B. The argument for external comparator adoption. Pharmaceut Execut. 2019;36:6.
NIH. NCI Dictionary: Retrieved from observational study. 2020. https://www.cancer.gov/publications/dictionaries/cancer-terms/def/observational-study. Accessed Jan 2021.
Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13(2):217–24.
Luce BR, Drummond M, Jönsson B, Neumann PJ, Schwartz JS, Siebert U, Sullivan SD. EBM, HTA, and CER: clearing the confusion. Milbank Q. 2010;88(2):256–76.
Zhu M, Sridhar S, Hollingsworth R, Chit A, Kimball T, Murmello K, Greenberg M, Gurunathan S, Chen J. Hybrid clinical trials to generate real-world evidence: design considerations from a sponsor’s perspective. Contemp Clin Trials. 2020;94: 105856.
Acknowledgements
Funding
Funding for the study and the journal’s Rapid Service and Open Access fees was provided by Bristol-Myers Squibb (BMS).
Medical Writing and Editorial Assistance
We thank Eva Oakkar, IQVIA, for medical writing assistance and interactive Decision Aid development. This assistance was funded by BMS.
Authorship
All named authors meet the International Committee of Medical Journal Editors (ICMJE) criteria for authorship for this article, take responsibility for the integrity of the work as a whole, and have given their approval for this version to be published.
Author Contributions
Employees of Bristol-Myers Squibb Company: Leah Burns, Conception, Data Acquisition, Data Analysis, Data Interpretation. Robert Kalesnik-Orszulak, Conception, Data Acquisition, Data Interpretation. Rick Spring, Conception, Data Acquisition, Data Interpretation. Fabienne Zeegers, Conception, Data Interpretation. Mark Rutstein, Conception, Data Interpretation. Mathias Hukkelhoven, Data Interpretation. John O’Donnell, Conception, Data Interpretation. Funded by Bristol-Myers Squibb Company: Lisa Wruck, Data Interpretation, has received research grants from Bristol-Myers Squibb Company.
Disclosures
Leah Burns, Robert Kalesnik-Orszulak, Rick Spring, Fabienne Zeegers, Mark Rutstein, Mathias Hukkelhoven and John O’Donnell are employees of Bristol-Myers Squibb Company. Lisa Wruck did not receive funding for this project, but has received research grants from BMS previously.
Compliance with Ethics Guidelines
This article is based on previously conducted studies and does not contain any new studies with human participants or animals performed by any of the authors.
Data Availability
No datasets were generated or analysed during the current study.
Author information
Authors and Affiliations
Corresponding author
Supplementary Information
Below is the link to the electronic supplementary material.
Appendix 1: Glossary
Appendix 1: Glossary
External Control
External control data is any data generated from patients external to the parameters of the parent trial and used to provide context to a single-arm study where it would be impractical or ethical to design the study with a placebo or active comparator arm [17].
External Benchmark
A subtype of external control, in which real-word populations are used to provide context on outcomes. In this case, no direct comparison is made with the trial data, the data is aggregated.
External Comparator
A subtype of external control, in which patient-level RWD are used for a comparison versus the clinical data.
Observational Study
A type of study in which individuals are observed or certain outcomes are measured. No attempt is made to affect the outcome (for example, no treatment is given) [18].
Pragmatic Trial
Pragmatic trial is designed to test interventions in the full spectrum of everyday clinical settings in order to maximize applicability and generalizability [19].
Bias
In a scientific research study or clinical trial, a flaw in the study design or the method of collecting or interpreting information. Biases can lead to incorrect conclusions about what the study or clinical trial showed [18].
Real World Evidence (RWE)
RWE is the clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of real-world data (RWD) [1].
Real World Data (RWD)
RWD are data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources [1].
Effectiveness (in Contrast to Efficacy)
While RCTs provide evidence of therapeutic efficacy in a controlled setting, real-world studies produce evidence of therapeutic effectiveness in real-world practice settings [20].
RW-Clinical Hybrid Design
RW-clinical hybrid design is an integration of a traditional clinical trial with pragmatic design aspects to collect RWD on patients (21).
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.
About this article
Cite this article
Burns, L., Kalesnik-Orszulak, R., Spring, R. et al. Real World-Evidence for Regulatory Use Decision Aid: An Interactive Tool To Inform Clinical Development and Regulatory Strategies. Adv Ther 39, 4772–4778 (2022). https://doi.org/10.1007/s12325-022-02257-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12325-022-02257-4