RWE Framework: An Interactive Visual Tool to Support a Real-World Evidence Study Design

Abstract

Objectives

Real-world evidence (RWE) is essential for the development of pharmaceutical and medical technologies and informs treatment-related decisions by regulatory agencies, payers, healthcare providers, and patients. Given that planning RWE studies present diverse challenges, we developed the RWE Framework, a concise, visual, interactive tool designed to align multidisciplinary stakeholders toward common goals and encourage a methodical approach to RWE study planning.

Methods

A search of published literature and internet-based resources was performed to identify guidance on RWE study planning with decision and/or visual aids. A conceptual framework for a study design tool was developed based on best practices for RWE studies, enhanced with an infographic design, and refined by multidisciplinary input from RWE researchers.

Results

The searches confirmed an unmet need for a concise tool to support a broad range of RWE study designs: only two sources with decision/visual aids were identified. The novel RWE Framework comprises sequential decision steps with instructions to guide users through consideration of research objectives, product approval status, study setting, outcomes of interest, data availability in routine practice, need for primary data collection and/or randomization, study type and methodology, and applicable regulatory standards. Pilot testing using case studies of pharmaceutical assets demonstrated the utility of RWE Framework and applicability for use in team environments.

Conclusions

The RWE Framework is a novel, concise, visual, and interactive tool to inform RWE study planning. It addresses a broad range of real-world study types and research objectives and was found to enhance RWE decision-making by multidisciplinary teams. Further validation is warranted.

Introduction

Real-world evidence (RWE), derived from data collected during routine clinical practice, provides important insight into clinical experience, which complements information obtained from traditional randomized controlled trials (RCTs). Traditional RCTs typically measure the efficacy and safety of an intervention versus a comparator or placebo under standardized double-blinded conditions and in a narrowly defined population to minimize bias and/or confounding factors [1]. Data obtained from traditional RCTs are considered the gold standard for regulatory decision-making during initial registration of a medicine and expansion of a product label and to support treatment guidelines [1]. However, these trials are costly to undertake, and the patients enrolled in strictly controlled settings frequently do not represent those encountered in everyday clinical practice who often have characteristics associated with higher risk [2, 3]. Consequently, extrapolation of outcomes reported in traditional RCTs to the diverse patients and treatment settings observed in everyday clinical practice may be limited [2]. For example, shorter overall survival is reported in real-world patients with metastatic non-small cell lung cancer treated with programmed cell death protein 1 inhibitors compared with traditional RCTs, potentially because of narrow trial eligibility criteria [4]. Also, smaller reductions in glycated hemoglobin levels were reported with glucagon-like peptide 1 receptor agonist and dipeptidyl peptidase 4 inhibitor therapy in patients with type 2 diabetes in real-world studies compared with traditional RCTs, which has been attributed to differences in treatment adherence [5].

RWE from non-randomized studies may bridge the gap between the controlled environment of clinical studies and more heterogeneous real-world medical settings, generating additional evidence about safety and effectiveness post-market authorization. For example, RWE can improve estimates of treatment effects across a broad range of patients and clinical practice conditions and enable longer term follow-up of patients compared with traditional RCTs [1, 6]. The large sample sizes associated with real-world data may identify clinically relevant subgroups and help tailor treatment strategies to support new indications. For example, analysis of electronic health records from over 32,000 patients diagnosed with cancer revealed that individuals prescribed metformin, a first-line therapy for type-2 diabetes, had better survival compared with diabetic and non-diabetic patients who were not taking metformin, indicating a potential chemotherapeutic role for metformin [7]. Extensive data from electronic health records and insurance claims have also revealed l-DOPA, a drug used for movement disorders, has a protective effect on development of age-related macular degeneration [8]. Furthermore, large populations can provide insight into rare adverse events that may not emerge in traditional RCTs [2, 9]. RWE can also facilitate assessment of treatment patterns, patients with unmet needs, disease burden, healthcare utilization, and associated costs [1]. Consequently, assessing data from a variety of sources is important to enable a holistic approach to decisions made by payers, healthcare providers, regulatory agencies, and organizations conducting medical research and drug development.

There are numerous considerations when planning a RWE study. For example, to address the research objectives in a particular study population, it must be determined whether new (primary) and/or existing (secondary) data are needed. If appropriate, the optimal secondary data must be identified from a broad array of sources including clinical registries, insurer claims or administrative data, and electronic health records [1, 2]. Also, to address the research objectives the optimal study type must be selected from a wide range of options including pragmatic, observational, prospective, retrospective, cohort, case control, and cross-sectional designs. Studies must also be conducted in accordance with local regulatory and data privacy requirements, which can vary according to study design and country/regions of interest. These choices can present considerable challenges to researchers. Individuals designing RWE studies may have differing levels of experience with RWE best practices, and perspective on particular research objectives may differ across diverse research teams, which may include the multidisciplinary stakeholders involved in drug development or the evaluation of existing medicines. Different study designs also require study personnel with divergent skill sets. For example, retrospective studies rely on secondary databases, programming skills, and epidemiologic and statistical approaches, whereas prospective studies more closely resemble clinical studies and rely to a greater extent on medical affairs and on regulatory and site management expertise. Finally, researchers may have limited resources to plan their studies and little opportunity to collaborate across complex team structures inherent within many large organizations, including pharmaceutical companies. This may lead to inefficiencies in study planning and potentially impact the quality of real-world data obtained.

Real-world studies need to be well designed, given the growing importance of RWE in evaluating diseases and medical interventions to inform clinicians, patients, payers, policy makers, and pharmaceutical companies [2, 10]. The aim of this research was to develop a concise, visual, and interactive tool to align multidisciplinary stakeholders toward common goals and encourage a methodologic approach to RWE study planning, which could be implemented in a team environment.

Methods

Development of the RWE study planning tool, the ‘RWE Framework,’ involved a multi-step process as described in Fig. 1. In the initial step, key components of best practice RWE study planning were identified during focused group discussions among the research team and based on experience with design and implementation of multiple RWE studies. Using this information, a list of key considerations that multidisciplinary teams routinely encounter during RWE study planning, and a visual tool that might help to address, was compiled. This included research objectives, the level of intervention required, study design, use of existing data, and regulatory guidance. In addition, preferred attributes of a potential visual tool were identified, which included concision, holistic nature, and incorporation of a decision tree and/or checklist.

Fig. 1

Development of the RWE Framework. RWE, real-world evidence. ^aGoogle.com and websites of relevant organizations: European Medicines Agency (http://www.ema.Europa.eu), US Food and Drug Administration (http://www.fda.gov), Association of the British Pharmaceutical Industry (http://www.abpi.org.uk), International Society for Pharmacoeconomics and Outcomes Research (http://www.ispor.org), Get Real (http://www.imi-getreal.eu), and Network for Excellence in Health Innovation (http://www.nehi.net). ^bExpertise in epidemiology, clinical outcomes assessment, health economics, outcomes research, market access, medical science, and medical affairs

Literature Review

A literature search was conducted to identify existing decision tools or visual aids that provide best practice guidance for planning or conducting RWE studies. Prior to conducting the literature review, the search strategy was defined by specifying objectives, information sources, search criteria, and eligibility criteria. A search of English language publications from January 2008 to December 2017 was conducted in Embase^® and MEDLINE^®. The following search terms were contained in the abstract or title: ‘real world evidence’ or ‘real world data’ or ‘pragmatic trial’ combined with ‘consensus statement’ or ‘flow diagram’ or ‘decision tool’ or ‘best practice’ or ‘generation’ or ‘algorithm.’ A validation search was performed using the search terms as free-text and MeSH or Emtree terms, limited to review articles.

A search of internet-based sources was also undertaken using terms derived from the Embase^® and MEDLINE^® searches. Supplemental online searches were conducted via Google.com and on websites of the following relevant professional organizations, government institutions, and public-private partnerships: European Medicines Agency (http://www.ema.Europa.eu), US Food and Drug Administration (http://www.fda.gov), Association of the British Pharmaceutical Industry (http://www.abpi.org.uk), International Society for Pharmacoeconomics and Outcomes Research (http://www.ispor.org), GetReal (http://www.imi-getreal.eu), and Network for Excellence in Health Innovation (http://www.nehi.net).

To warrant inclusion in this literature review, papers and online resources had to report best practices for designing RWE studies and include decision and/or visual aids. Exclusion criteria included commentaries, editorials, letters, case reports, legal cases, news, technical reports, and outputs focused on children. The titles and abstracts of the publications and online sources identified were screened by one assessor; all authors reviewed the selection of the screened papers and were involved in their appraisal.

Development of the RWE Framework

Following the literature review, the conceptual framework of a tool to aid the planning and design of RWE studies was developed using a staged approach. First, the key components of best practice RWE study planning were organized logically and sequentially to create a flow diagram. An infographic design based on a ‘Q&A’ framework was then applied to facilitate its use. Finally, resources were developed to provide guidance on the use of the flow diagram including instructions, background information on key definitions and checklists, and references to external resources [11,12,13,14,15,16,17].

Each section of the flow diagram and associated resources was refined following qualitative input from RWE researchers from Covance and Mundipharma. Feedback was obtained during meetings with researchers with expertise in epidemiology, clinical outcomes assessment, health economics, outcomes research, market access, medical science, and medical affairs. This included a five-member RWE cross-functional working group comprising representatives from five departments (Medical Science, Drug Safety, Epidemiology, Regulatory, and Market Access), an eight-member European market access team, and RWE specialists from France, Germany, Italy, Spain, the UK and Nordic region.

Pilot Testing of the RWE Framework

The resulting RWE framework was then pilot tested during two workshops. Together, these workshops comprised 40 individuals from 9 European countries with RWE expertise, which included medical science, epidemiology, drug safety, research and development, regulatory, and market access aspects of drug development. The first workshop comprised 20 members of the Mundipharma Nordic team, and the second comprised 20 regulatory and research and development experts from Mundipharma. Using a series of group exercises, the RWE framework was used to design potential studies to address unmet needs associated with Mundipharma assets across a range of settings (pain, asthma, and diabetes).

Data sharing was not applicable to this article as no data sets were generated or analyzed during the current study.

Results

Literature Review

A total of 371 abstracts were identified in the Embase^® and MEDLINE^® searches and screened (Fig. 2). Of these, three articles were selected for detailed review based on their titles and abstracts [18,19,20]. All were excluded following full-text review because none contained decision and/or visual aids to assist in the design of RWE studies, although one referred to the Innovative Medicines Initiative (IMI) GetReal Consortium, which published an online tool captured under the online search, as described below [19]. The validation search using the search terms as free-text and MeSH or Emtree terms, limited to review articles, did not return any additional relevant results.

Fig. 2

Literature and online search results

From the online searches a further nine sources were identified for detailed review (Fig. 2). Six of these online sources did not include decision tools and/or visual aids [21,22,23,24,25,26], while one source included a visually supported decision-tool (Sure-Real) but was limited in scope to pragmatic trials, a subset of RWE studies [27]. The remaining two documents in which visual aids were identified are described below.

The first relevant document, produced by the Association of the British Pharmaceutical Industry (ABPI), provided guidance on how real-world data can support the marketing of medicines by generating new evidence or increasing the robustness of existing claims [13]. It included a brief diagram that indicated non-interventional real-world studies could be used to address questions regarding effectiveness, resource use, patient experience, service models, and treatment pathways. The suggested studies in the diagram were ‘audit’ to check against standards, ‘service evaluation’ for describing or evaluating a local service, and ‘research (observational)’ for generalizable new knowledge [13]. However, this diagram is limited in scope and, as such, lacked sufficient detail to inform the design of RWE studies. The second visual tool identified was RWE Navigator, developed by the IMI GetReal consortium (https://rwe-navigator.eu/what-is-the-imi-getreal-project/). RWE Navigator is an interactive online resource that assists users in identifying potential issues that may be faced in demonstrating the relative effectiveness of medicines and guides for potential types of RWE study designs depending on the development stage of a medicine and the relevant PICO (Population, Intervention, Comparator, Outcome) category [28]. The first step of the RWE Navigator aims to clarify potential effectiveness issues using a PICO framework for drugs in early (strategy), middle (operational), or late (submission) stages of development. Potential issues and corresponding real-world study design options are listed in stage 2. Links to further information and resources are also provided. While the RWE Navigator is a comprehensive repository of information that can be used as a resource to provide additional information about study methodologies, its utility in a multidisciplinary team environment may be limited by the complexity of the tool.

The literature review highlighted an unmet need for a concise, interactive, visual tool to support RWE study design in multidisciplinary team environments.

RWE Framework Development

The RWE Framework was constructed to address the gap identified in the literature as a concise, visual, holistic, and interactive tool that supports decision-making about RWE study design in a team environment. Key considerations captured in the RWE Framework include:

• Part I: establish objectives and study end points.

  • Research objective(s).

  • Product approval status.

  • Setting of study conduct.

  • Outcomes of interest.

• Part II: define the RWE study design.

  • Data availability in routine practice.

  • Need for primary data collection and randomization.

The RWE Framework comprises two parts that together capture a sequential, multi-step decision process. Based on series of ‘yes/no’ responses, it leads users through a set of considerations to support identifying the most suitable RWE study design from a broad range of study types. Instructions and checklists guide users through the RWE study planning exercise step by step and provide background information on key definitions.

The first part of the RWE Framework flow diagram considers whether the RWE research objective(s) are to inform decision-making within pharmaceutical companies, for example, indication selection or clinical trial planning, and/or to inform decision-making by other stakeholders such as regulatory agencies or payers (Fig. 3a). It then establishes whether the research objectives are to study a therapeutic area more broadly or to study a specific therapeutic intervention and/or to demonstrate product value. Approval status of the therapeutic intervention and whether the study will be conducted in routine practice are also considered to determine if a RWE study (or interventional study) is warranted. For example, a study would be considered interventional if an intervention (e.g., diagnostic test or procedure) that is not normally conducted in routine clinical practice or another alteration to routine clinical practice is necessary to address the research objectives.

Fig. 3

a RWE Framework flow diagram. Interactive RWE Framework is available online: https://rweframework.com/. b RWE Framework instructions

The framework suggests consultation with RWE and clinical research teams for clinical trials or expanded access studies that will be conducted in settings of routine clinical practice. The RWE flow diagram then seeks to establish the outcomes of interest required to investigate the research objectives. These categories include but are not limited to epidemiology, disease burden, patient experience, treatment patterns/pathways, treatment adherence/switching, off-label use, safety, effectiveness, patient-, clinician-, or observer-reported outcomes, resource utilization, and cost. Users are then asked to determine and specify study end points within the categories of outcomes selected. For example, users might establish the EQ-5D score, length of hospital stay, time to treatment switch, or occurrence of treatment-related adverse events as specific end points within the patient-reported outcomes, resource utilization, treatment patterns, or safety outcomes of interest categories, respectively.

Once the outcomes of interest and specific study end points have been established, the second part of the RWE framework ascertains whether the data of interest and appropriate quality are recorded during routine practice. Users are asked to determine if the selected patient population and data variables can be consistently identified in existing data sets and, if so, to select the appropriate database and methodology, which may require a feasibility assessment to ensure the data are consistently and completely captured in order to address the research objective (Fig. 3a). This information is used to determine if new data need to be collected (primary data) or if retrospective analysis of existing data sets (secondary data) or a hybrid approach (primary data collection supplemented with secondary data analysis) is appropriate. If primary data collection is indicated, the user must select whether randomization is necessary. Based on these outcomes, the final part of the flow diagram recommends which type of RWE study design may be appropriate. This includes study types within the categories of non-interventional/observational (retrospective [data already collected] or prospective [data collected after the study is designed]) and pragmatic, interventional trials (Fig. 3a). For non-interventional/observational case control, cohort, and cross-sectional studies, potential data sources also require consideration and may include electronic health/medical records, patient portal data, administrative data, payer claims data, laboratory results databases, consumer/employee/social data, registry, chart review/abstraction, survey, focus groups, and interviews. The RWE Framework also directs the user to applicable regulatory standards that should be considered in consultation with a RWE and/or clinical team prior to proceeding with the study design. The instructions developed to guide users through the steps of the RWE Framework flow diagram, key terms, and definitions and checklists are shown in Fig. 3b.

The RWE professionals generally provided positive feedback on the value of the conceptual RWE Framework and its ease of use. Their feedback was used to refine the flow and sequence of the framework components, including the starting point and next steps in the flow, and to include additional steps in the framework. Other implemented recommendations included labeling the stages of the framework, listing items under key steps (e.g., types of outcomes of interest), including terminology descriptions in the supporting information, and improving infographic design (e.g., use of icons).

Pilot Testing: Case Studies

The RWE Framework was used to inform the design of potential studies to address unmet needs associated with three Mundipharma assets (Table 1). These included an oral antidiabetic medication for which European data regarding real-world effectiveness and outcomes were lacking. The second asset was a portable, non-opioid analgesic inhaler for moderate-to-severe trauma pain, which was launched recently in Europe and required evidence regarding outcomes and the impact of the product on healthcare resource utilization in real-world settings. The third asset was an asthma inhaler for which real-world effectiveness data are required while maintaining some control over patient selection.

Table 1 RWE Framework case studies

The challenges associated with these assets resulted in distinct research objectives and outcomes of interest being identified in the first steps of the RWE Framework (Table 1). Based on the data available to address these research objectives, the RWE Framework identified several potential real-world studies of differing designs. RWE Framework use resulted in two studies being conducted to investigate the impact of the non-opioid analgesic inhaler on healthcare resource utilization. The first was a retrospective, non-interventional study of hospital chart data from patients with musculoskeletal trauma injuries to assess time frames for analgesia, and the second was a prospective cohort study to assess pain treatment effectiveness and hospital resource use [29, 30]. Study designs identified to address the unmet needs of the other two Mundipharma assets included a retrospective, cohort, non-interventional study using data from electronic health/medical records or national registries and a pragmatic trial and retrospective, cohort, non-interventional study using data from electronic health/medical records (Table 1).

Based on input from the pilot testing, several refinements to the RWE Framework were made including changes to terminology, further instructions to indicate when to consult clinical versus RWE teams, along with expanded instructions capturing definitions and additional resources. Following these refinements, the interactive RWE Framework along with instructions and resources was made available online (https://rweframework.com/).

Discussion

RWE can provide important insight into the effects of an intervention beyond the usual scope of traditional RCTs. For example, real-world studies may capture outcomes from a broad range of patients and clinical practice conditions and provide insight into long-term clinical benefits, rare adverse events, patient experience, and healthcare resource utilization that may not be available at product launch. RWE can also play an important role in decision-making by organizations conducting medical research and development. For instance, consideration of RWE can enhance clinical trial planning and clinical development strategy compared with traditional RCT data alone [31]. RWE can also be useful to evaluate existing medicines beyond the efficacy and safety outcomes reported in traditional RCTs and address post-approval health technology assessment or reimbursement requirements [32]. Moreover, RWE can assist patients themselves to make informed decisions, for example, gaining insight into factors that may impact outcomes of different cancer treatments [4, 24]. However, selecting the optimal design for a real-world study can present challenges to researchers due in part to the wide range of potential study types, diverse data sources of varying quality, and the need for multidisciplinary expertise. Here, we describe the newly developed RWE Framework, which was created to assist researchers to design robust real-world studies.

Our novel, holistic tool aimed to define the objectives and outcomes of interest for RWE and determine the appropriate study design to meet the research aims. It informs the design of a broad range of RWE studies from retrospective and prospective observational studies with case-control, cohort, and cross-sectional designs to pragmatic randomized trials in which data are collected prospectively in diverse study populations and settings. To our knowledge, the RWE Framework is the first concise, visual, interactive RWE study design tool based on sequential decision steps that considers research objective(s), product development status, outcomes of interest, data availability in routine practice, study setting, need for primary data collection and randomization, and applicable regulatory standards. Our search of the published literature and websites from relevant professional organizations, government institutions, and public-private partnerships revealed no comparable visual, real-world study design tools to the RWE Framework. The utility of the RWE Framework was demonstrated in a team environment by pilot testing across therapeutic settings (pain, asthma, and diabetes) with differing unmet RWE needs. During this process, the step-wise questions facilitated discussions among individuals with diverse RWE expertise. This enabled research objectives and outcomes of interest to be aligned across the multidisciplinary group, and, based on the identified data sets, resulted in a variety of potential study designs. These included a pragmatic trial to demonstrate the real-world effectiveness of a recently launched asthma inhaler and a prospective cohort study of a portable non-opioid inhaler for trauma pain to assess its impact on healthcare resource utilization. Based on this pilot testing experience, we believe the RWE Framework has the potential to enhance RWE capabilities by facilitating communication and collaboration among multidisciplinary teams and help to align individuals toward common goals. An online version of this tool is available at https://rweframework.com/.

RWE is anticipated to play an increasing role in informing clinical development planning and decisions that impact patients’ access to medicines [31, 33]. Indeed, recent guidance from the US Food and Drug Administration (FDA) discusses the possibility of using real-world data to evaluate the safety and effectiveness of new medical devices as well as a multifaceted framework to optimize the utility of RWE to help support regulatory decision-making for drugs and biologic products [21, 34]. Furthermore, the Adaptive Pathways approach of the European Medicines Agency (EMA) to license new interventions in settings of high unmet needs may consider RWE to supplement clinical trial data [35]. Consequently, it is not surprising that there is a growing call for pharmaceutical companies to utilize the full value of real-world data throughout a product’s lifecycle [13, 20, 25, 36, 37]. Steps must be taken to ensure that the challenges associated with RWE, such as potential bias due to lack of randomization, representativeness of data based on the data sources analyzed, and criticisms pertaining to multiplicity of studies and conflicting results, are addressed [23, 38]. High-quality data are a key element of study design to generate credible RWE, and variations in the recording and completeness of clinical narratives in electronic health records and other relevant data sources can be challenging. Data must be cleaned, harmonized, and linked to fill in gaps and include relevant end points, all of which require careful consideration when designing a study [10]. The joint ISPOR-ISPE Special Task Force, one of the resources included in the RWE Framework (Fig. 3b), has provided guidance to enhance confidence in evidence derived from real-world studies, focusing on transparency and replicability. Factors that should be considered during study design include establishing whether the study is exploratory or testing a specific hypothesis and for the latter publicly disclosing the study protocol and analysis plan [22]. This underscores the importance of well-designed real-world studies and the potential utility of study design tools to facilitate the generation of robust RWE to support healthcare decision-makers and to eliminate personal biases when studies are designed.

The RWE Framework is associated with some limitations. Its use as a study design tool requires some knowledge of study design, and users may be required to refer to external resources referred to in RWE Framework (Fig. 3b) and consult colleagues with particular expertise to address some of the questions in the flow chart, such as those around data quality. Furthermore, as the RWE Framework aims to provide a concise tool to inform decision processes, not all study design considerations can be captured, as is the case with any framework. Consequently, the design of potential studies initially identified using this tool should be refined in consultation with a RWE expert team, as is suggested in the RWE Framework flowchart. In addition to the data quality considerations mentioned above, other points that researchers may wish to address when finalizing the design of studies indicated by the RWE Framework may include (but are not limited to) statistical power, which may be particularly relevant for limited data sets such as new treatments and rare diseases, the impact of patient attrition if long-term outcomes are being assessed, and potential sources of bias or confounding factors and how these may be overcome or addressed when interpreting study outcomes. While the RWE Framework may be suitable for use by a broader range of medical researchers, it was developed with guidance from multidiscipline experts in RWE from the pharmaceutical and healthcare consulting industries. Input from other stakeholders including academic groups and regulatory and health technology assessment agencies was not obtained. Finally, while pilot testing of the RWE Framework was conducted to address a range of unmet RWE needs across differing pharmaceutical assets, it is feasible that there may be additional study design considerations unique to particular indications. Consequently, future development of geographical region-specific and/or therapy area-specific versions of the RWE Framework may be warranted.

Conclusions

The RWE Framework is a novel, visual, and concise study design tool that was developed to address an unmet need for an easy-to-follow, interactive aid that can replace ad hoc RWE study planning processes. Holistic in nature, the RWE Framework informs the design of a broad range of real-world study types in order to address a variety of research objectives and outcomes of interest, and it has demonstrated utility for study planning in team environments. The RWE Framework may complement the use of additional resources including knowledge regarding the product and/or therapy area when planning RWE studies. Further validation is required to ascertain how this tool can enhance collaborative decision-making by the multidisciplinary RWE stakeholders regarding study design.