Pharmaceutical Medicine

, Volume 31, Issue 4, pp 213–216 | Cite as

Different Weights of the Evidence-Based Medicine Triad in Regulatory, Health Technology Assessment, and Clinical Decision Making

  • Evelyn Schlegl
  • Pierre Ducournau
  • Jörg RuofEmail author
Open Access
Current Opinion


Evidence-based medicine (EBM) is defined as a systematic approach to clinical problem solving by the integration of best research evidence with real-world clinical expertise and patient values. Since those early days, decision requirements expanded from patient-centric clinical decision making to a population-based view including regulatory health technology assessments (HTAs). Regulatory bodies mainly rely on the totality of research evidence, which includes preclinical and all available clinical data. HTA bodies primarily focus on clinical data with a strong preference for comparative data from randomized controlled clinical trials (RCTs). Conversely, bedside clinical decisions are largely driven by real-world clinical expertise, which takes into account the individual patients’ preferences, as well as the availability of supportive research evidence. While the focus on research evidence is a typical feature of the early part of the adoption curve for innovative technologies, HTA decision makers need to ensure that clinical expertise is also appropriately included in their decisions, in order to avoid beneficial medications from being not available to patients.

Key Points

The evidence-based medicine (EBM) triad serves as the cornerstone of EBM; the well-established foundation for clinical, regulatory and health technology assessment (HTA) decision making.

HTA and regulatory decisions are jeopardized by confusing EBM for research evidence alone, and neglecting the other two pillars of the triad: real-world experience and patient preferences.

HTA decision making should take recommendations and the clinical expertise of respective scientific organizations into account, in order to avoid ‘false negative’ decisions on treatment benefit.

1 The Origin of the Evidence-Based Medicine (EBM) Triad

In the early 1970s, Archie Cochrane described his fundamental clinical decision dilemma: “I had considerable clinical choice of therapy: my trouble was that I did not know which to use and when” [1]. Consequently, EBM and the respective triad was gradually established. As David Sackett, one of its founding fathers, has put it, EBM is: “a systematic approach to clinical problem solving by the integration of best research evidence with clinical expertise and patient values” [2]. Since those early days, decision requirements gradually expanded from patient-centric clinical decision making to a broader, population-based view, including the development of clinical guidelines, regulatory assessments, and, more recently, health technology assessments (HTAs), respective appraisals, and related reimbursement decisions. The different scope of decision situations (i.e. patient-centric vs. population-based) necessitates a different weight of the three pillars of evidence within EBM.

2 The EBM Triad in Regulatory Decision Making

Regulatory bodies, such as the US Food and Drug Administration (FDA) and the European Medicines Agency (EMA), are the most critical public decision makers in the early lifecycle of new products. EBM forms the conceptual framework for their decisions. However, the early phases in the lifecycle of new products are characterized by limited ‘real-world experience’ and a limited understanding of ‘patient preferences’, which are both not formally required to obtain approval for a new technology. For the purpose of this manuscript, we used the term ‘real-world experience’ to describe the body of real-world clinical experience and evidence gathered by clinicians when treating their patients. While initial clinical experience may also be developed within a clinical trial setting, the term primarily includes clinical information and experience gathered outside the controlled, standardized, well-documented results of clinical trials. Regulatory decisions primarily rely on data from pivotal clinical trials, which we shall refer to as ‘research evidence’. Randomized clinical trials (RCTs) are still the gold standard of research evidence. However, a recent review of pharmaceutical regulatory approvals found that, between 1999 and 2014, 76 drug approvals for indications were granted without evidence from RCTs [3]. The traditional RCT-based research evidence paradigm is being challenged by the establishment of accelerated regulatory pathways, a shift towards continuous development of clinical data with the integration of comprehensive information into ‘Big Data’, and the targeting of smaller patient populations through precision medicine, or development for orphan and ‘super-orphan’ indications [4]. The EMA’s guidance on adaptive pathways is taking this discussion to the next level. The three guiding principles of adaptive pathways are (1) iterative, stepwise generation of evidence, (2) development and consideration of real-world data as supplement to clinical trial data, and (3) early involvement of patients and HTA bodies [5]. This represents a reliable and pragmatic approach by regulators to balance the benefit and risk of innovative medicines in a new, more targeted research environment.

Regulatory bodies commonly use the phrase ‘totality of evidence’ in describing what is to be considered for any regulatory decision [4]. This term usually refers to a systematic review of the existing body of knowledge in the early phase of a product, which can entail not only clinical but also pre-clinical data [6]. While this approach is more inclusive than, for example, many HTA-body assessments that primarily rely on RCT evidence, it still reflects only the totality of ‘research evidence’, as real-world expertise and data on patient preferences are largely unavailable in those early phases.

3 The EBM Triad in HTA Decision Making

The decision competency of HTA bodies is reaching far beyond the regulatory assessment of benefit and risk. HTA bodies are the gatekeepers of reimbursement for new products. Economic aspects are coming into play that, by their nature, necessitate a conflict. Richard Smith, the former editor of the British Medical Journal, pointed to this conflict of interest when arguing, that NICE “corrupts the concept of evidence-based medicine” [7]. Until today, NICE keeps a ‘Save Money’ link on its homepage, providing insights of the savings they generate [8]. This focus creates a general conflict with the aims of EBM. The founders of EBM have anticipated this challenge and mentioned that there is a risk of fundamentally misunderstanding the financial consequences of EBM, as it ‘may raise rather than lower the costs of care’ [2]. However, efforts are made to better align the assessments by regulatory authorities (i.e. EMA) and HTA bodies with the aim of providing patients with early and broad access to innovative medicines [9].

HTA bodies have a different scope on the available ‘research evidence’ than regulatory bodies. They rather focus on the available clinical data. Pre-clinical data and data on e.g. mechanism of action, which are part of the totality of data that regulatory bodies are referring to, are of less relevance to HTA bodies. Furthermore, they apply a very strict ranking of available evidence, with anything but RCTs having a difficult time in their assessments.

With regards to the EBM triad, the German IQWiG institute is rather explicit about its scope. Thus, predictions of success for individual cases are considered principally not possible. Additionally, ‘experiences based on individual cases are unsuitable for a benefit assessment, as it is not possible to ascribe the results of an individual case to the effect of an intervention’ [10]. Real-world experience’, which is based on experience in individual patients, is therefore mostly excluded from the benefit assessments. In fact, IQWiG mostly selects physicians who have not participated in the respective clinical development programs as advisors for respective benefit assessments. It is hard to determine when real-world experience should overrule RCT results. However, RCTs are always limited to certain questions and a certain patient population, while clinical reality reflects the whole variety of ‘real world’. As such, the concept of denying patients access to potentially beneficial treatments based solely upon an HTA decision-maker’s interpretation of RCT results alone is flawed.

‘Patient preferences’ are a challenging area for HTA bodies. While the German Federal Joint Committee (FJC) continuously supports and requests those data, none of the available assessments has been granted an additional benefit solely based on the patient-reported outcomes [11]. The generic European Quality of Life Instrument (EQ-5D) is requested as a base for utility calculations by NICE but is not highly regarded by FJC and the French Haut Autorite de Sante (HAS). Furthermore, it needs to be taken into account that utilities, as measured by the EQ-5D, are reflecting the societal perception on a certain condition, rather than the individual preferences of a patient suffering from a certain condition. Current public-private efforts such as the PREFER initiative (Patient Preferences in Benefit and Risk Assessments during the Treatment Life Cycle) are aiming to systematically examine and include the patient perspective throughout the regulatory and HTA process [12].

4 The EBM Triad in Clinical Decision Making

Optimizing clinical decision making was the initial aim of the founders of EBM [1, 2]. To achieve this goal, they aimed to integrate the three pillars: ‘research evidence’, ‘real-world expertise’, and ‘patient preferences’. Finding the right balance was considered critical to practicing good medicine. “Without clinical expertise, practice risks becoming tyrannized by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient [2]. Soon the question came up, if one of the three pillars should be considered first among the equals. Haynes et al [13] suggested a higher ranking for real-world expertise, in the sense that it should encompass all other sources of evidence, consider the trade-offs of the various perspectives and finally determine the best way forward in the interest of the patient. However, this patient-centric view on EBM gradually separated from the population-centric approaches of regulatory authorities, HTA bodies, and guideline committees [14] leading to the perception that EBM is a movement in crisis [15]. According to Greenhalgh et al [16] EBM, as it is applied by HTA bodies, regulatory bodies and guideline committees, tends to devalue the patient and carer experience. They therefore suggest a couple of approaches to increase the emphasis on these two pillars of EBM, which are ultimately relevant for good clinical decision making.

5 Summary

Roger’s theory on ‘Diffusion of Innovations’ [17] indicates that knowledge and data are usually the first steps in the adoption curve of new technologies. Experience of peers and own experience occur later in the lifecycle but are the key drivers for successful implementation. In EBM, decisions on a public health level (i.e. regulatory and HTA organizations) occur early in the lifecycle of new products. Their primary source of evidence is research data. Real-world experience and preferences of individual patients are scarce in those early days, but are decisive in the later phases of the adoption curve. The art of appropriate decision making in a regulatory and HTA environment is to consider all available data but always keep an open mind for the other two essential pillars of EBM. The lack of strong input from a real-world clinical experience perspective, or a patient preference perspective, within HTA decision making needs to be proactively addressed and managed in order to avoid, for example, unavailability of beneficial treatments for patients (see Table 1). The example of osimertinib, an innovative oncology product that has been pulled from the German market after a negative benefit assessment, indicates such a system failure. Although largely supported by the physician community [18], the product did not receive a positive benefit rating by the FJC, as the comparative RCT data were not yet available.
Table 1

Relevance of the three pillars of evidence in various decision-making situations


Research evidence

Real-world expertise

Patient preferences

Clinical decision making




Regulatory decision making




Health technology assessment




++ high relevance; + moderate relevance; (+) limited relevance

Those failures are the consequence of an imbalanced approach with regards to the three pillars of EBM and require correction in the future. In a most recent appraisal in ROS1-positive lung cancer, the FJC acknowledges, that despite the fact that no additional benefit has been demonstrated, crizotinib may be a relevant therapy option in individual cases [19]. From our point of view, this indicates an improved reflection of the three pillars of evidence in the FJC’s decision making.


Compliance with Ethical Standards

Conflict of interest

Evelyn Schlegl declares no conflict of interest. Pierre Ducournau is a full-time employee of F. Hoffmann-La Roche. Jörg Ruof is a former employee of Roche.


No financial support was received for the preparation of this manuscript.


  1. 1.
    Cochrane AL. Effectiveness and Efficiency: Random Reflections on Health Services. London: Nuffield Provincial Hospitals Trust; 1973.Google Scholar
  2. 2.
    Sackett DL, Rosenberg WMC, Gray JAM, Hayens RB. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–2.CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Hatswell AJ, Baio G, Berlin JA, Irs A, Freemantle N. Regulatory approval of pharmaceuticals without a randomized controlled study: analysis of EMA and FDA approvals 1999–2014. BMJ Open. 2016;6:e011666.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Sherman RE, Davies KM, Robb MA, Hunter NL, Califf RM. Accelerating development of scientific evidence for medical products within the existing US regulatory framework. Nat Rev Drug Discov. 2017;16:297–8.CrossRefPubMedGoogle Scholar
  5. 5.
    EMA. Final report on the adaptive pathways pilot. 28 July 2016. EMA/276376/2017.Google Scholar
  6. 6.
    Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet. 2017;. doi: 10.1016/S0140-6736(16)31592-6.Google Scholar
  7. 7.
    Smith R. The failings of NICE. Time to start work on version 2. BMJ. 2000;321:1363–4.CrossRefPubMedPubMedCentralGoogle Scholar
  8. 8.
    The National Institute for Health and Care Excellence. Accessed 6 Mar 2017.
  9. 9.
    Enzmann H, Norta M. Development of innovative medicines for European patients and impact of Brexit. Pharm Regul Aff. 2016. doi: 10.4172/2167-7689.1000171.Google Scholar
  10. 10.
    Institute for quality and efficiency in health care. General methods. Version 4.2 of 22 April 2015. Accessed 6 Mar 2017.
  11. 11.
    Staab T, Isbary G, Amelung VE, Ruof J. Inconsistent approaches of the G-BA regarding acceptance of primary study endpoints as being relevant to patients—an analysis of three disease areas: oncological, metabolic, and infectious diseases. BMC Health Serv Res. 2016;16:651.CrossRefPubMedPubMedCentralGoogle Scholar
  12. 12.
    de Bekker-Grob EW, Berlin C, Levitan B, Raza K, Christoforidi K, et al. Giving patients’ preferences a voice in medical treatment life cycle: the PREFER Public-Private Project. Patient. 2017;10(3):263–6.CrossRefPubMedGoogle Scholar
  13. 13.
    Haynes RB, Devereaux PJ, Guyatt GH. Clinical expertise in the era of evidence-based medicine and patient choice. EBM. 2002;7:36–8.Google Scholar
  14. 14.
    Eddy DM. Evidence-based medicine: a unified approach. Health Aff. 2005;24:9–17.CrossRefGoogle Scholar
  15. 15.
    Greenhalgh T, Howick J, Maskrey N. Evidence based medicine: a movement in crisis? BMJ. 2014;348:g3725.CrossRefPubMedPubMedCentralGoogle Scholar
  16. 16.
    Greenhalgh T, Snow R, Ryan S, Rees S, Salisbury H. Six biases against patients and carers in evidence-based medicine. BMC Med. 2015;13:200.CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press; 2003.Google Scholar
  18. 18.
    Deutsche Gesellschaft für Hämatologie und Medizinische Onkologie, DGHO. Pressemitteilung. Weiteres neues Krebsmedikament vom Markt genommen: Alle beteiligten Institutionen haben Recht, aber den Schaden haben die Patienten. Berlin, 3. November 2016. Accessed 6 Mar 2017.
  19. 19.
    Gemeinsamer Bundesausschuss (G-BA). Tragende Gründe zum Beschluss des Gemeinsamen Bundesausschusses über eine Änderung der Arzneimittel-Richtlinie (AM-RL): Anlage XII—Beschlüsse über die Nutzenbewertung von Arzneimitteln mit neuen Wirkstoffen nach § 35a SGB V—Crizotinib (neues Anwendungsgebiet: ROS1-positives, fortgeschrittenes nicht kleinzelliges Lungenkarzinom). 16. März 2017. Accessed 29 Mar 2017.

Copyright information

© The Author(s) 2017

Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (, which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Evelyn Schlegl
    • 1
  • Pierre Ducournau
    • 2
  • Jörg Ruof
    • 1
    • 3
    • 4
    Email author
  1. 1.HealthEcon LtdBaselSwitzerland
  2. 2.F. Hoffmann-La-Roche LtdBaselSwitzerland
  3. 3.Medical School of HanoverHanoverGermany
  4. 4.General Practitioner Office HugenschmidtLörrachGermany

Personalised recommendations