1 Introduction

There has been a recent increase in interest in the concept of ‘real world’ evidence (RWE) with the increasing digitisation of health care records and the ability, through increased computing power and interconnectivity, to analyse what is known as ‘big data’. The purpose of this brief commentary is to consider the benefits and drawbacks that are present within RWE, to examine how such data are being used and what might, in practice, limit its usefulness to healthcare professionals (HCPs).

2 Terminology

The first point of confusion often arises when trying to distinguish between real world data (RWD) and RWE. Real world data can be derived from numerous sources (for example Electronic Health Records [EHR], claims and billing data, product and disease registries or increasingly data gathered from mobile devices) and this makes it hard to define as a neat concept, and therefore most definitions have tended to focus on what RWD isn’t rather than what it is. A good example is the definition offered in the 21st Century Cures Act, which defines it as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than traditional clinical trials” [1]. Real world evidence, on the other hand, is the interpretation which we put upon such data which explains its relevance, and hence value, to the real world of clinical decision making.

3 RWE: Current Use

As pressure on healthcare budgets has increased payers are increasingly forced to restrict access to drugs, often citing the lack of clear and compelling data on clinical/cost-effectiveness as the justification for their decisions. There is also an increasing emphasis on recognising the importance of patient-related outcomes rather than the more narrow traditional focus on clinical efficacy and safety.

The potential of RWE to impact upon how we deliver healthcare has raised interest from a wide variety of groups, ranging from international and national regulatory bodies such as the European Medicines Agency (EMA) and the Food and Drug Administration (FDA), through to societies such as the American Society of Clinical Oncology (ASCO) down to the levels of individual health plans and patient groups. Health Technology Assessment bodies including the National Institute for Health and Care Excellence (NICE) have also started to use RWE in risk-sharing schemes and the UK’s National Health Service (NHS) has established the Cancer Drugs Fund which gathers RWE on what it regards as promising new treatments which still lack what has traditionally been viewed as adequate evidence to support use and reimbursement.

One of the most prominent exponents of the use of RWE is the pharmaceutical industry. There is a recognition that payers are interested in using RWD/RWE not only as a basis for decision making (where RWE can address issues around the uncertainty of the evidence base) but also potentially increase its use in pricing and reimbursement A recent report published by Deloitte has highlighted the pharmaceutical industry’s belief in the value of using RWE and proposing a number of measures against which its use can be judged. These include the number of RWE publications produced, increased use in Health Technology Assessment (HTA) and regulatory decision making and use in ‘value-based ‘contracting of the kind we have seen in various ‘risk-sharing’ schemes where the manufacturers are only reimbursed if the drug achieves specified clinical results within a certain timeframe [2].

4 RWE: Strengths and Weaknesses

One of the key strengths of RWE is that we have actually been using it successfully in healthcare for decades, but since we tended not to call it RWE many people are unaware of its long history. In a study published in 2005 the authors noted that, when examining the types of studies published in 25 clinical journals, the number of non-randomised controlled trials published had been consistently higher than the number of Randomised Controlled Trials (RCTs) published over the same period (15.4% vs 8.4%) [3]. As noted above, non-randomised trails, which can include a range of different study designs, for example case series, case–control studies, cohort studies would all now be considered to meet the kind of definitions of RWE we are now using. We have also successfully used RWE in pharmacovigilance, for example the European Union’s Drug Regulating Authorities Pharmacovigilance (EUDRAVIGILANCE) network, which monitors adverse event reporting receives over 1 million reports each year [4] and the FDA has explicitly noted its long history of prior use of RWE with the Sentinel System which has data on over 100 million individuals [5].

Its other strengths include the wide range and number of patients that provide the data on which RWE is based, unlike RCTs where there are carefully imposed inclusion and exclusion criteria, and the ease with which, thanks to new technology the data, can be gathered. Using Apple’s Research Kit, Stanford University recruited 11,000 iPhone users in 24 h to participate in a cardiovascular study, a rate of recruitment which they estimate would take a year and require 50 centres in a traditional clinical study [6]. This ties into the final major strength of RWE, that it has the potential to quickly fill gaps in the evidence base which until now have required waiting for RCTs to be completed and their results interpreted and then included in clinical guidelines.

RWE does also have significant weaknesses. Some of these are linked to the study design (in particular the fact that in many RWE studies the patients have not been randomly allocated), but some have arisen simply through our long acceptance of the RCT as being the ‘gold standard’ basis of decision-making. The very nature of those real world studies which use a non-randomised design means that the biases which are eliminated in RCTs are present in such RWD and these biases need to be accounted for, often using statistical techniques such as propensity matching. In addition, in much RWE there is a suspicion that the researchers, often representing a pharmaceutical company, have simply kept scanning through the data until they found something, even if it wasn’t what they were looking for originally and then present this as evidence. There are also issues around the ownership of RWD. In a RCT funded by a company it is usually very clear who owns the data generated in the study, but this is much less obvious in RWD and this, coupled with issues of individual’s rights to privacy, limits which data can be accessed (for further details see for example Asche et al. 2017) [7]. A second weakness is that this data is still not widely accepted by regulatory bodies, either in terms of submissions for licencing requests or for use in promotional materials, where it might not meet the requirement to be considered ‘substantial evidence’, although this weakness might perhaps simply reflect the currently developing nature of RWE and the conservatism of decision makers rather than an underlying weakness in the data per se. Perhaps the most serious weakness of RWE in the eyes of HCPs and decision makers is not simply in its design but rather in history. We have become used to a system in which the RCT has been seen as the preferred methodology for conducting clinical studies for over 50 years, and with the advent of evidence-based medicine created an ‘hierarchy of evidence’ which has placed many of the study designs which might be categorised as generating RWE at the lowest levels of the pyramid, encouraging people to view this evidence as in some way being inferior to that collected by RCTs, systematic reviews and meta analyses. Whist acknowledging the undoubted strengths of the RCT this has led to people tending to view other designs as in some way being inherently inferior, although in many circumstances they can be the most appropriate, or indeed only ethical or feasible alternative.

5 RWE: Its Use in Practice

It is true that there are an ever increasing number of publications presenting RWD/RWE (930 publications in 2018 used RWE or RWD in either the title or abstract on PubMed, compared to 501 in 2017 and 326 in 2016) but there is very little evidence that it is having a significant impact on decision making. [8].

There is, however, evidence emerging of a far more serious impediment to the use of RWE in decision making and that is the sheer lack of knowledge among HCPs as to how to understand and interpret RWD and RWE. The lack of statistical literacy among HCPs has been frequently identified as an issue which is common over both time and geography [9,10,11,12]. This shouldn’t really be terribly surprising as a finding since they have generally been trained in some aspect of the practice of medicine and not trained in statistics, but as noted above, we are required to use even more obscure (to the non-statistician) statistical techniques to account for potential biases/confounding (often caused by the un-matched patients) within RWD. This illiteracy has clearly been shown to hamper the use of RWE [8, 13]. As one of the interviewees’ notes “If you cannot explain this process to clinicians in a way that confirms its accuracy…you will find that, to them, such data has no value.” Another survey of US decision-makers confirms this difficulty; whilst 60% of respondents reported that RWE delivered information in diverse populations only ‘sometimes’ (which would seem to suggest that one of the supposed benefits of RWE was perhaps less apparent) only 24% were confident in their ability to interpret results [8].

If we couple this difficulty in understanding with the worldwide prevalence of defensive medicine (the perception that HCPs can be sued by patients or relatives if they have made a presumed medical error) that has been well documented (see for example Berlin 2017 or Vento et al. 2018) [14, 15], then there can be little hope that the findings from RWE are likely to be affecting clinical practice in anything but the most limited way, since why would anyone change their behaviour on the basis of something they don’t understand if they feared the end result might expose them to litigation? If these new findings were to be incorporated into guidelines then clearly this problem could be minimised; however, this returns us to the problem that current guideline coverage is relatively low (in terms of the number of treatments/procedures covered) and, until we can utilise the undoubted potential of RWE to expand guideline coverage, is likely to remain so for the immediate future.

If we continue as we are doing currently, simply producing more publications that are clearly not being seen as comprehensible to the majority of practitioners, then we are simply expending effort and resource in a manner which is not helpful and RWE isn’t helping to improve healthcare to the extent that it is capable of.

Therefore, if we are serious in our beliefs that RWE can help make a difference to how we practise healthcare then we need to do much more than simply publish results, important though this is. We have to be willing to invest in much more basic education of the audience to help them understand how RWE and the RCT can happily co-exist and complement each other, how these studies are conducted and how the statistical matching techniques help them to have more, rather than less, faith in the results. We also need to work with journals to ensure that they understand that publications featuring what they might consider to be ‘non-traditional’ designs can have as much value as those based on more conventional randomised studies and that the traditional reliance on the hierarchy of evidence may be hampering, rather than assisting, our understanding.

If we fail to do this then we are likely to be providing RWE that supports the famous quote, often erroneously attributed to Albert Einstein, “insanity is doing the same thing over and over again and expecting a different result”.