Useful Interplay Between Spontaneous ADR Reports and Electronic Healthcare Records in Signal Detection
- 1.6k Downloads
Background and Objective
Spontaneous reporting systems (SRSs) remain the cornerstone of post-marketing drug safety surveillance despite their well-known limitations. Judicious use of other available data sources is essential to enable better detection, strengthening and validation of signals. In this study, we investigated the potential of electronic healthcare records (EHRs) to be used alongside an SRS as an independent system, with the aim of improving signal detection.
A signal detection strategy, focused on a limited set of adverse events deemed important in pharmacovigilance, was performed retrospectively in two data sources—(1) the Exploring and Understanding Adverse Drug Reactions (EU-ADR) database network and (2) the EudraVigilance database—using data between 2000 and 2010. Five events were considered for analysis: (1) acute myocardial infarction (AMI); (2) bullous eruption; (3) hip fracture; (4) acute pancreatitis; and (5) upper gastrointestinal bleeding (UGIB). Potential signals identified in each system were verified using the current published literature. The complementarity of the two systems to detect signals was expressed as the percentage of the unilaterally identified signals out of the total number of confirmed signals. As a proxy for the associated costs, the number of signals that needed to be reviewed to detect one true signal (number needed to detect [NND]) was calculated. The relationship between the background frequency of the events and the capability of each system to detect signals was also investigated.
The contribution of each system to signal detection appeared to be correlated with the background incidence of the events, being directly proportional to the incidence in EU-ADR and inversely proportional in EudraVigilance. EudraVigilance was particularly valuable in identifying bullous eruption and acute pancreatitis (71 and 42 % of signals were correctly identified from the total pool of known associations, respectively), while EU-ADR was most useful in identifying hip fractures (60 %). Both systems contributed reasonably well to identification of signals related to UGIB (45 % in EudraVigilance, 40 % in EU-ADR) but only fairly for signals related to AMI (25 % in EU-ADR, 20 % in EudraVigilance). The costs associated with detection of signals were variable across events; however, it was often more costly to detect safety signals in EU-ADR than in EudraVigilance (median NNDs: 7 versus 5).
An EHR-based system may have additional value for signal detection, alongside already established systems, especially in the presence of adverse events with a high background incidence. While the SRS appeared to be more cost effective overall, for some events the costs associated with signal detection in the EHR might be justifiable.
KeywordsAcute Pancreatitis Signal Detection Acute Myocardial Infarction Proportional Reporting Ratio Anatomical Therapeutic Chemical Code
Overall, a spontaneous reporting system (SRS) is better suited to detection of signals than an electronic health record (EHR)-based system, especially for certain types of reactions (rare events and those with a high drug-attributable risk).
Use of EHRs might be justifiable in some situations where SRSs perform poorly (e.g. outcomes with a high background incidence), provided that the additional costs can be taken into account.
SRSs and EHR-based signal detection systems can be complementary, the additional value of one to the other varying across events, as a function of the background incidence of the event.
Spontaneous reporting systems (SRSs) have long been the foundation of post-marketing drug safety surveillance . Despite their broad utilization, information found in such systems is sometimes limited  and, as a consequence, decisions based solely on data from these systems often need to take into account significant uncertainties . In many instances, it is not sufficient to use a single source of information to understand a particular drug safety issue , and there is a recognized need to put together most, if not all, available relevant sources in an efficient way.
Following the drug safety concerns surfacing between 2004 and 2007, many discussions started as to whether we can have a more pro-active approach to signal detection instead of relying on passive surveillance systems. In both Europe and the USA, it was explored whether electronic healthcare record (EHR) databases, which comprise detailed data collected longitudinally and routinely in actual care for large-scale populations , may be used for post-marketing safety surveillance. EHRs have been primarily used for signal evaluation studies; however, in recent years, various projects have explored ways of using them as an additional source for signal detection systems, e.g. the Observational Medical Outcomes Partnership (OMOP) [6, 7], Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium (PROTECT)  and exploring and understanding adverse drug reactions (EU-ADR) [9, 10].
To date, only two studies [11, 12] have tried to combine both sources in order to support the signal detection process, while the majority of the available research has focused rather on comparing the two systems in terms of overall performance and usefulness [13, 14, 15, 16].
The aim of this study was to investigate in which particular situations EHR-based signal detection systems may add value to already existing SRSs, focusing on a limited set of adverse events of considerable importance in pharmacovigilance. To express this added value, we used performance indicators, including percentages of unilaterally identified signals and sensitivity to describe the ‘gains’, as well as the number needed to detect (NND), for the ‘costs’ associated with signal detection.
A signal detection strategy focused on a limited set of adverse events was performed retrospectively in two database systems—(1) EU-ADR and (2) EudraVigilance—from 1 January 2000 to 1 January 2010. These are described separately below. The two systems were considered individually, and the most sound event definitions possible and implementation of signal detection methods were taken into account in each database independently in order to optimize the performance of each one. In this study, we used the term ‘signal’ to refer to a signal of disproportionate reporting (SDR), as defined by the Council for International Organizations of Medical Sciences (CIOMS) Working Group VIII , in the context of signal detection in EudraVigilance and equivalent to a statistically significant drug–adverse event association that met a specific threshold of increased risk in the context of EU-ADR. All drugs captured in either of the two systems were considered. Drugs not identifiable at the fifth level of the WHO Anatomical Therapeutic Chemical (ATC) classification system, as well as herbal supplements, were excluded.
2.2 Events of Interest
We considered the following five events, selected from a list of events previously identified as important on the basis of expert judgment and predefined criteria : (1) acute myocardial infarction (AMI), (2) bullous eruption, (3) hip fracture, (4) acute pancreatitis and (5) upper gastrointestinal bleeding (UGIB). These events were chosen because of their diversity in aetiology, background incidence and drug-attributable risk attributes, which we considered might have an impact on the performance of the two systems. To investigate the possible correlation between the signal detection performance of each system and the frequency of the events, we ranked our events of interest according to the empirically determined background frequency (i.e. the incidence rate in the general population). These incidence rates were derived from the EU-ADR network in order to maintain the same base population across events, which allowed for a more reasonable comparison .
2.3 The SRS: EudraVigilance
As the exemplar for an SRS, we used EudraVigilance, a web-based information system launched in December 2001 and designed to manage information on suspected adverse drug reactions (ADRs) that are reported for drugs licensed in Europe. The total number of individual reports as of December 2013 was 4.5 million, with 38 % of cases originating from the European Union and 62 % from the rest of the world .
2.3.1 Capturing Events of Interest
In EudraVigilance, suspected ADRs are coded using the Medical Dictionary for Regulatory Activities (MedDRA®), an international medical terminology system developed under the auspices of the International Conference on Harmonisation (ICH) . For capturing the events of interest, we used adapted searches derived ad hoc from standardized MedDRA® queries (SMQs),  similar to the approach used by Patadia et al. .
2.3.2 Method of Signal Detection
Signal detection in EudraVigilance was performed using the proportional reporting ratio (PRR) method , previously validated in this database by Alvarez et al. . Only cases received within the study period (from 1 January 2000 to 1 January 2010) were considered for identification of signals.
The threshold chosen to define a signal was a lower limit of the confidence interval of the PRR greater than 1, together with at least three cases being reported with the investigated association . No further adjustment was done for possible confounding variables.
2.4 The EHR-Based System: EU-ADR
As the exemplar for an EHR-based system, we used EU-ADR, a computerized system designed to detect potential ADRs and built on a network of established databases from various European countries . Data from seven databases in three countries (Denmark, Italy and The Netherlands) were used in this study. EU-ADR includes both population-based primary care databases (the Integrated Primary Care Information [IPCI] database [The Netherlands] and Health Search/CSD Patient and Pedianet [Italy]) and record-linkage systems (the Aarhus University Hospital Database [Denmark], the PHARMO Network [The Netherlands] and the regional Italian claims databases of Lombardy and Tuscany). The source population covered by the database network is approximately 20 million patients. Drug exposure in EU-ADR was identified from prescription or dispensing data (depending on the database), using ATC codes. Prescriptions with the same ATC code where the start date of one prescription preceded the end date of the other prescription were merged into a single episode of drug use, starting at the beginning of the first prescription and ending at the end of the last prescription. Periods of concomitant drug use were labelled as separate episodes. Only current exposure (within 30 days of an event of interest) was considered . The characteristics of the EU-ADR network have been extensively described elsewhere [9, 24].
2.4.1 Capturing Events of Interest
Definitions for each event of interest were previously constructed by a team of experts and, on the basis of those, queries were performed in each database in the network, using the corresponding diagnosis coding schemes—the International Classification of Diseases (ICD) 9th and 10th revisions, and the International Classification of Primary Care—supplemented with additional criteria such as laboratory values and unstructured free-text searches, where applicable. The results were subsequently pooled across all databases . The events AMI and UGIB had previously been validated in the databases concerned [26, 27].
2.4.2 Method of Signal Detection
For EU-ADR, a signal detection method specifically developed for EHR data was used: the Longitudinal Gamma Poisson Shrinker (LGPS) . LGPS is a cohort-based method, adapted from a Bayesian method (DuMouchel’s Gamma Poisson Shrinker, designed for use in an SRS [see the Electronic Supplementary Material]), which uses person-time rather than case counts for estimation of the expected number of events. Previous evaluation against other signal detection methodologies showed that LGPS is the best-performing method in this database system . We applied a threshold to the LGPS risk ratio (RRLGPS) of a lower limit of the 95 % credible interval >1 . After LGPS, we applied a second method, Longitudinal Evaluation of Observational Profiles of Adverse events Related to Drugs (LEOPARD), which adjusts for possible protopathic bias and improves performance. LEOPARD is based on comparison of rates of drug prescriptions initiated within a fixed time window (±25 days) prior to and after the occurrence of an event, on the basis of the assumption that an increase in the number of prescriptions started after an event relative to the number of prescriptions started prior to the event is an indication of protopathic bias. From a statistical perspective, this is a binomial test, which compares the distributions of prescriptions across those two time windows .
2.5 Verification of Signals
We reviewed the currently available literature in order to determine which drug–adverse event associations identified in the dataset represented already known associations. In contrast to the approach used in previous studies, verification was performed for all drug–event associations, irrespective of whether a potential signal was flagged or not by any of the signal detection methods used in either EudraVigilance or EU-ADR.
For the purposes of this study, drug–event associations that were suggested to be ADRs according to the criteria described above were assumed to be ADRs, otherwise these associations were assumed to be non-ADRs.
2.6 Performance Indicators
In order to assess the complementarity of the systems and to calculate the costs associated with identification of potential signals from different sources, we used the following indicators:
Percentage of unilaterally identified signals is a variation of sensitivity (recall) metrics, which uses as the numerator the number of true associations identified in one system that were not identified in the other. We considered this variable to be useful in quantifying the incremental value of each system.
Sensitivity was calculated as the proportion of associations correctly identified by the method out of the total pool of ADRs known to be true from the literature .
Number needed to detect (NND), originally described by Hauben et al.  in the context of signal detection within an SRS, was used as a proxy to express the costs associated with each source of signals. This represents the number of signals that would have to be reviewed to detect a single signal that was proven to be true.
Spearman’s correlation coefficient (R value) was calculated to determine whether there was a correlation between the background incidence of the events and each system’s capability to detect signals.
The therapeutic classes comprising the majority of potential signals identified in EudraVigilance were agents acting on the renin–angiotensin system, antivirals for systemic use and antithrombotic agents, while for EU-ADR they were anti-asthmatics, psychoanaleptics and antiepileptics (see Fig. 2). The percentages of potential signals identified in both systems ranged from 2 to 24 %.
In this study, we aimed to investigate an additional value of the EHR-based system for signal detection on top of the traditional SRS. We focused on five different adverse events deemed to be important in pharmacovigilance: bullous eruption, AMI, acute pancreatitis, hip fracture and UGIB.
Although EudraVigilance identified more signals overall than EU-ADR—41 % of signals (187/458) compared with 32 % (147/458)—as previously shown in other studies , this was not unexpected, considering that EudraVigilance has worldwide coverage, whereas EU-ADR covers only three countries. Furthermore, the EudraVigilance system is primarily designed for signal detection.
Patadia et al.  performed a time-restricted analysis (before and after a safety communication/media attention) and showed that this has an impact on the numbers of signals detected in both data sources in opposite ways (i.e. an increase in the number of signals in the SRS after media attention and a decrease in the EHR). While this effect might partially explain our findings of higher sensitivity for EudraVigilance, we consider it unlikely that it entirely explains the difference, since the majority of signals that were tested did not attract media attention.
The capacity of EU-ADR and EudraVigilance to detect signals was shown to differ depending on the nature of the adverse event being investigated. The relatively poor performance of EudraVigilance in detecting hip fractures and AMI might be due to the fact that both events are not usually perceived as being drug induced and thus often fail to be recognized and reported as ADRs, as has previously been hypothesized . The suspected ADRs documented in an SRS such as EudraVigilance are highly dependent on the reporter’s ability to recognize them as such, and some characteristics are helpful in this respect: biological concordance with the drug mechanism of action, a short time to onset, a positive dechallenge and lack of alternative causes. The adverse events that are not so obviously attributed to drugs (because they are multifactorial), or that already have a high background incidence, are likely to be poorly captured by spontaneous reports [1, 3]. EHR-based systems do not rely on reporter judgment; therefore, these events may be better represented in such systems.
On the other hand, there is very low sensitivity for detecting rare drug-induced events in EU-ADR, and this is in line with previous research, which showed that, despite the broad coverage of the EU-ADR network (around 20 million patients), there is simply not enough (statistical) power to identify very rare events in the database . Thus, for rare events that have drug treatment as their primary aetiology, SRSs still seems to be the better solution so far. Our findings are consistent with those of Patadia et al. , although different parameters were used to determine the complementarity of the two systems.
We found a correlation between the background incidence of the events and the contribution of each database to signal detection; the correlation was statistically significant for EudraVigilance but not for EU-ADR, possibly because of the small number of events tested.
The burden associated with screening any data source for signals depends on the number of signals that require further assessment or investigation and the workload involved in each of these investigations. The amount of work needed to confirm or refute a signal is highly variable, ranging from simple product information checks to more complex analyses and formal pharmacoepidemiological studies. In their study, Pizzoglio et al.  reported a median time of 6 hours for initial assessment of a signal (range 2–26 h). We did not collect similar information on the time spent on assessment of the signals in our study, since a semi-automatic method was used; however, we considered the number of signals that needed further investigation to be a reasonable proxy to express the associated costs. We found that for all of the events (with the exception of bullous eruption), it was more costly to identify signals within the EHR-based system. However there was a notable difference across the events as follows: for hip fracture, where EU-ADR provided the most added value, an 80 % increase in the cost per signal was observed in comparison with detection in EudraVigilance. On the other hand, for acute pancreatitis, the cost associated with signal detection in EU-ADR was not much higher than that in EudraVigilance; therefore, in this case, it may be justifiable to use both systems. Because the two systems provided roughly the same contribution to detection of UGIB-related signals, the value of supplementing SRSs with EHR-based systems is probably dependent on the types of drugs that can be captured in the particular EHR database. Detecting AMI proved to be costly in both systems; however, the seriousness of the event, its public health impact and the difficulty of detection might justify the extra cost of using both an SRS and an EHR-based system. For bullous eruption, because of the lack of an additional gain (only one extra signal was identified), it might not be efficient to use the EHR as a secondary signal source.
The range of events tested in this study, albeit carefully selected, represented only a small sample of all possible adverse events and therefore limits the external validity of this research. The applicability of our findings to a broader range of events will require further investigation. In addition, the overall background incidence of the events that were selected was slightly skewed towards more common events and, if our hypothesis holds true, this might have resulted in a bias favouring EHR performance.
Another limitation was the retrospective nature of the study: we actually tested the capacity of systems not to identify signals (i.e. new associations) but rather to detect already identified safety issues. As Norén et al.  pointed out, ideally the evaluation should be done using emerging safety issues and not well-established ADRs. However, this is a common limitation in signal detection research because of the difficulties that accompany building a ‘reference standard’ and the long time needed to gather data prospectively.
Moreover, combining the two systems involved many decisions regarding the choice of the signal detection method and their implementation (e.g. the choice of thresholds and precision estimates), which had a huge impact on the results . In addition, it was repeatedly demonstrated that the aggregation level at which data mining is performed influences the results [38, 39], and we consider that the appropriate level is still an open question. Nevertheless, the decision to use SMQs instead of preferred terms (PTs) may have had an impact on the results. We acknowledge that use of different signal detection methodologies and reference standards, as well as definitions of events used in data mining, might lead to different results. Therefore, our results might, to some extent, reflect the relative strengths of each data source, but they would also be influenced by the algorithms that were chosen. Although we acknowledge that the time to signal detection is an important element to take into account, for this study we focused primarily on other quantitative measures, such as the number of signals and the number of false positives, which are equally important.
This evaluation did not take into account the fact that an EHR-based surveillance system would require additional work for implementation and subsequent maintenance for the purposes of signal detection (versus SRSs, which are already established for such activities) and would thus incur extra costs, which would be difficult to estimate.
Our study is one of the few studies performed so far that have tried to explore how an SRS and an EHR-based system might be used together with the aim of augmenting drug safety surveillance. A previous study by Harpaz et al.  had a similar aim but a different strategy, trying to combine information from both data sources at a very early stage in order to improve the ranking of signals by replication of findings. In contrast, we explored scenarios where use of EHRs can fill the gaps and add value to already existing systems.
The more prudent goal in signal detection is identification not of all signals but of the majority of signals in the most efficient way—with the least time- and resource-consuming approach. With this aim in mind, we showed that an EHR-based system may complement an SRS in certain situations, especially in the presence of adverse events with a high background incidence. While the SRS appeared to be more cost effective overall, for some events that are very hard to pick up, the costs associated with additional signal detection in an EHR-based system may be justifiable.
The authors would like to thank Dr. Jim Slattery of the European Medicines Agency for providing insightful comments on the paper, and Dr. Paul Avillach of the Center for Biomedical Informatics at Harvard University for providing support on construction of the reference set.
Compliance with Ethical Standards
The European Commission’s Seventh Framework Programme (FP7/2007–2013) funded the EU-ADR Project under Grant No. 215847. The funding agency had no role in the design and conduct of the study; the collection and management of the data; the analysis or interpretation of the data; and the preparation, review or approval of the manuscript.
Conflicts of interest
Giampiero Mazzaglia has become an employee of the European Medicines Agency since contributing to the EU-ADR Project. Peter Arlett is the Head of the Pharmacovigilance Department at the European Medicines Agency. Ron Herings is an employee of the PHARMO Institute, which performs financially supported studies for government and related healthcare authorities and several pharmaceutical companies. Miriam Sturkenboom heads a research group that conducts unconditional research on the safety and effects of drugs for several pharmaceutical companies (AstraZeneca, Pfizer and Eli Lilly), none of which is related to this current research. Alexandra C. Pacurariu, Sabine M. Straus, Gianluca Trifirò, Martijn J. Schuemie, Rosa Gini, Gino Picelli, Lorenza Scotti, Lars Pedersen, Johan van der Lei and Preciosa M. Coloma have no conflicts of interest that are directly relevant to the content of this study.
- 1.CIOMS Working Group. Practical aspects of signal detection in pharmacovigilance: report of CIOMS Working Group VIII. Geneva: Council for International Organizations of Medical Sciences; 2010.Google Scholar
- 7.Observational Medical Outcomes Partnership [OMOP]. Available from: http://omop.fnih.org/. Accessed 22 Mar 2014.
- 8.Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium [PROTECT]. Available from: http://www.imi-protect.eu/. Accessed 22 Mar 2014.
- 10.EU-ADR webpage. Available from: https://bioinformatics.ua.pt/euadr/. Accessed 22 Mar 2014.
- 18.European Medicines Agency. Annual report on EudraVigilance for the European Parliament, the Council and the Commission. European Medicines Agency. 2014. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Report/2014/04/WC500165780.pdf. Accessed 22 Mar 2014.
- 19.International Federation of Pharmaceutical Manufacturers & Associations. Introductory guide: MedDRA version 15.1 [MSSO-DI-6003-15.1.0]. International Federation of Pharmaceutical Manufacturers & Associations. 2012. Available from: http://www.meddra.org/sites/default/files/guidance/file/intguide_15_1_English_0.pdf. Accessed 28 Aug 2015.
- 20.International Federation of Pharmaceutical Manufacturers & Associations. Introductory guide for standardised MedDRA queries (SMQs) version 15.0. International Federation of Pharmaceutical Manufacturers & Associations, 2012. 2012. Available from: http://www.meddra.org/sites/default/files/guidance/file/smq_intguide_15_0_english.pdf. Accessed 28 Aug 2015.
- 26.Valkhoff VE, Coloma PM, Masclee GMC, Gini R, Innocenti F, Lapi F, EU-ADR Consortium, et al. Validation study in four health-care databases: upper gastrointestinal bleeding misclassification affects precision but not magnitude of drug-related upper gastrointestinal bleeding risk. J Clin Epidemiol. 2014;67:921–31.CrossRefPubMedGoogle Scholar
- 27.Coloma PM, Valkhoff VE, Mazzaglia G, Nielsson MS, Pedersen L, Molokhia M, EU-ADR Consortium, et al. Identification of acute myocardial infarction from electronic healthcare records using different disease coding systems: a validation study in three European countries. BMJ Open. 2013;3(6). doi: 10.1136/bmjopen-2013-002862 (pii: e002862).
- 34.Patadia VK, Coloma P, Schuemie MJ, Herings R, Gini R, Mazzaglia G, EU-ADR Consortium, et al. Using real-world healthcare data for pharmacovigilance signal detection—the experience of the EU-ADR Project. Expert Rev Clin Pharmacol. 2015;8:95–102.Google Scholar
- 35.Pizzoglio V, Ahmed I, Auriche P, Tuber-Bitter P, Haramburu F, Kreft-Jaïs C, with French Network of Pharmacovigilance Centres, et al. Implementation of an automated signal detection method in the French pharmacovigilance database: a feasibility study. Eur J Clin Pharmacol. 2012;68:793–9.Google Scholar
- 39.Hill R, Hopstadius J, Lerch M, Norén N. An attempt to expedite signal detection by grouping related adverse reaction terms. In: 24th European Medical Informatics Conference (MIE 2012): Pisa; August 2012.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.