Data Mining and Knowledge Discovery

, Volume 14, Issue 3, pp 305–328

Duplicate detection in adverse drug reaction surveillance

  • G. Niklas Norén
  • Roland Orre
  • Andrew Bate
  • I. Ralph Edwards
Article

Abstract

The WHO Collaborating Centre for International Drug Monitoring in Uppsala, Sweden, maintains and analyses the world’s largest database of reports on suspected adverse drug reaction (ADR) incidents that occur after drugs are on the market. The presence of duplicate case reports is an important data quality problem and their detection remains a formidable challenge, especially in the WHO drug safety database where reports are anonymised before submission. In this paper, we propose a duplicate detection method based on the hit-miss model for statistical record linkage described by Copas and Hilton, which handles the limited amount of training data well and is well suited for the available data (categorical and numerical rather than free text). We propose two extensions of the standard hit-miss model: a hit-miss mixture model for errors in numerical record fields and a new method to handle correlated record fields, and we demonstrate the effectiveness both at identifying the most likely duplicate for a given case report (94.7% accuracy) and at discriminating true duplicates from random matches (63% recall with 71% precision). The proposed method allows for more efficient data cleaning in post-marketing drug safety data sets, and perhaps other knowledge discovery applications as well.

Keywords

Data cleaning Duplicate detection Hit-miss model 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  • G. Niklas Norén
    • 1
    • 2
  • Roland Orre
    • 3
  • Andrew Bate
    • 1
  • I. Ralph Edwards
    • 1
  1. 1.WHO Collaborating Centre for International Drug MonitoringUppsalaSweden
  2. 2.Mathematical StatisticsStockholm UniversityStockholmSweden
  3. 3.NeuroLogic Sweden ABStockholmSweden

Personalised recommendations