Data Mining and Knowledge Discovery

, Volume 14, Issue 3, pp 305–328 | Cite as

Duplicate detection in adverse drug reaction surveillance

  • G. Niklas NorénEmail author
  • Roland Orre
  • Andrew Bate
  • I. Ralph Edwards


The WHO Collaborating Centre for International Drug Monitoring in Uppsala, Sweden, maintains and analyses the world’s largest database of reports on suspected adverse drug reaction (ADR) incidents that occur after drugs are on the market. The presence of duplicate case reports is an important data quality problem and their detection remains a formidable challenge, especially in the WHO drug safety database where reports are anonymised before submission. In this paper, we propose a duplicate detection method based on the hit-miss model for statistical record linkage described by Copas and Hilton, which handles the limited amount of training data well and is well suited for the available data (categorical and numerical rather than free text). We propose two extensions of the standard hit-miss model: a hit-miss mixture model for errors in numerical record fields and a new method to handle correlated record fields, and we demonstrate the effectiveness both at identifying the most likely duplicate for a given case report (94.7% accuracy) and at discriminating true duplicates from random matches (63% recall with 71% precision). The proposed method allows for more efficient data cleaning in post-marketing drug safety data sets, and perhaps other knowledge discovery applications as well.


Data cleaning Duplicate detection Hit-miss model 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, De Freitas RM (1998) A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol 54:315–321CrossRefGoogle Scholar
  2. Belin T, Rubin D (1995) A method for calibrating false-match rates in record linkage. J Am Stat Assoc 90: 694–707zbMATHCrossRefGoogle Scholar
  3. Bilenko M, Mooney RJ (2003a) Adaptive duplicate detection using learnable string similarity measures. In: KDD ’03: proceedings of the 9th ACM SIGKDD international conference on knowledge discovery and data mining. ACM Press, New York, NY, USA, pp 39–48Google Scholar
  4. Bilenko M, Mooney RJ (2003b) On evaluation and training-set construction for duplicate detection. In: Proceedings of the KDD-2003 workshop on data cleaning, record linkage and object consolidation, pp 7–12Google Scholar
  5. Bortnichak EA, Wise RP, Salive ME, Tilson HH (2001) Proactive safety surveillance. Pharmacoepidemiol Drug Safety 10:191–196CrossRefGoogle Scholar
  6. Brinker AD, Beitz J (2002) Spontaneous reports of thrombocytopenia in association with quinine: clinical attributes and timing related to regulatory action. Am J Hematol 70:313–317CrossRefGoogle Scholar
  7. Copas J, Hilton F (1990) Record linkage: statistical models for matching computer records. J R Stat Soc: Sers A 153(3):287–320CrossRefGoogle Scholar
  8. De Veaux RD, Hand DJ (2005) How to lie with bad data. Stat Sci 20(3):231–238zbMATHCrossRefGoogle Scholar
  9. Edwards IR (1997) Adverse drug reactions: finding the needle in the haystack. Br Med J 315(7107):500Google Scholar
  10. Edwards IR (1999) Spontaneous reporting – of what? Clinical concerns about drugs. Br J Clin Pharmacol 48(2):138–141CrossRefGoogle Scholar
  11. Edwards IR, Aronson JK (2000) Adverse drug reactions: definitions, diagnosis and management. Lancet 356(9237):1255–1259CrossRefGoogle Scholar
  12. Evans SJW (2000) Pharmacovigilance: a science or fielding emergencies? Stat Med 19(23):3199–3209CrossRefGoogle Scholar
  13. Fayyad U, Piatetsky-Shapiro G, Smyth P (1996) The KDD process for extracting useful knowledge from volumes of data. Commun ACM 39(11):27–34CrossRefGoogle Scholar
  14. Fellegi IP, Sunter AB (1969) A theory for record linkage. J Am Stat Assoc 64:1183–1210CrossRefGoogle Scholar
  15. Hernández MA, Stolfo SJ (1998) Real-world data is dirty: data cleansing and the merge/purge problem. Data Min Knowl Discov 2(1):9–37CrossRefGoogle Scholar
  16. Jaro M (1989) Advances in record-linkage methodology as applied to matching the 1985 census of Tampa, Florida. J Am Stat Assoc 84:414–420CrossRefGoogle Scholar
  17. Kim WY, Choi B-J, Hong EK, Kim S-K, Lee D (2003) A taxonomy of dirty data. Data Min Knowl Discov 7(1):81–99CrossRefMathSciNetGoogle Scholar
  18. Lindquist M (2004) Data quality management in pharmacovigilance. Drug Safety 27(12):857–870CrossRefGoogle Scholar
  19. Monge AE, Elkan C (1997) An efficient domain-independent algorithm for detecting approximately duplicate database records. Research issues on data mining knowledge discovery, Tucson, AZGoogle Scholar
  20. Newcombe HB, Kennedy JM (1962) Record linkage: making maximum use of the discriminating power of identifying information. Commun ACM 5(11):563–566CrossRefGoogle Scholar
  21. Nkanza JN, Walop W (2004) Vaccine associated adverse event surveillance (VAAES) and quality assurance. Drug Safety 27:951–952Google Scholar
  22. Norén GN, Bate A, Orre R, Edwards IR (2006) Extending the methods used to screen the WHO drug safety database towards analysis of complex associations and improved accuracy for rare events. Stat Med 25(21): 3740–3757CrossRefMathSciNetGoogle Scholar
  23. Norén GN, Orre R, Bate A (2005) A hit-miss model for duplicate detection in the WHO drug safety database. In: KDD ’05: proceeding of the 11th ACM SIGKDD international conference on knowledge discovery and data mining. ACM Press, New York, NY, USA, pp 459–468Google Scholar
  24. Orre R, Lansner A, Bate A, Lindquist M (2000) Bayesian neural networks with confidence estimations applied to data mining. Comput Stat Data Anal 34:473–493zbMATHCrossRefGoogle Scholar
  25. Rawlins MD (1988) Spontaneous reporting of adverse drug reactions. II: Uses. Br J Clin Pharmacol 1(26):7–11MathSciNetGoogle Scholar
  26. Sarawagi S, Bhamidipaty A (2002) Interactive deduplication using active learning. In: KDD ’02: proceedings of the 8th ACM SIGKDD international conference on knowledge discovery and data mining. ACM Press, New York, NY, USA, pp 269–278Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  • G. Niklas Norén
    • 1
    • 2
    Email author
  • Roland Orre
    • 3
  • Andrew Bate
    • 1
  • I. Ralph Edwards
    • 1
  1. 1.WHO Collaborating Centre for International Drug MonitoringUppsalaSweden
  2. 2.Mathematical StatisticsStockholm UniversityStockholmSweden
  3. 3.NeuroLogic Sweden ABStockholmSweden

Personalised recommendations