Abstract
The WHO Collaborating Centre for International Drug Monitoring in Uppsala, Sweden, maintains and analyses the world’s largest database of reports on suspected adverse drug reaction (ADR) incidents that occur after drugs are on the market. The presence of duplicate case reports is an important data quality problem and their detection remains a formidable challenge, especially in the WHO drug safety database where reports are anonymised before submission. In this paper, we propose a duplicate detection method based on the hit-miss model for statistical record linkage described by Copas and Hilton, which handles the limited amount of training data well and is well suited for the available data (categorical and numerical rather than free text). We propose two extensions of the standard hit-miss model: a hit-miss mixture model for errors in numerical record fields and a new method to handle correlated record fields, and we demonstrate the effectiveness both at identifying the most likely duplicate for a given case report (94.7% accuracy) and at discriminating true duplicates from random matches (63% recall with 71% precision). The proposed method allows for more efficient data cleaning in post-marketing drug safety data sets, and perhaps other knowledge discovery applications as well.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, De Freitas RM (1998) A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol 54:315–321
Belin T, Rubin D (1995) A method for calibrating false-match rates in record linkage. J Am Stat Assoc 90: 694–707
Bilenko M, Mooney RJ (2003a) Adaptive duplicate detection using learnable string similarity measures. In: KDD ’03: proceedings of the 9th ACM SIGKDD international conference on knowledge discovery and data mining. ACM Press, New York, NY, USA, pp 39–48
Bilenko M, Mooney RJ (2003b) On evaluation and training-set construction for duplicate detection. In: Proceedings of the KDD-2003 workshop on data cleaning, record linkage and object consolidation, pp 7–12
Bortnichak EA, Wise RP, Salive ME, Tilson HH (2001) Proactive safety surveillance. Pharmacoepidemiol Drug Safety 10:191–196
Brinker AD, Beitz J (2002) Spontaneous reports of thrombocytopenia in association with quinine: clinical attributes and timing related to regulatory action. Am J Hematol 70:313–317
Copas J, Hilton F (1990) Record linkage: statistical models for matching computer records. J R Stat Soc: Sers A 153(3):287–320
De Veaux RD, Hand DJ (2005) How to lie with bad data. Stat Sci 20(3):231–238
Edwards IR (1997) Adverse drug reactions: finding the needle in the haystack. Br Med J 315(7107):500
Edwards IR (1999) Spontaneous reporting – of what? Clinical concerns about drugs. Br J Clin Pharmacol 48(2):138–141
Edwards IR, Aronson JK (2000) Adverse drug reactions: definitions, diagnosis and management. Lancet 356(9237):1255–1259
Evans SJW (2000) Pharmacovigilance: a science or fielding emergencies? Stat Med 19(23):3199–3209
Fayyad U, Piatetsky-Shapiro G, Smyth P (1996) The KDD process for extracting useful knowledge from volumes of data. Commun ACM 39(11):27–34
Fellegi IP, Sunter AB (1969) A theory for record linkage. J Am Stat Assoc 64:1183–1210
Hernández MA, Stolfo SJ (1998) Real-world data is dirty: data cleansing and the merge/purge problem. Data Min Knowl Discov 2(1):9–37
Jaro M (1989) Advances in record-linkage methodology as applied to matching the 1985 census of Tampa, Florida. J Am Stat Assoc 84:414–420
Kim WY, Choi B-J, Hong EK, Kim S-K, Lee D (2003) A taxonomy of dirty data. Data Min Knowl Discov 7(1):81–99
Lindquist M (2004) Data quality management in pharmacovigilance. Drug Safety 27(12):857–870
Monge AE, Elkan C (1997) An efficient domain-independent algorithm for detecting approximately duplicate database records. Research issues on data mining knowledge discovery, Tucson, AZ
Newcombe HB, Kennedy JM (1962) Record linkage: making maximum use of the discriminating power of identifying information. Commun ACM 5(11):563–566
Nkanza JN, Walop W (2004) Vaccine associated adverse event surveillance (VAAES) and quality assurance. Drug Safety 27:951–952
Norén GN, Bate A, Orre R, Edwards IR (2006) Extending the methods used to screen the WHO drug safety database towards analysis of complex associations and improved accuracy for rare events. Stat Med 25(21): 3740–3757
Norén GN, Orre R, Bate A (2005) A hit-miss model for duplicate detection in the WHO drug safety database. In: KDD ’05: proceeding of the 11th ACM SIGKDD international conference on knowledge discovery and data mining. ACM Press, New York, NY, USA, pp 459–468
Orre R, Lansner A, Bate A, Lindquist M (2000) Bayesian neural networks with confidence estimations applied to data mining. Comput Stat Data Anal 34:473–493
Rawlins MD (1988) Spontaneous reporting of adverse drug reactions. II: Uses. Br J Clin Pharmacol 1(26):7–11
Sarawagi S, Bhamidipaty A (2002) Interactive deduplication using active learning. In: KDD ’02: proceedings of the 8th ACM SIGKDD international conference on knowledge discovery and data mining. ACM Press, New York, NY, USA, pp 269–278
Author information
Authors and Affiliations
Corresponding author
Additional information
Responsible editor: Hannu Toivonen.
Rights and permissions
About this article
Cite this article
Norén, G.N., Orre, R., Bate, A. et al. Duplicate detection in adverse drug reaction surveillance. Data Min Knowl Disc 14, 305–328 (2007). https://doi.org/10.1007/s10618-006-0052-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10618-006-0052-8