Many data mining and machine learning projects require information from various data sources to be integrated and linked before they can be used for further analysis. A crucial task of such data integration is to identify which records refer to the same real-world entities across databases when no common entity identifiers are available and when records can contain errors and variations. This process of record linkage therefore has to rely upon the attributes that are available in the databases to be linked. For databases that contain personal information, for example, of customers, taxpayers, or patients, these are commonly their names, addresses, phone numbers, and dates of birth.To improve the scalability of the linkage process, blocking or indexing techniques are commonly applied to limit the comparison of records to pairs or groups that likely correspond to the same entity. Records are compared using a variety of comparison functions, most commonly approximate string comparators that account for typographical errors and variations in textual attributes. The compared records are then classified into matches, non-matches, and potential matches, depending upon the decision model used. If training data in the form of true matches and non-matches are available, supervised classification techniques can be employed. However, in many practical record linkage applications, no ground truth data are available, and therefore unsupervised approaches are required. An approach known as probabilistic record linkage is commonly employed. In this article we provide an overview of record linkage with an emphasis on the classification aspects of this process.
- Bhattacharya I, Getoor L (2007) Collective entity resolution in relational data. ACM Trans Knowl Discov Data 1(1), 5-es, pp 1–35Google Scholar
- Christen P (2012) Data matching – concepts and techniques for record linkage, entity resolution, and duplicate detection. Data-centric systems and applications. Springer, Berlin/New YorkGoogle Scholar
- Li P, Dong XL, Maurino A, Srivastava D (2011) Linking temporal records. The VLDB conference was in Seattle, WA. In: Proceedings of the VLDB endowment, Seattle, vol 4, issue 11Google Scholar
- On BW, Koudas N, Lee D, Srivastava D (2007) Group linkage. In: IEEE international conference on data engineering, Istanbul, pp 496–505Google Scholar
- Ramadan B, Christen P, Liang H, Gayler RW (2015) Dynamic sorted neighborhood indexing for real time entity resolution. ACM J Data Inf Qual 6(4):15Google Scholar
- Winkler WE (1988) Using the EM algorithm for weight computation in the Fellegi-Sunter model of record linkage. The American Statistical Association that is located in Alexandria, VA publishes the proceedings. In: Proceedings of the section on survey research methods, New Orleans, Washington, pp 667–671Google Scholar