Record Linkage

Abstract

Many data mining and machine learning projects require information from various data sources to be integrated and linked before they can be used for further analysis. A crucial task of such data integration is to identify which records refer to the same real-world entities across databases when no common entity identifiers are available and when records can contain errors and variations. This process of record linkage therefore has to rely upon the attributes that are available in the databases to be linked. For databases that contain personal information, for example, of customers, taxpayers, or patients, these are commonly their names, addresses, phone numbers, and dates of birth.To improve the scalability of the linkage process, blocking or indexing techniques are commonly applied to limit the comparison of records to pairs or groups that likely correspond to the same entity. Records are compared using a variety of comparison functions, most commonly approximate string comparators that account for typographical errors and variations in textual attributes. The compared records are then classified into matches, non-matches, and potential matches, depending upon the decision model used. If training data in the form of true matches and non-matches are available, supervised classification techniques can be employed. However, in many practical record linkage applications, no ground truth data are available, and therefore unsupervised approaches are required. An approach known as probabilistic record linkage is commonly employed. In this article we provide an overview of record linkage with an emphasis on the classification aspects of this process.