Computer Science - Research and Development

, Volume 27, Issue 1, pp 45–63

Multi-pass sorted neighborhood blocking with MapReduce

Special Issue Paper

DOI: 10.1007/s00450-011-0177-x

Cite this article as:
Kolb, L., Thor, A. & Rahm, E. Comput Sci Res Dev (2012) 27: 45. doi:10.1007/s00450-011-0177-x

Abstract

Cloud infrastructures enable the efficient parallel execution of data-intensive tasks such as entity resolution on large datasets. We investigate challenges and possible solutions of using the MapReduce programming model for parallel entity resolution using Sorting Neighborhood blocking (SN). We propose and evaluate two efficient MapReduce-based implementations for single- and multi-pass SN that either use multiple MapReduce jobs or apply a tailored data replication. We also propose an automatic data partitioning approach for multi-pass SN to achieve load balancing. Our evaluation based on real-world datasets shows the high efficiency and effectiveness of the proposed approaches.

Keywords

MapReduce Hadoop Cloud computing Entity resolution Blocking Sorted Neighborhood 

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  1. 1.Institut für Informatik, Fakultät für Mathematik und InformatikUniversität LeipzigLeipzigGermany