The Journal of Supercomputing

, Volume 75, Issue 3, pp 1137–1149 | Cite as

Exploring hybrid parallel systems for probabilistic record linkage

  • Murilo Boratto
  • Pedro AlonsoEmail author
  • Clicia Pinto
  • Pedro Melo
  • Marcos Barreto
  • Spiros Denaxas


Record linkage is a technique widely used to gather data stored in disparate data sources that presumably pertain to the same real world entity. This integration can be done deterministically or probabilistically, depending on the existence of common key attributes among all data sources involved. The probabilistic approach is very time-consuming due to the amount of records that must be compared, specifically in big data scenarios. In this paper, we propose and evaluate a methodology that simultaneously exploits multicore and multi-GPU architectures in order to perform the probabilistic linkage of large-scale Brazilian governmental databases. We present some algorithmic optimizations that provide high accuracy and improve performance by defining the best algorithm-architecture combination for a problem given its input size. We also discuss performance results obtained with different data samples, showing that a hybrid approach outperforms other configurations, providing an average speedup of 7.9 when linking up to 20.000 million records.


Probabilistic linkage Public health Performance evaluation Multicore Multi-GPU 



This work has been partially supported by CNPq, FAPESB, Bill & Melinda Gates Foundation, The Royal Society (UK), Medical Research Council (UK), NVIDIA Hardware Grant Program, Generalitat Valenciana (Grant PROMETEOII/2014/003), Spanish Government and European Commission through TEC2015-67387-C4-1-R (MINECO/FEDER), and network CAPAP-H. We have also worked in cooperation with the EU-COST Programme Action IC1305, “Network for Sustainable Ultrascale Computing (NESUS)”.


  1. 1.
    Andrade G, Viegas F, Ramos GS, Almeida J, Rocha L, Gonçalves M, Ferreira R (2013) GPU-NB: a fast CUDA-based implementation of Naïve Bayes. In: 2013 25th International Symposium on Computer Architecture and High Performance Computing, pp 168–175Google Scholar
  2. 2.
    Bloom BH (1970) Space/time trade-offs in hash coding with allowable errors. Commun ACM 13(7):422–426CrossRefzbMATHGoogle Scholar
  3. 3.
    Cook S (2013) CUDA Programming: A Developer’s Guide to Parallel Computing with GPUs, 1st edn. Morgan Kaufmann, San FranciscoGoogle Scholar
  4. 4.
    Doan A, Halevy A, Ives Z (2012) Principles of Data Integration. Elsevier, AmsterdamGoogle Scholar
  5. 5.
    Étienne EY (2012) Hyper-threading. TurbsPublishing, SaarbrückenGoogle Scholar
  6. 6.
    Fellegi IP, Sunter AB (1969) A theory for record linkage. J Am Stat Assoc 64:1183–1210CrossRefzbMATHGoogle Scholar
  7. 7.
    Feng X, Jin H, Zheng R, Zhu L (2014) Near-duplicate detection using GPU-based simhash scheme. In: 2014 International Conference on Smart Computing, pp 223–228Google Scholar
  8. 8.
    Forchhammer B, Papenbrock T, Stening T, Viehmeier S, Naumann U.D.F (2013) Duplicate detection on GPUs. In: BTW. Köllen-Verlag, pp 165–184Google Scholar
  9. 9.
    Kim H.s, Lee D (2007) Parallel linkage. In: Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management, CIKM 2007. ACM, New York, NY, USA, pp 283–292Google Scholar
  10. 10.
    Mamun AA, Aseltine R, Rajasekaran S (2015) RLT-S: a web system for record linkage. PLoS ONE 10(5):1–9CrossRefGoogle Scholar
  11. 11.
    Mamun AA, Aseltine R, Rajasekaran S (2016) Efficient record linkage algorithms using complete linkage clustering. PLoS ONE 11(4):1–21CrossRefGoogle Scholar
  12. 12.
    Mamun AA, Mi T, Aseltine R, Rajasekaran S (2014) Efficient sequential and parallel algorithms for record linkage. J Am Med Inform Assoc 21(2):252–262CrossRefGoogle Scholar
  13. 13.
    Mizell E, Biery R (2017) How GPUs are defining the future of data analyticsGoogle Scholar
  14. 14.
    Munshi A, Gaster B, Mattson TG, Fung J, Ginsburg D (2011) OpenCL Programming Guide, 1st edn. Addison-Wesley, ReadingGoogle Scholar
  15. 15.
    NVIDIA Corporation: NVIDIA CUDA C programming guide (2010). Version 3.2Google Scholar
  16. 16.
    OpenMP Architecture Review Board: OpenMP application program interface version 4.0 (2013)Google Scholar
  17. 17.
    Pokorny J (2011) NoSQL databases: a step to database scalability in web environment. In: Proceedings of the 13th International Conference on Information Integration and Web-based Applications and Services, iiWAS ’11. ACM, New York, NY, USA, pp 278–283Google Scholar
  18. 18.
    Rendle S, Schmidt-Thieme L (2008) Scaling Record Linkage to Non-uniform Distributed Class Sizes. Springer, Berlin, pp 308–319Google Scholar
  19. 19.
    Sehili Z, Kolb L, Borgs C, Schnell R, Rahm E (2015) Privacy preserving record linkage with ppjoin. In: Datenbanksysteme für Business, Technologie und Web (BTW), pp 85–104Google Scholar
  20. 20.
    Winkler WE (1999) The state of record linkage and current research problemsGoogle Scholar
  21. 21.
    Zhong Z, Rychkov V, Lastovetsky A (2015) Data partitioning on multicore and multi-GPU platforms using functional performance models. IEEE Trans Comput 64(9):2506–2518MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Núcleo de Arquitetura de Computadores e Sistemas OperacionaisUniversidade do Estado da BahiaSalvadorBrazil
  2. 2.Departament of Information Systems and ComputationUniversitat Politècnica de ValènciaValenciaSpain
  3. 3.Laboratório de Sistemas DistribuídosUniversidade Federal da BahiaSalvadorBrazil
  4. 4.Institute of Health Informatics Research, School of Computer Science and InformaticsUniversity College LondonLondonUK

Personalised recommendations