Set Similarity Joins with Complex Expressions on Distributed Platforms

  • Diego Junior do Carmo Oliveira
  • Felipe Ferreira Borges
  • Leonardo Andrade Ribeiro
  • Alfredo Cuzzocrea
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11019)


A set similarity join finds all similar pairs from a collection of sets. This operation is essential for many important tasks in Big Data analytics including string data integration and cleaning. The vast majority of set similarity join algorithms proposed so far considers string data represented by a single set over which a simple similarity predicate is defined. However, real data is typically multi-attribute and, thus, better represented by multiple sets. Such a representation requires complex expressions to capture a given notion of similarity. Moreover, similarity join processing under this new formulation is clearly more expensive, which calls for distributed algorithms to deal with large datasets. In this paper, we present a distributed algorithm for set similarity joins with complex similarity expressions. Our approach supports complex Boolean expressions over multiple predicates. We propose a simple, but effective data partitioning strategy to reduce both communication and computation costs. We have implemented our algorithm in Spark, a popular distributed data processing engine. Experimental results show that the proposed approach is efficient and scalable.


Similarity join Distributed algorithms Data integration 



This work was partially supported by Brazilian agency CAPES.


  1. 1.
    Bayardo, R.J., Ma, Y., Srikant, R.: Scaling up all pairs similarity search. In: Proceedings of the WWW Conference, pp. 131–140 (2007)Google Scholar
  2. 2.
    Broder, A.Z., Charikar, M., Frieze, A.M., Mitzenmacher, M.: Min-wise independent permutations (extended abstract). In: Proceedings of the STOC Symposium, pp. 327–336 (1998)Google Scholar
  3. 3.
    Chaudhuri, S., Ganti, V., Kaushik, R.: A primitive operator for similarity joins in data cleaning. In: Proceedings of the ICDE Conference, p. 5 (2006)Google Scholar
  4. 4.
    Chu, X., Ilyas, I.F., Koutris, P.: Distributed data deduplication. Proc. VLDB Endow. 9(11), 864–875 (2016)CrossRefGoogle Scholar
  5. 5.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of the USENIX Symposium on Operating Systems Design and Implementation, pp. 137–150 (2004)Google Scholar
  6. 6.
    Deng, D., Li, G., Hao, S., Wang, J., Feng, J.: MassJoin: a mapreduce-based method for scalable string similarity joins. In: Proceedings of the ICDE Conference, pp. 340–351 (2014)Google Scholar
  7. 7.
    Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: Proceedings of the STOC Symposium, pp. 604–613 (1998)Google Scholar
  8. 8.
    Leskovec, J., Rajaraman, A., Ullman, J.D.: Mining of Massive Datasets. Cambridge University Press, Cambridge (2014)CrossRefGoogle Scholar
  9. 9.
    Li, G., He, J., Deng, D., Li, J.: Efficient similarity join and search on multi-attribute data. In: Proceedings of the SIGMOD Conference, pp. 1137–1151 (2015)Google Scholar
  10. 10.
    Ribeiro, L.A., Härder, T.: Generalizing prefix filtering to improve set similarity joins. Inf. Syst. 36(1), 62–78 (2011)CrossRefGoogle Scholar
  11. 11.
    Ribeiro, L.A., Schneider, N.C., de Souza Inácio, A., Wagner, H.M., von Wangenheim, A.: Bridging database applications and declarative similarity matching. J. Inf. Data Manag. 7(3), 217–232 (2016)Google Scholar
  12. 12.
    Ribeiro-Júnior, S., Quirino, R.D., Ribeiro, L.A., Martins, W.S.: Fast parallel set similarity joins on many-core architectures. J. Inf. Data Manag. 8(3), 255–270 (2017)Google Scholar
  13. 13.
    Rong, C., Lin, C., Silva, Y.N., Wang, J., Lu, W., Du, X.: Fast and scalable distributed set similarity joins for big data analytics. In: Proceedings of the ICDE Conference, pp. 1059–1070 (2017)Google Scholar
  14. 14.
    Sarawagi, S., Kirpal, A.: Efficient set joins on similarity predicates. In: Proceedings of the SIGMOD Conference, pp. 743–754 (2004)Google Scholar
  15. 15.
    Shi, J., et al.: Clash of the titans: mapreduce vs. spark for large scale data analytics. Proc. VLDB Endow. 8(13), 2110–2121 (2015)CrossRefGoogle Scholar
  16. 16.
    Sidney, C.F., Mendes, D.S., Ribeiro, L.A., Härder, T.: Performance prediction for set similarity joins. In: Proceedings of the SAC Conference, pp. 967–972 (2015)Google Scholar
  17. 17.
    Vernica, R., Carey, M.J., Li, C.: Efficient parallel set-similarity joins using MapReduce. In: Proceedings of the SIGMOD Conference, pp. 495–506 (2010)Google Scholar
  18. 18.
    Xiao, C., Wang, W., Lin, X., Yu, J.X., Wang, G.: Efficient similarity joins for near-duplicate detection. ACM Trans. Database Syst. 36(3), 15:1–15:41 (2011)CrossRefGoogle Scholar
  19. 19.
    Zaharia, M., et al.: Apache Spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Diego Junior do Carmo Oliveira
    • 1
  • Felipe Ferreira Borges
    • 1
  • Leonardo Andrade Ribeiro
    • 1
  • Alfredo Cuzzocrea
    • 2
  1. 1.Instituto de InformáticaUniversidade Federal de GoiásGoiâniaBrazil
  2. 2.DIA DepartmentUniversity of Trieste and ICAR-CNRTriesteItaly

Personalised recommendations