The VLDB Journal

, Volume 28, Issue 2, pp 267–292 | Cite as

Leveraging set relations in exact and dynamic set similarity join

  • Xubo Wang
  • Lu QinEmail author
  • Xuemin Lin
  • Ying Zhang
  • Lijun Chang
Regular Paper


Set similarity join, which finds all the similar set pairs from two collections of sets, is a fundamental problem with a wide range of applications. Existing works study both exact set similarity join and approximate similarity join problems. In this paper, we focus on the exact set similarity join problem. The existing solutions for exact set similarity join follow a filtering-verification framework, which generates a list of candidate pairs through scanning indexes in the filtering phase and reports those similar pairs in the verification phase. Though much research has been conducted on this problem, set relations have not been well studied on improving the algorithm efficiency through computational cost sharing. Therefore, in this paper, we explore the set relations in different levels to reduce the overall computational cost. First, it has been shown that most of the computational time is spent on the filtering phase, which can be quadratic to the number of sets in the worst case for the existing solutions. Thus, we explore index-level set relations to reduce the filtering cost while keeping the same filtering power. We achieve this by grouping related sets into blocks in the index and skipping useless index probes in joins. Second, we explore answer-level set relations to further improve the algorithm based on the intuition that if two sets are similar, their answers may have a large overlap. We derive an algorithm which incrementally generates the answer of one set from an already computed answer of another similar set rather than compute the answer from scratch to reduce the computational cost. In addition, considering that in real applications, the data are usually updated dynamically, we extend our techniques and design efficient algorithms to incrementally update the join result when any element in the sets is updated. Finally, we conduct extensive performance studies using 21 real datasets with various data properties from a wide range of domains. The experimental results demonstrate that our algorithm outperforms all the existing algorithms across all datasets.


Incremental algorithm Set similarity join Set relations 



Lu Qin is supported by ARC DE140100999 and DP160101513. Xuemin Lin is supported by NSFC61232006, ARC DP150102728, DP140103578, and DP170101628. Ying Zhang is supported by ARC DE140100679 and DP170103710. Lijun Chang is supported by ARC DE150100563 and ARC DP160101513.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
    Anastasiu, D.C., Karypis,G.: L2AP: fast cosine similarity search with prefix L-2 norm bounds. In: Proceedings of ICDE’14 (2014)Google Scholar
  21. 21.
    Arvind, A., Venkatesh, G., Raghav, K.: Efficient exact set-similarity joins. In: Proceedings of VLDB’06 (2006)Google Scholar
  22. 22.
    Bayardo, R.J., Ma, Y., M., Srikant, R.: Scaling up all pairs similarity search. In: Proceedings of WWW’07 (2007)Google Scholar
  23. 23.
    Bouros, P., Ge, S., Mamoulis, N.: Spatio-textual similarity joins. PVLDB 6(1), 1–12 (2012)Google Scholar
  24. 24.
    Broder, A.Z., Charikar, M., Frieze, A.M., Mitzenmacher, M.: Min-wise independent permutations (extended abstract). In: Proceeding of STOC’98 (1998)Google Scholar
  25. 25.
    Chakrabarti, A., Parthasarathy, S.: Sequential hypothesis tests for adaptive locality sensitive hashing. In: Proceedings of WWW’15 (2015)Google Scholar
  26. 26.
    Chakrabarti, K., Chaudhuri, S., Ganti, V., Xin, D.: An efficient filter for approximate membership checking. In: Proceedings of SIGMOD’08 (2008)Google Scholar
  27. 27.
    Chaudhuri, S., Ganti, V., Kaushik, R.: A primitive operator for similarity joins in data cleaning. In: Proceedings of ICDE’06 (2006)Google Scholar
  28. 28.
    Cohen, W.W.: Integration of heterogeneous databases without common domains using queries based on textual similarity. In: Proceedings of SIGMOD’98 (1998)Google Scholar
  29. 29.
    Das, A., Datar, M., Garg, A., Rajaram, S.: Google news personalization: scalable online collaborative filtering. In: Proceedings of WWW’07 (2007)Google Scholar
  30. 30.
    Deng, D., Li, G., Wen, H., Feng, J.: An efficient partition based method for exact set similarity joins. PVLDB 9(4), 360–371 (2015)Google Scholar
  31. 31.
    Gionis, A., Indyk, P., Motwani, R.: Similarity search in high dimensions via hashing. In: Proceedings of VLDB’99 (1999)Google Scholar
  32. 32.
    Indyk, P.,Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: Proceedings of STOC’98 (1998)Google Scholar
  33. 33.
    Mann, W., Augsten, N.: PEL: position-enhanced length filter for set similarity joins. In: Proceedings of GVD’14 (2014)Google Scholar
  34. 34.
    Mann, W., Augsten, N., Bouros, P.: An empirical evaluation of set similarity join techniques. PVLDB 9(9), 636–647 (2016)Google Scholar
  35. 35.
    Metwally, A., Faloutsos, C.: V-smart-join: a scalable mapreduce framework for all-pair similarity joins of multisets and vectors. PVLDB 5(8), 704–715 (2012)Google Scholar
  36. 36.
    Ribeiro, L.A., Härder, T.: Generalizing prefix filtering to improve set similarity joins. Inf. Syst. 36(1), 62–78 (2011)CrossRefGoogle Scholar
  37. 37.
    Sarma, A.D., He, Y., Chaudhuri, S.: Clusterjoin: a similarity joins framework using map-reduce. PVLDB 7(12), 1059–1070 (2014)Google Scholar
  38. 38.
    Satuluri, V., Parthasarathy, S.: Bayesian locality sensitive hashing for fast similarity search. PVLDB 5(5), 430–441 (2012)Google Scholar
  39. 39.
    Schelter, S., Kunegis, J.: Tracking the trackers: a large-scale analysis of embedded web trackers. In: Tenth International AAAI Conference on Web and Social Media (2016)Google Scholar
  40. 40.
    Spertus, E., Sahami, M., Buyukkokten, O.: Evaluating similarity measures: a large-scale study in the orkut social network. In: Proceedings of SIGKDD’05 (2005)Google Scholar
  41. 41.
    Theobald, M., Siddharth, J., Paepcke, A.: Spotsigs: robust and efficient near duplicate detection in large web collections. In: Proceedings of SIGIR’08 (2008)Google Scholar
  42. 42.
    Vernica, R., Carey, M.J., Li, C.: Efficient parallel set-similarity joins using mapreduce. In: Proceedings of SIGMOD’10 (2010)Google Scholar
  43. 43.
    Wang, J., Kraska, T., Franklin, M.J., Feng, J.: Crowder: Crowdsourcing entity resolution. PVLDB 5(11), 1483–1494 (2012)Google Scholar
  44. 44.
    Wang, J., Li, G., Feng, J.: Can we beat the prefix filtering?: an adaptive framework for similarity join and search. In: Proceedings of SIGMOD’12 (2012)Google Scholar
  45. 45.
    Wang, X., Qin, L., Lin, X., Zhang, Y., Chang, L.: Leveraging set relations in exact set similarity join. PVLDB 10(9), 925–936 (2017)Google Scholar
  46. 46.
    Xiao, C., Wang, W., Lin, X., Shang, H.: Top-k set similarity joins. In: Proceedings of ICDE’09 (2009)Google Scholar
  47. 47.
    Xiao, C., Wang, W., Lin, X., Yu, J.X..: Efficient similarity joins for near duplicate detection. In: Proceedings of WWW’08 (2008)Google Scholar
  48. 48.
    Xiao, C., Wang, W., Lin, X., Yu, J.X., Wang, G.: Efficient similarity joins for near-duplicate detection. ACM Trans. Database Syst. 36(3), 15 (2011)CrossRefGoogle Scholar
  49. 49.
    Zhai, J., Lou, Y., Gehrke, J.: ATLAS: a probabilistic algorithm for high dimensional similarity search. In: Proceedings of SIGMOD’11 (2011)Google Scholar
  50. 50.
    Zhu, X., Goldberg, A.B.: Introduction to Semi-supervised Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, San Rafael (2009)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.University of New South WalesSydneyAustralia
  2. 2.University of Technology SydneySydneyAustralia
  3. 3.The University of SydneySydneyAustralia

Personalised recommendations