Enhancing Duplicate Collection Detection Through Replica Boundary Discovery

  • Zhigang Zhang
  • Weijia Jia
  • Xiaoming Li
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3918)


Web documents are widely replicated on the Internet. These replicated documents bring potential problems to Web based information systems. So replica detection on the Web is an indispensable task. The challenge is to find these duplicated collections from a very large data set with limited hardware resources in acceptable time. In this paper, we first introduce the notion of replica boundary to roughly reflect the situation of the replicas; then we propose an effective and efficient approach to discover the boundary of the replicas. The advantages of the proposed approach include: first, it dramatically reduces pair-wise document similarity computation, making it much faster than traditional replicated document detection approaches; second, it can identify the boundary of the replicated collections accurately, demonstrating to what extent two collections are replicated. On two web page sets containing 24 million and 30 million Web pages respectively, we evaluated the accuracy of the approach.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Broder, A.Z.: On the resemblance and containment of documents. In: Proceedings of Compression and Complexity of Sequences 1997, pp. 21–29. IEEE Computer Society, Los Alamitos (1997)Google Scholar
  2. 2.
    Broder, A.Z.: Identifying and Filtering Near-Duplicate Documents. In: 11th Annual Symposium on Combinatorial Pattern Matching, June 2000, pp. 1–10 (2000)Google Scholar
  3. 3.
    Broder, Z., Glassman, S.C., Manasse, M.S., Eig, G.: Syntactic clustering of the Web. In: Proceedings of the sixth International World Wide Web Conference, pp. 391–404 (1997)Google Scholar
  4. 4.
    Cho, J., Shivakumar, N., Garcia-Molina, H.: Finding Replicated Web Collections. In: SIGMOD Conference 2000, pp. 355–366 (2000)Google Scholar
  5. 5.
    Heintze, N.: Scalable Document Fingerprinting. In: Proceedings of the Second USENIX Workshop on Electronic Commerce, pp. 191–200 (1996)Google Scholar
  6. 6.
    Kotcz, A., Chowdhury, A., Alspector, J.: Improved robustness of signature-based near-replica detection via lexicon randomization. In: Proceedings of the 2004 ACM SIGKDD Conference, pp. 605–610 (2004)Google Scholar
  7. 7.
    Bharat, K., Broder, A.Z.: Mirror, Mirror, on the Web: A study of host pairs with replicated content. In: Proceedings of 8th International Conference on World Wide Web (WWW 1999) (May 1999)Google Scholar
  8. 8.
    Zhang, Z., Chen, J., Li, X.: A Preprocessing Framework and Approach for Web Applications. Journal of Web Engineering 2(3), 175–191 (2004)Google Scholar
  9. 9.
    Chowdhury, A., Frieder, O., Grossman, D.A., McCabe, M.C.: Collection statistics for fast duplicated document detection. ACM Transactions on Information Systems 20(2), 171–191 (2002)CrossRefGoogle Scholar
  10. 10.
    Brin, S., Davis, J., Garcia-Molina, H.: Copy detection mechanisms for digital documents. In: Proceedings of the ACM SIGMOD Annual Conference, San Francisco, CA (May 1995)Google Scholar
  11. 11.
    Shivakumar, N., Garcia-Molina, H.: SCAM: A Copy Detection Mechanism for Digital Documents. In: Proceedings of the 2nd International Conference on Theory and Practice of Digital Libraries (1995)Google Scholar
  12. 12.
    Shivakumar, N., Garcia-Molina, H.: Building a Scalable and Accurate Copy Detection Mechanism. In: Proceedings of the 3nd International Conference on Theory and Practice of Digital Libraries (1996)Google Scholar
  13. 13.
    Xi, W., Fox, E.A., Tan, R.P., Shu, J.: Machine Learning Approach for Homepage Finding Task. In: Proceedings of the 9th International Symposium on String Processing and Information Retrieval, Lisbon, Portugal, September 11-15, pp. 145–159 (2002)Google Scholar
  14. 14.
    Shivakumar, N., Garcia-Molina, H.: Finding near-replicas of documents on the Web. In: Atzeni, P., Mendelzon, A.O., Mecca, G. (eds.) WebDB 1998. LNCS, vol. 1590, pp. 204–212. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  15. 15.
    Henzinger, M., Motwani, R., Silverstein.: Challenges in Web Search Engines. In: Proceedings of the 18th International Joint Conference on Artificial Intelligene (2003)Google Scholar
  16. 16.
    Bharat, K., Broder, A., Dean, J., Henzinger, M.R.: A Comparison of Techniques to find mirrored hosts on the WWW. Journal of the American Society for Information Science 51(12), 1114–1122Google Scholar
  17. 17.
    Tianwang Web search engine,

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Zhigang Zhang
    • 1
  • Weijia Jia
    • 1
  • Xiaoming Li
    • 2
  1. 1.Department of Computer ScienceCity University of Hong KongKowloonHong Kong
  2. 2.Institute of Network Computing and Information Systems, School of Electronics Engineering and Computer SciencePeking UniversityBeijingChina

Personalised recommendations