Reliable Evaluations of URL Normalization

  • Sung Jin Kim
  • Hyo Sook Jeong
  • Sang Ho Lee
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3984)


URL normalization is a process of transforming URL strings into canonical form. Through this process, duplicate URL representations for web pages can be reduced significantly. There are a number of normalization methods. In this paper, we describe four metrics for evaluating normalization methods. The reliability and consistency of a URL is also considered in our evaluation. With the metrics proposed, we evaluate seven normalization methods. The evaluation results on over 25 million URLs, extracted from the web, are reported in this paper.


Normalization Method True Positive Rate Path Segment Uniform Resource Locator Path Component 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Berners-Lee, T., Fielding, R., Masinter, L.: Uniform Resource Identifiers (URI): Generic Syntax (2005),
  2. 2.
    Burner, M.: Crawling Towards Eternity: Building an Archive of the World Wide Web. Web Techniques Magazine 2(5), 37–40 (1997)Google Scholar
  3. 3.
    Heydon, A., Najork, M.: Mercator: A Scalable, Extensible Web Crawler. International Journal of WWW 2(4), 219–229 (1999)Google Scholar
  4. 4.
    Kim, S.J., Lee, S.H.: Implementation of a Web Robot and Statistics on the Korean Web. In: Chung, C.-W., Kim, C.-k., Kim, W., Ling, T.-W., Song, K.-H. (eds.) HSI 2003. LNCS, vol. 2713, pp. 341–350. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  5. 5.
    Kim, S.-J., Lee, S.-H.: An empirical study on the change of web pages. In: Zhang, Y., Tanaka, K., Yu, J.X., Wang, S., Li, M. (eds.) APWeb 2005. LNCS, vol. 3399, pp. 632–642. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  6. 6.
    Lee, S.-H., Kim, S.-J., Hong, S.H.: On URL normalization. In: Gervasi, O., Gavrilova, M.L., Kumar, V., Laganá, A., Lee, H.P., Mun, Y., Taniar, D., Tan, C.J.K. (eds.) ICCSA 2005. LNCS, vol. 3481, pp. 1076–1085. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  7. 7.
    Shkapenyuk, V., Suel, T.: Design and Implementation of a High-performance Distributed Web Crawler. In: Proceedings of 18th Data Engineering Conference, pp. 357–368 (2002)Google Scholar
  8. 8.

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Sung Jin Kim
    • 1
  • Hyo Sook Jeong
    • 2
  • Sang Ho Lee
    • 2
  1. 1.School of Computer Science and EngineeringSeoul National UniversitySeoulKorea
  2. 2.School of ComputingSoongsil UniversitySeoulKorea

Personalised recommendations