Semantic URL Analytics to Support Efficient Annotation of Large Scale Web Archives

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9398)


Long-term Web archives comprise Web documents gathered over longer time periods and can easily reach hundreds of terabytes in size. Semantic annotations such as named entities can facilitate intelligent access to the Web archive data. However, the annotation of the entire archive content on this scale is often infeasible. The most efficient way to access the documents within Web archives is provided through their URLs, which are typically stored in dedicated index files. The URLs of the archived Web documents can contain semantic information and can offer an efficient way to obtain initial semantic annotations for the archived documents. In this paper, we analyse the applicability of semantic analysis techniques such as named entity extraction to the URLs in a Web archive. We evaluate the precision of the named entity extraction from the URLs in the Popular German Web dataset and analyse the proportion of the archived URLs from 1,444 popular domains in the time interval from 2000 to 2012 to which these techniques are applicable. Our results demonstrate that named entity recognition can be successfully applied to a large number of URLs in our Web archive and provide a good starting point to efficiently annotate large scale collections of Web documents.


Uniform Resource Locator (URLs) Named Entity Extraction (NER) Named Entity Recognition Techniques Popular Domain Language Detection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was partially funded by the European Research Council under ALEXANDRIA (ERC 339233) and the COST Action IC1302 (KEYSTONE). Tarcisio Souza is sponsored by a scholarship from CNPq, a Brazilian government institution for scientific development.


  1. 1.
    Abramson, M., Aha, D.: What’s in a URL? genre classification from URLs. In: Proceedings of AAAI workshop on Intelligent Techniques for Web Personalization and Recommender Systems (2012)Google Scholar
  2. 2.
    Anastácio, I., Martins, B., Calado, P.: Classifying documents according to locational relevance. In: Lopes, L.S., Lau, N., Mariano, P., Rocha, L.M. (eds.) EPIA 2009. LNCS, vol. 5816, pp. 598–609. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  3. 3.
    Baykan, E., Henzinger, M., Weber, I.: Web page language identification based on URLs. PVLDB Endow. 1(1), 176–187 (2008)CrossRefGoogle Scholar
  4. 4.
    Baykan, E., Henzinger, M., Marian, L., Weber, I.: A comprehensive study of features and algorithms for URL-based topic classification. ACM Transactions Web (2011)Google Scholar
  5. 5.
    Baykan, E., Henzinger, M., Weber, I.: A comprehensive study of techniques for URL-based web page language classification. ACM Transactions Web (2013)Google Scholar
  6. 6.
    Brügger, N.: Probing a nation’s web sphere: a new approach to web history and a new kind of historical source. In Proceedings of the 2014 ACM conference on Web science (2014)Google Scholar
  7. 7.
    Craswell, N., Hawking, D., Robertson, S.: Effective site finding using link anchor information. In: Proceedings of the 24th Annual International ACM SIGIR, SIGIR 2001, ACM, New York (2001)Google Scholar
  8. 8.
    Hernández, I., Rivero, C.R., Ruiz, D., Corchuelo, R.: A statistical approach to URL-based web page clustering. In: Proceedings of the 21st International Conference Companion on World Wide Web, WWW 2012, ACM, New York (2012)Google Scholar
  9. 9.
    Hernández, I., Rivero, C.R., Ruiz, D., Arjona, J.L.: An experiment to test URL features for web page classification. In: Rodríguez, J.M.C., Pérez, J.B., Golinska, P., Giroux, S., Corchuelo, R. (eds.) Trends in PAAMS. AISC, vol. 157, pp. 109–116. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  10. 10.
    Hernndez, I., Rivero, C.R., Ruiz, D., Corchuelo, R.: CALA: an unsupervised URL-based web page classification system. Knowl. Based Syst. 57, 168–180 (2014)CrossRefGoogle Scholar
  11. 11.
    Kan, M.-Y., Thi, H.O.N.: Fast webpage classification using URL features. In: Proceedings of the 14th ACM International Conference on Information and Knowledge Management, CIKM 2005, ACM, New York (2005)Google Scholar
  12. 12.
    Koppula, H.S., Leela, K.P., Agarwal, A., Chitrapura, K.P., Garg, S., Sasturkar, A.: Learning URL patterns for webpage de-duplication. In: Proceedings of the Third ACM International Conference on Web Search and Data Mining, WSDM 2010, New York (2010)Google Scholar
  13. 13.
    Raju, S., Udupa, R.: Extracting advertising keywords from URL strings. In: Proceedings of the 21st International Conference Companion on World Wide Web, WWW 2012, ACM, New York (2012)Google Scholar
  14. 14.
    Risse, T., Demidova, E., Gossen, G.: What do you want to collect from the web? In: Proceedings of the Building Web Observatories Workshop, BWOW 2014 (2014)Google Scholar
  15. 15.
    Zhao, P., Hoi, S.C.H.: Cost-sensitive online active learning with application to malicious URL detection. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, ACM, New York (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (, which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Authors and Affiliations

  1. 1.L3S Research Center and Leibniz Universität HannoverHannoverGermany
  2. 2.Gdansk University of TechnologyGdańskPoland

Personalised recommendations