Journal of Intelligent Information Systems

, Volume 22, Issue 3, pp 215–235 | Cite as

An Analytical Approach to Concept Extraction in HTML Environments

  • Victor Fresno
  • Angela Ribeiro


The core of the Internet and World Wide Web revolution comes from their capacity to efficiently share the huge quantity of data, but the rapid and chaotic growth of the Net has extremely complicated the task of sharing or mining useful information. Each inference process, from Internet information, requires an adequate characterization of the Web pages. The textual part of a page is one of the most important aspects that should be considered to appropriately perform a page characterization. The textual characterization should be made through the extraction of an appropriate set of relevant concepts that properly represent the text included in the Web page. This paper presents a method to obtain such a set of relevant concepts from a Web page, essentially based on a relevance estimation of each word in the text of a Web page. The word-relevance is defined by a combination of criteria that take into account characteristics of the HTML language as well as more classical measures such as the frequency and the position of a word in a document. Besides, heuristic rules to obtain the most suitable fusion of criteria is achieved via a statistical study. Several experiments are conducted to test the performance of the proposed concept extraction method compared to other approaches including a commercial tool. The results obtained here exhibit a greater success in the concept extraction by the proposed technique against other tested methods.

concept extraction feature vector in HTML texts Web page characterization Web page representation Web page classification 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Baeza-Yates, R. and Ribeiro-Neto, B. (1999). Modern Information Retrieval. Addison-Wesley: ACM Press Books.Google Scholar
  2. Chen, H. and Dumais, S.T. (2000). Bringing Order to the Web: Automatically Categorizing Search Results. In Proc. Of CHI'00, Human Factor in Computing Systems (pp. 145-152). Den Haag, New York, US: ACM Press.Google Scholar
  3. Dunham, M.H. (2002). Data Mining. Introductory and Advanced Topics. Upper Saddle River, NJ: Prentice Hall.Google Scholar
  4. Fresno, V. and Ribeiro, A. (2001). Feature Selection and Dimensionality Reduction inWeb Pages Representation. In International ICSC Congress on Computational Intelligence: Methods & Applications (pp. 416-421). Bangor, Wales, U.K.Google Scholar
  5. Gales, W., Kenneth, W.C., and Yarowsky, D. (1992). A Method for Disambiguating Word Senses in a Large Corpus. Computers and the Humanities, 26, 415-439.Google Scholar
  6. Gudivada, V.N., Raghavan, V.V., Grosky, W.I., and Kasanagottu, R. (1997). Information Retrieval on the World Wide Web, IEEE Internet Computing. Sept.-Oct., 58-68.Google Scholar
  7. Henzinger, M. (2000). Link Analysis in Web Information Retrieval. Bulletin of the Technical Committee on Data Engineering, 23, 3-8.Google Scholar
  8. Hovy, E. and Lin, C.Y. (1999). Automated Text Summarization in SUMMARIST. In I. Mani and M.T. Maybury (Eds.), Advances in Automatic Text Summarization. Cambridge, MA. The MIT Press.Google Scholar
  9. Koller, D. and Sahami, M. (1996). Toward Optimal Feature Selection. In ICML-96: Proceedings of the Thirteenth International Conference on Machine Learning (pp. 284-292). San Francisco, CA: Morgan Kaufmann.Google Scholar
  10. Kosala, R. and Blockeel, H. (2000). Web Mining Research: A Survey. ACM SIGKDD Explorations, 2(1), 1-15.Google Scholar
  11. Manning, C.D. and Schtze, H. (2001). Foundations of Statistical Natural Language Processing, Cambridge, MA: The MIT Press.Google Scholar
  12. Mitchell, T.M. (1997). Machine Learning. McGraw-Hill International Editions.Google Scholar
  13. Mladenic, D. (1999). Text-Learning and Related Intelligent Agents. IEEE Expert Special issue on Applications of Intelligent Information Retrieval. July-August.Google Scholar
  14. Musciano, C. and Kennedy, B. (1997). HTML The Complete Guide. McGraw Hill.Google Scholar
  15. Salton, G., Wong, A., and Yang, C.S. (1975). A Vector Space Model for Information Retrieval. Communications of the ACM 18(11), 613-620.Google Scholar
  16. UNCTAD(2002). E-Commerce and Development Report 2002. Report of the United Nations Conference on Trade and Development. United Nations, New York and Geneva.Google Scholar
  17. Yang, Y., Slattery, S., and Ghani, R. (2002). A Study of Approaches to Hypertext Categorization. Journal of Intelligent Information Systems, 18(2), 219-241.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Victor Fresno
    • 1
  • Angela Ribeiro
    • 2
  1. 1.Escuela Superior de Ciencias Experimentales y TecnologiaRey Juan Carlos UniversityMostoles, MadridSpain
  2. 2.Industrial Automation Institute (IAI)Spanish Council for Scientific Research (CSIC)MadridSpain

Personalised recommendations