A Novel Web Page Categorization Algorithm Based on Block Propagation Using Query-Log Information

  • Wenyuan Dai
  • Yong Yu
  • Cong-Le Zhang
  • Jie Han
  • Gui-Rong Xue
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4016)


Most existing web page classification algorithms, including content-based, link-based, or query-log analysis methods, treat the pages as smallest units. However, web pages usually contain some noisy or biased information which could affect the performance of classification. In this paper, we propose a Block Propagation Categorization (BPC) algorithm which deep mines web structure and views blocks as basic semantic units. Moreover, with query log information, BPC propagates only suitable information (block) among web pages to emphasize their topics. We also optimize the BPC algorithm to significantly speed up the block propagation process, without losing any precision. Our experiments on ODP and MSN search engine log show that BPC achieves a great improvement over traditional approaches.


Block Propagation Noisy Information Open Directory Project Query Session Virtual Page 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Beeferman, D., Berger, A.: Agglomerative clustering of a search engine query log. In: Proceedings of the sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 407–415 (2000)Google Scholar
  2. 2.
    Chakrabati, S., Dom, B., Indyk, P.: Enhanced hypertext categorization using hyperlinks. In: Proceedings of the ACM SIGMOD International Conference of Management of Data, Seattle, Washington, June 1998, pp. 307–318 (1998)Google Scholar
  3. 3.
    Chakrabarti, S.: Mining the Web: Discovering Knowledge from Hypertext Data. Morgan Kaufmann Publishers, San Francisco (2002)Google Scholar
  4. 4.
    Chuang, S.L., Chien, L.F.: Enriching Web taxonomies through subject categorization of query terms from search engine logs. Decision Support System 35(1) (April 2003)Google Scholar
  5. 5.
    Cohn, D., Hofmann, T.: The missing link – a probabilistic model of document content and hypertext connectivity. In: Advances in Neural Information Processing Systems, vol. 13, pp. 430–436. MIT Press, Cambridge (2001)Google Scholar
  6. 6.
    Cortes, C., Vapnik, V.: Support Vector Networks. Machine Learning 20, 1–25 (1995)Google Scholar
  7. 7.
    Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13, 21–27 (1967)MATHCrossRefGoogle Scholar
  8. 8.
    Glover, E.J., Tsioutsiouliklis, K., Lawrence, S., Pennock, D.M., Flake, G.W.: Using Web structure for classifying and describing Web pages. In: Proceedings of WWW 2002, International Conference on the World Wide Web (2002)Google Scholar
  9. 9.
    Lang, K.: Newsweeder: Learning to filter netnews. In: Proceedings of the 12th International Conference on Machine Learning, San Francisco, pp. 331–339 (1995)Google Scholar
  10. 10.
    Lewis, D.: Representation and learning in information retrieval. (COINS Technical Report 91-93). Dept. of Computer and Information Science, University of Massachusetts (1991)Google Scholar
  11. 11.
    Joachims, T.: A probabilistic analysis of the Rocchio algorithm with IFIDF for text categorization. Computer Science Technical Report CMU-CS-96-118. Carnegie Mellon UniversityGoogle Scholar
  12. 12.
    Joachims, T.: Text categorization with support vector machines: learning with many relevant features. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 137–142. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  13. 13.
    Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)MATHGoogle Scholar
  14. 14.
    Panteleeva, N.: Using neighborhood information for automated categorization of Web,
  15. 15.
    Salton, G.: The SMART Retrieval System – Experiments in Automatic Document rocessing. Prentice Hall Inc., Englewood Cliffs (1971)Google Scholar
  16. 16.
    Salton, G., Lesk, M.E.: Computer evaluation of indexing and text processing. Journal of the ACM 15(1), 8–36 (1968)MATHCrossRefGoogle Scholar
  17. 17.
    Slattery, S., Craven, M.: Discovery test set regularities in relational domains. In: Proceedings of ICML 2000, 17th International Conference on Machine Learning, Stanford, US, pp. 895–902 (2000)Google Scholar
  18. 18.
    Xue, G.R., Shen, D., Yang, Q., Zeng, H.J., Chen, Z., Yu, Y., Ma, W.Y.: IRC: An Iterative Reinforcement Categorization Algorithm for Interrelated Web Objects. In: Proceedings of the 2004 IEEE International Conference on Data Mining (ICDM 2004), Brighton, United Kingdom (November 2004)Google Scholar
  19. 19.
    Wang, J.D., Zeng, H.J., Chen, Z., Lu, H.J., Tao, L., Ma, W.Y.: ReCoM: reinforcement clustering of multi-type interrelated data objects. In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, Toronto, CA, July 2003, pp. 274–281 (2003)Google Scholar
  20. 20.
    Yang, Y., Pedersen, J.O.: A comparative study on feature selection in text categorization. In: Proceeding of the Fourteenth International Conference of Machine Learning (1997)Google Scholar
  21. 21.
    Yang, Y.: An evaluation of statistical approaches to text categorization. Journal of Information Retrieval 1(1/2), 67–88 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Wenyuan Dai
    • 1
  • Yong Yu
    • 1
  • Cong-Le Zhang
    • 1
  • Jie Han
    • 1
  • Gui-Rong Xue
    • 1
  1. 1.Apex Data & Knowledge Management Lab, Department of Computer Science and EngineeringShanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations