Hybrid Theme Crawler Based on Links and Semantics

  • Kang Zhao
  • Yang YangEmail author
  • Zhipeng Gao
  • Long Cheng
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 905)


Common theme crawler generally analyses the page content or link structure, without solving the problem of computational complexity and easy “myopia”, resulting in the page of recall and precision is not high. This paper introduces a mixed theme decision strategy, which fully considers the text content and link structure of the page. By introducing knowledge map database and entity database, the computational complexity is simplified and the judgment accuracy is increased. The experiment shows that the rate of inspection and precision is greatly improved.


Crawler HowNet Links Semantics Similarity 


  1. 1.
    Yang, X.U., Wang, W.Y.: Research on Subject relevance algorithm of theme crawler. Mod. Comput. (2016)Google Scholar
  2. 2.
    Qiu, L., Lou, Y., Chang, M.: Research on theme crawler based on shark-search and PageRank algorithm. In: International Conference on Cloud Computing and Intelligence Systems, pp. 268–271. IEEE (2016)Google Scholar
  3. 3.
    Zhang, Y.F., Zheng, S.H.: Heritrix based theme crawler design. J. Chang. Univ. Technol. (2016)Google Scholar
  4. 4.
    Qiu, L., Lou, Y.S., Chang, M.: An improved shark-search algorithm for theme crawler. Microcomput. Appl. (2017)Google Scholar
  5. 5.
    Kumar, N., Singh, M.: Framework for distributed semantic web crawler. In: International Conference on Computational Intelligence and Communication Networks, pp. 1403–1407. IEEE (2016)Google Scholar
  6. 6.
    Wu, T., Liang, Y., Wu, C., Piao, S.F., Ma, D.Y., Zhao, G.Z., Han, X.S.: A chinese topic crawler focused on customer development. Procedia CIRP 56, 476–480 (2016)CrossRefGoogle Scholar
  7. 7.
    Wu, L., Wang, Y.B.: The research of the topic crawler algorithm based on semantic similarity aggregation. J. Commun. Univ. China (Sci. Technol.) (2018)Google Scholar
  8. 8.
    Yuan, Y.W., Lu, P.J.: Design of a topic crawler for college bidding announcement. Softw. Guid. (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.State Key Laboratory of Networking and Switching TechnologyBeijing University of Posts and TelecommunicationsBeijingChina

Personalised recommendations