Research and Realization of News Gathering and Editing System Based on Distributed Search Engine

  • Yamin Han
  • Kun Liu
  • Kun Ma
Conference paper
Part of the Lecture Notes on Data Engineering and Communications Technologies book series (LNDECT, volume 8)


According to the investigation and survey on the demands of, the depth analysis was performed to explore the new changes in the news material gathering and material search methods in the system in the context of big data. Meanwhile, the related technologies of web crawler, replicated documents elimination and distributed search engine are integrated in the news gathering and editing system. The experimental results showed that the system had the functions of efficiently gathering Web materials, eliminating the replicated documents and fast retrieving information, which can meet the actual needs required by the industry of news.


Web crawler Replicated documents elimination ElasticSearch Distributed search 


  1. 1.
    Jiangwen, S., Kong, X., Song, L.: Non structured data unified storage platform. Comput. Modernization 11, 69–73 (2016)Google Scholar
  2. 2.
    Shi, R.: A new approach to content extraction from web page. Telecom World 19, 210–211 (2015)Google Scholar
  3. 3.
    Narawade, S.M., Prabhakar, N.M., Maruti, N.S.: A web based data extraction using hierarchical (DOM) tree approach. Int. J. Innovative Res. Sci. Technol. 2(11), 255–257 (2016)Google Scholar
  4. 4.
    Xiong, Z., Ya, M., Zhang, Y.: Detection and elimination of similar web pages based on text structure and string of feature code. J. Comput. Appl. 33(2), 554–557 (2013)Google Scholar
  5. 5.
    Gormley, C., Tong, Z.: Elasticsearch: The Definitive Guide. O’Reilly Media, Sebastopol (2015)Google Scholar
  6. 6.
    Meng, B., Wang, Z.: Two chinese word segmentation algorithms in the realization of the cloud computing platform. Netw. Secur. Technol. Appl. 12, 67–71 (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Shandong Provincial Key Laboratory of Network Based Intelligent ComputingUniversity of JinanJinan, ShandongChina
  2. 2.School of Information Science and EngineeringUniversity of JinanJinanPeople’s Republic of China

Personalised recommendations