Advertisement

DTR: A Novel Topic Generate Algorithm Based on Dbscan and TextRank

  • Yingbao CuiEmail author
  • Di Liu
  • Qiang Li
  • Zhen Qiu
  • Xusheng Yang
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1075)

Abstract

With the advent of the era of big data, information overload has become a universal problem. How to extract valuable information from the complicated data, and let users quickly understand the main content of these data has become the focus of research. Current topic generation is a way to solve this problem. Some unsupervised topic models generate results for the probability distribution of topics and words, which are less explanatory to the topic, unable to generate subject matter that can be read smoothly. Supervised topic classification methods rely too much on tagged training data and are not universally applicable. This paper proposes a novel topic generation algorithm DTR, which is based on Dbscan and TextRank. By combining text clustering with abstract algorithm, it can automatically generate refined and understandable document topics. At the same time, the similarity analysis method is used to quickly process the document, which narrows the data range of the generated topic and improves the accuracy of the generated topic. The effectiveness of this algorithm is verified by experiments based on a large amount of real data.

Keywords

Information overload Relevance analysis Text clustering topic generation 

References

  1. 1.
    Mu, X., Hao, W., Gang, C., Zhao, S., Jin, D.: Research based on concept maps and hidden Markov model for multi-document summary. In: IEEE International Conference on Broadband Network Multimedia Technology (2011)Google Scholar
  2. 2.
    Nema, P., Khapra, M.M., Laha, A., Ravindran, B.: Diversity driven attention model for query-based abstractive summarization (2017)Google Scholar
  3. 3.
    Perotte, A., Bartlett, N., Elhadad, N., Wood, F.: Hierarchically supervised latent Dirichlet allocation. In: International Conference on Neural Information Processing Systems (2011)Google Scholar
  4. 4.
    Koltcov, S., Koltsova, O., Nikolenko, S.: Latent Dirichlet allocation: stability and applications to studies of user-generated content. In: ACM Conference on Web Science (2014)Google Scholar
  5. 5.
    Blei, D.M., Mcauliffe, J.D.: Supervised topic models. In: Advances in Neural Information Processing Systems, vol. 3, pp. 327–332 (2010)Google Scholar
  6. 6.
    Ramage, D., Hall, D., Nallapati, R., Manning, C.D.: Labeled LDA: a supervised topic model for credit attribution in multi-labeled corpora. In: Proceedings of EMNLP, vol. 1, pp. 248–256 (2009)Google Scholar
  7. 7.
    Ma, S., Sun, X., Lin, J., Wang, H.: Autoencoder as assistant supervisor: Improving text representation for Chinese social media text summarization (2018)Google Scholar
  8. 8.
    Shanshan, Y., Jindian, S., Pengfei, L.: Improved TextRank-based method for automatic summarization (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Yingbao Cui
    • 1
    Email author
  • Di Liu
    • 1
  • Qiang Li
    • 1
  • Zhen Qiu
    • 1
  • Xusheng Yang
    • 1
  1. 1.State Grid Information and Telecommunication GroupBeijingChina

Personalised recommendations