World Wide Web

, Volume 18, Issue 5, pp 1201–1217 | Cite as

Mining streams of short text for analysis of world-wide event evolutions

  • Guangyan Huang
  • Jing He
  • Yanchun Zhang
  • Wanlei Zhou
  • Hai Liu
  • Peng Zhang
  • Zhiming Ding
  • Yue You
  • Jian Cao


Streams of short text, such as news titles, enable us to effectively and efficiently learn the real world events that occur anywhere and anytime. Short text messages that are companied by timestamps and generally brief events using only a few words differ from other longer text documents, such as web pages, news stories, blogs, technical papers and books. For example, few words repeat in the same news titles, thus frequency of the term (i.e., TF) is not as important in short text corpus as in longer text corpus. Therefore, analysis of short text faces new challenges. Also, detecting and tracking events through short text analysis need to reliably identify events from constant topic clusters; however, existing methods, such as Latent Dirichlet Allocation (LDA), generates different topic results for a corpus at different executions. In this paper, we provide a Finding Topic Clusters using Co-occurring Terms (FTCCT) algorithm to automatically generate topics from a short text corpus, and develop an Event Evolution Mining (EEM) algorithm to discover hot events and their evolutions (i.e., the popularity degrees of events changing over time). In FTCCT, a term (i.e., a single word or a multiple-words phrase) belongs to only one topic in a corpus. Experiments on news titles of 157 countries within 4 months (from July to October, 2013) demonstrate that our FTCCT-based method (combining FTCCT and EEM) achieves far higher quality of the event’s content and description words than LDA-based method (combining LDA and EEM) for analysis of streams of short text. Our method also visualizes the evolutions of the hot events. The discovered world-wide event evolutions have explored some interesting correlations of the world-wide events; for example, successive extreme weather phenomenon occur in different locations - typhoon in Hong Kong and Philippines followed hurricane and storm flood in Mexico in September 2013.


Text mining Clustering Topic discovery Streams of short text Event evolutions 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Allan, J., Papka, R., and Lavrenko, V.: On-line new event detection and traking, Proc. of the 21st ACM International Conference on Research and Development in Information Retrieval, 1998.Google Scholar
  2. 2.
    Becker, H., Naaman, M., and Gravano, L.: Beyond trending topics: real-world event identification on Twitter, ICWSM′11, 2011.Google Scholar
  3. 3.
    Blei, D. M., and Lafferty, J. D.: Dynamic topic models, Proc. of ICML′06, pp. 113–120, 2006.Google Scholar
  4. 4.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J Mach Learn Res 3, 993–1022 (2003)zbMATHGoogle Scholar
  5. 5.
    Cui, W., Liu, S., Tan, L., Shi, C., Song, Y., Gao, Z.J., Tong, X., Qu, H.: Textflow: towards better understanding of evolving topics in text. IEEE Trans Vis Comput Graph 17(12), 2412–2421 (2011)CrossRefGoogle Scholar
  6. 6.
    Dou, W., Wang, X., Skau, D., Ribarsky, W., and Zhou, M. X.: LeadLine: interactive visual analysis of text data through event identification and exploration, Proc. of IEEE Conference on Visual Analytics Science and Technology, pp. 93–102, 2012.Google Scholar
  7. 7.
    Ester, M., Kregel, H. P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise, SIGKDD′96, pp. 226–231, 1996.Google Scholar
  8. 8.
    Ha-Thuc, V., Mejova, Y., Harris C., and Srinivasan, P.: A relevance-based topic model for news event tracking, SIGIR′09, pp. 764–765, 2009.Google Scholar
  9. 9.
    Havre, S., Hetzler, B., Nowell, L.: ThemeRiver: visualizing theme changes over time, Proc. of IEEE Symposium on Information Visualization, 2000.Google Scholar
  10. 10.
    Hu, Y., John, A., Wang, F., Kambhampati, S.: Et-lda: Joint topic modeling for aligning events and their twitter feedback, AAAI′12, pp. 59–65, 2012.Google Scholar
  11. 11.
    Jaccard, P.: Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bull Soc vaud sci nat 37, 547–579 (1901)Google Scholar
  12. 12.
    Kleinberg, J.: Bursty and hierarchical structure in streams, Proc. of ACM SIGKDD′02, 2002.Google Scholar
  13. 13.
    Kleinberg, J.: Temporal dynamics of on-line information streams, In Data Stream Management: Processing High-Speed Data, 2006.Google Scholar
  14. 14.
    Lau, J., Collier, N., Baldwin, T.: On-line trend analysis with topic models: twitter trends detection topic model online, Proc. of COLING′12, pp. 1519–1534, 2012.Google Scholar
  15. 15.
    Michelson, M., Macskassy, S. A.: Discovering users topics of interest on twitter: a first look. Proc. of 4th Workshop on Analytics for Noisy Unstructured Text Data, 2010.Google Scholar
  16. 16.
    Reed, C.: Latent Dirichlet allocation: towards a deeper understanding,, 2012.Google Scholar
  17. 17.
    Robertson, S.: Understanding inverse document frequency: on theoretical arguments for IDF. J Doc 60(5), 503–520 (2004)CrossRefGoogle Scholar
  18. 18.
    Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Inf Process Manag 24(5), 513–523 (1988)CrossRefGoogle Scholar
  19. 19.
    Shin, Y., Ryo, C., Park, J.: Automatic extraction of persistent topics from social text streams. World Wide Web J (2013). doi: 10.1007/s11280-013-0251-3 Google Scholar
  20. 20.
    Van Rijsbergen, C.J.: A theoretical basis for the use of co-occurrence data in information retrieval. J Doc 33(2), 106–119 (1977)CrossRefGoogle Scholar
  21. 21.
    Wang, C., Blei, D. M., and Heckerman, D.: Continuous time dynamic topic models, Proc. of the 24th Conference on Uncertainty in Artificial Intelligence (UAI′08), pp. 579–586, 2008.Google Scholar
  22. 22.
    Wang, X. and McCallum, A.: Topics over time: a non-Markov continuous time model of topical trends, Proc. of ACM SIGKDD′06, pp. 424–433, 2006.Google Scholar
  23. 23.
    Wang, X., Mohanty, N., and McCallum, A.: Group and topic discovery from relations and text, Proc. of SIGKDD Workshop on Link Discovery: Issues, Approaches and Applications (LinkKDD′05), pp. 28–35, 2005.Google Scholar
  24. 24.
    You, Y., Huang, G., Cao, J., Chen, E., He, J., Zhang, Y., and Hu, L.: GEAM: A general and event-related aspects model for twitter event detection, WISE 2013, Part II, LNCS 8181, pp. 319–332, 2013.Google Scholar
  25. 25.
    Zhang, J., Song, Y., Zhang, C., and Liu, S.: Evolutionary hierarchical dirichlet processes for multiple correlated time-varying corpora, Proc. of ACM SIGKDD′10, pp. 1079–1088, 2010.Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Guangyan Huang
    • 1
  • Jing He
    • 2
  • Yanchun Zhang
    • 2
  • Wanlei Zhou
    • 1
  • Hai Liu
    • 3
  • Peng Zhang
    • 2
  • Zhiming Ding
    • 4
  • Yue You
    • 5
  • Jian Cao
    • 5
  1. 1.School of Information TechnologyDeakin UniversityVictoriaAustralia
  2. 2.Centre for Applied Informatics, College of Engineering and ScienceVictoria UniversityVictoriaAustralia
  3. 3.Computer Science DepartmentSouth China Normal UniversityGuangzhouChina
  4. 4.State Key Laboratory of Computer Science, Institute of SoftwareChinese Academy of SciencesBeijingChina
  5. 5.Department of Computer Science and EngineeringShanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations