Multimedia Tools and Applications

, Volume 78, Issue 6, pp 6409–6440 | Cite as

Cross the data desert: generating textual-visual summary on the evolutionary microblog stream

  • Yu Xiong
  • Xiangmin Zhou
  • Yifei Zhang
  • Shi Feng
  • Daling WangEmail author


Effectively and efficiently summarizing social media is crucial and non-trivial to analyze social media. On social streams, events which are the main concept of semantic similar social messages, often bring us a firsthand story of daily news. However, to identify the valuable news, it is almost impossible to plough through millions of multi-modal messages one by one with traditional methods. Thus, it is urgent to summarize events with a few representative data samples on the streams. In this paper, we provide a vivid textual-visual media summarization approach for microblog streams, which exploits the incremental latent semantic analysis (LSA) of detected events. Firstly, with a novel weighting scheme for keyword relationship, we can detect and track daily sub-events on a keyword relation graph (WordGraph) of microblog streams effectively. Then, to summarize the stream with representative texts and images, we use cross-modal fusion to analyze the semantics of microblog texts and images incrementally and separately, with a novel incremental cross-modal LSA algorithm. The experimental results on a real microblog dataset show that our method is at least 1.31% better and 23.67% faster than existing state-of-the-art methods, and cross-modal fusion can improve the summarization performance by 4.16% on average.


Event detection and tracking Textual-visual summarization Incremental latent semantic analysis Cross-modal data fusion Social media event Microblog stream 



The project is supported by National Natural Science Foundation of China (61772122, 61402091).


  1. 1.
    Abdelhaq H, Sengstock C, Gertz M (2013) Eventweet: online localized event detection from twitter. Proc of VLDB Endow 6(12):1326–1329CrossRefGoogle Scholar
  2. 2.
    Aiello LM, Petkos G, Martin C, Corney D, Papadopoulos S, Skraba R, Goker A, Kompatsiaris I, Jaimes A (2013) Sensing trending topics in twitter. TMM 15(6):1268–1282Google Scholar
  3. 3.
    Alqadah F, Bhatnagar R (2011) A game theoretic framework for heterogenous information network clustering. In: Proc. of 17th KDD. ACM, pp 795–804Google Scholar
  4. 4.
    Atefeh F, Khreich W (2015) A survey of techniques for event detection in twitter. Comput Intell 31(1):132–164MathSciNetCrossRefGoogle Scholar
  5. 5.
    Bian J, Yang Y, Zhang H, Chua TS (2015) Multimedia summarization for social events in microblog stream. TMM 17(2):216–228Google Scholar
  6. 6.
    Cai H, Yang Y, Li X, Huang Z (2015) What are popular: exploring twitter features for event detection, tracking and visualization. In: Proc. of 23rd MM. ACM, pp 89–98Google Scholar
  7. 7.
    Cai H, Huang Z, Srivastava D, Zhang Q (2016) Indexing evolving events from tweet streams. In: Proc. of 32nd ICDE. IEEE, pp 1538–1539Google Scholar
  8. 8.
    Chang Y, Wang X, Mei Q, Liu Y (2013) Towards twitter context summarization with user influence models. In: Proc. of 6th WSDM. ACM, pp 527–536Google Scholar
  9. 9.
    Chen X, Candan KS (2014) Lwi-svd: low-rank, windowed, incremental singular value decompositions on time-evolving data sets. In: Proc. of 20th KDD. ACM, pp 987–996Google Scholar
  10. 10.
    Eitel A, Scheiter K, Schüler A, Nyström M, Holmqvist K (2013) How a picture facilitates the process of learning from text: evidence for scaffolding. Learn Instr 16:48–63CrossRefGoogle Scholar
  11. 11.
    Gao X, Cao J, Jin Z, Li X, Li J (2013) Gesodeck: a geo-social event detection and tracking system. In: Proc. of 21st MM. ACM, pp 471–472Google Scholar
  12. 12.
    Inouye D, Kalita JK (2011) Comparing twitter summarization algorithms for multiple post summaries. In: Proc. of 3rd PASSAT/SocialCom. IEEE Computer Society, pp 298–306Google Scholar
  13. 13.
    Jiang J, Tao X, Li K (2018) Dfc: density fragment clustering without peaks. J Intell Fuzzy Syst 34(1):525–536CrossRefGoogle Scholar
  14. 14.
    Kumar S, Udupa R (2011) Learning hash functions for cross-view similarity search. In: Proc. of 22nd IJCAI, pp 1360–1365Google Scholar
  15. 15.
    Long R, Wang H, Chen Y, Jin O, Yu Y (2011) Towards effective event detection, tracking and summarization on microblog data. In: Proc. of 12th WAIM. Springer, pp 652–663Google Scholar
  16. 16.
    Lu T, Jin Y, Su F, Shivakumara P, Tan CL (2015) Content-oriented multimedia document understanding through cross-media correlation. Multimed Tools Appl 74:8105–8135CrossRefGoogle Scholar
  17. 17.
    Nguyen DT, Jung JE (2017) Real-time event detection for online behavioral analysis of big social data. Future Gener Comput Syst 66:137–145CrossRefGoogle Scholar
  18. 18.
    Nichols J, Mahmud J, Drews C (2012) Summarizing sporting events using twitter. In: Proc. of 17th IUI. ACM, pp 189–198Google Scholar
  19. 19.
    Petrović S., Osborne M, Lavrenko V (2010) Streaming first story detection with application to twitter. In: Proc. of NAACL HLT’10. ACL, pp 181–189Google Scholar
  20. 20.
    Popescu AM, Pennacchiotti M, Paranjpe D (2011) Extracting events and event descriptions from twitter. In: Proc. of 20th WWW. ACM, pp 105–106Google Scholar
  21. 21.
    Qian S, Zhang T, Xu C, Shao J (2016) Multi-modal event topic model for social event analysis. TMM 18(2):233–246Google Scholar
  22. 22.
    Rafailidis D, Manolopoulou S, Daras P (2013) A unified framework for multimodal retrieval. Pattern Recognit 46(12):3358–3370CrossRefGoogle Scholar
  23. 23.
    Ramos AMS, Woloszyn V, Wives LK (2017) An experimental analysis of feature selection and similarity assessment for textual summarization. In: Proc. of 12th CCC. Springer, pp 146–155Google Scholar
  24. 24.
    Sakaki T, Okazaki M, Matsuo Y (2010) Earthquake shakes twitter users: real-time event detection by social sensors. In: Proc. of 19th WWW. ACM, pp 851–860Google Scholar
  25. 25.
    Sayyadi H, Raschid L (2013) A graph analytical approach for topic detection. ACM Trans Internet Technol 13(2):Article 4CrossRefGoogle Scholar
  26. 26.
    Shah RR, Yu Y, Verma A, Tang S, Shaikh AD, Zimmermann R (2016) Leveraging multimodal information for event summarization and concept-level sentiment analysis. Knowl-Based Syst 108:102–109CrossRefGoogle Scholar
  27. 27.
    Sharifi B, Hutton MA, Kalita JK (2010) Experiments in microblog summarization. In: Proc. of 2nd PASSAT/SocialCom. IEEE Computer Society, pp 685–688Google Scholar
  28. 28.
    Sorden SD (2013) The cognitive theory of multimedia learning. In: Irby BJ, Brown G, Lara-Alecio R, Jackson S (eds) Handbook of educational theories. chap. 8. Information Age Pub, pp 155–165Google Scholar
  29. 29.
    Sun Y, Aggarwal CC, Han J (2012) Relation strength-aware clustering of heterogeneous information networks with incomplete attributes. Proc VLDB Endow 5 (5):394–405CrossRefGoogle Scholar
  30. 30.
    Wang J, Korayem M, Blanco S, Crandall DJ (2016) Tracking natural events through social media and computer vision. In: Proc. of 24th MM. ACM, pp 1097–1101Google Scholar
  31. 31.
    Wang W, Ooi BC, Yang X, Zhang D, Zhuang Y (2014) Effective multi-modal retrieval based on stacked auto-encoders. Proc VLDB Endow 7(8):649–660CrossRefGoogle Scholar
  32. 32.
    Wang Z, Shou L, Chen K, Chen G, Mehrotra S (2015) On summarization and timeline generation for evolutionary tweet streams. TKDE 27(5):1301–1315Google Scholar
  33. 33.
    Wei S, Zhao Y, Yang T, Zhou Z, Ge S (2018) Enhancing heterogeneous similarity estimation via neighborhood reversibility. Multimed Tools Appl 77:1437–1452CrossRefGoogle Scholar
  34. 34.
    Wu G, Zhang L (2016) A new method for computing \(\varphi \)-functions and their condition numbers of large sparse matrices. ArXiv e-prints, pp 1–21Google Scholar
  35. 35.
    Wu F, Yu Z, Yang Y, Tang S, Zhang Y, Zhuang Y (2014) Sparse multi-modal hashing. TMM 16(2):427–439Google Scholar
  36. 36.
    Xiong Y, Wang D, Zhang Y, Feng S, Wang G (2014) Multimodal data fusion in text-image heterogeneous graph for social media recommendation. In: Proc. of 15th WAIM. Springer, pp LNCS 8485 96–99Google Scholar
  37. 37.
    Xiong Y, Zhang Y, Wang D, Feng S (2017) Picture or it didnt happen: catch the truth for events. Multimed Tools Appl 76(14):15,681–15,706CrossRefGoogle Scholar
  38. 38.
    Yan R, Wan X, Otterbacher J, Kong L, Li X, Zhang Y (2011) Evolutionary timeline summarization: a balanced optimization framework via iterative substitution. In: Proc. of 34th SIGIR. ACM, pp 745–754Google Scholar
  39. 39.
    Yang Y, Zha ZJ, Gao Y, Zhu X, Chua TS (2014) Exploiting web images for semantic video indexing via robust sample-specific loss. TMM 16(6):1677–1689Google Scholar
  40. 40.
    Yang Z, Li Q, Liu W, Ma Y, Cheng M (2017) Dual graph regularized nmf model for social event detection from flickr data. WWW 20(5):995–1015CrossRefGoogle Scholar
  41. 41.
    Yang Z, Li Q, Lu Z, Ma Y, Gong Z, Liu W (2017) Dual structure constrained multimodal feature coding for social event detection from flickr data. TOIT 17(2):Article No 19CrossRefGoogle Scholar
  42. 42.
    Yue G, Zhao S, Yang Y, Chua TS (2015) Multimedia social event detection in microblog. In: Proc. of 21st MMM. Springer, pp 269–281Google Scholar
  43. 43.
    Zhang W, Chen J, Shen J, Yu Y (2014) Location-based hierarchical event summary for social media photos. In: Proc. of 15th PCM. Springer, pp LNCS 8879 254–257Google Scholar
  44. 44.
    Zhou X, Chen L (2014) Event detection over twitter social media streams. VLDB J 23(3):381–400MathSciNetCrossRefGoogle Scholar
  45. 45.
    Zhou X, Chen L, Zhang Y, Cao L, Huang G, Wang C (2015) Online video recommendation in sharing community. In: Proc. of SIGMOD’15. ACM, pp 1645–1656Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Yu Xiong
    • 1
  • Xiangmin Zhou
    • 2
  • Yifei Zhang
    • 1
    • 3
  • Shi Feng
    • 1
    • 3
  • Daling Wang
    • 1
    • 3
    Email author
  1. 1.School of Computer Science and EngineeringNortheastern UniversityShenyangChina
  2. 2.School of Computer Science and Information TechnologyRMIT UniversityMelbourneAustralia
  3. 3.Key Laboratory of Medical Image Computing (Northeastern University)Ministry of EducationShenyangChina

Personalised recommendations