Skip to main content
SpringerLink
Log in

A survey of recent methods on deriving topics from Twitter: algorithm to evaluation

  • Survey Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

In recent years, studies related to topic derivation in Twitter have gained a lot of interest from businesses and academics. The interconnection between users and information has made social media, especially Twitter, an ultimate platform for propagation of information about events in real time. Many applications require topic derivation from this social media platform. These include, for example, disaster management, outbreak detection, situation awareness, surveillance, and market analysis. Deriving topics from Twitter is challenging due to the short content of the individual posts. The environment itself is also highly dynamic. This paper presents a review of recent methods proposed to derive topics from social media platform from algorithms to evaluations. With regard to algorithms, we classify them based on the features they exploit, such as content, social interactions, and temporal aspects. In terms of evaluations, we discuss the datasets and metrics generally used to evaluate the methods. Finally, we highlight the gaps in the research this far and the problems that remain to be addressed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. According to the Digital Policy Council (DPC) annual report on 2015 World Leader Ranking on Twitter [24], a total of 139 world leaders from 167 countries have an account in Twitter.

  2. http://trec.nist.gov/data/tweets/.

  3. http://trec.nist.gov/data/microblog.html, accessed April 24, 2019.

  4. https://github.com/zfz/twitter_corpus, accessed April 24, 2019.

  5. http://www.nltk.org.

References

  1. Aggarwal C, Subbian K (2014) Evolutionary network analysis: a survey. ACM Comput Surv 47(1):10:1–10:36. https://doi.org/10.1145/2601412

    Article  MATH  Google Scholar 

  2. Alghamdi R, Alfalqi K (2015) A survey of topic modeling in text mining. Int J Adv Comput Sci Appl (IJACSA) 6(1):147–153

    Google Scholar 

  3. Allan J (2002) Topic detection and tracking: event-based information organization, vol 12. Springer, Berlin

    Book  Google Scholar 

  4. AlSumait L, Barbarà D, Domeniconi C (2008) On-line LDA: Adaptive topic models for mining text streams with applications to topic detection and tracking. In: Proceedings of the 2008 eighth IEEE international conference on data mining. pp 3–12. https://doi.org/10.1109/ICDM.2008.140

  5. Alvarez-Melis D, Saveski M (2016) Topic modeling in twitter: aggregating tweets by conversations. In: Tenth international AAAI conference on web and social media

  6. Atefeh F, Khreich W (2015) A survey of techniques for event detection in twitter. Comput Intell 31(1):132–164

    Article  MathSciNet  Google Scholar 

  7. Bellegarda JR (1998) Exploiting both local and global constraints for multi-span statistical language modeling. In: Proceedings of the 1998 IEEE international conference on acoustics, speech and signal processing, 1998. vol 2, pp 677–680. https://doi.org/10.1109/ICASSP.1998.675355

  8. Bellegarda JR, Butzberger JW, Chow YL, Coccaro NB, Naik D (1996) A novel word clustering algorithm based on latent semantic analysis. In: Proceedings of the 1996 IEEE international conference on acoustics, speech, and signal processing, 1996. ICASSP-96. Conference proceedings. vol 1, pp 172–175. https://doi.org/10.1109/ICASSP.1996.540318

  9. Blei D, Ng A, Jordan M (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022

    MATH  Google Scholar 

  10. Blei DM (2012) Probabilistic topic models. Commun ACM 55(4):77–84

    Article  Google Scholar 

  11. Blei DM, Lafferty JD (2007) A correlated topic model of science. Ann Appl Stat 1:17–35

    Article  MathSciNet  Google Scholar 

  12. Campos R, Dias G, Jorge AM, Jatowt A (2014) Survey of temporal information retrieval and related applications. ACM Comput Surv 47(2):15:1–15:41. https://doi.org/10.1145/2619088

    Article  Google Scholar 

  13. Cao G, Nie JY, Gao J, Robertson S (2008) Selecting good expansion terms for pseudo-relevance feedback. In: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, pp 243–250

  14. Carlson N (2011) The real history of Twitter. http://www.businessinsider.com.au/how-twitter-was-founded-2011-4. Online, Accessed 6 Oct 2016

  15. Cataldi M, Di Caro L, Schifanella C (2010) Emerging topic detection on Twitter based on temporal and social terms evaluation. In: Proceedings of the tenth international workshop on multimedia data mining. ACM, New York, NY, USA, MDMKDD ’10, pp 4:1–4:10. https://doi.org/10.1145/1814245.1814249

  16. Chang J, Boyd-Graber J, Gerrish S, Wang C, Blei DM (2009) Reading tea leaves: how humans interpret topic models. In: Proceedings of the 22nd international conference on neural information processing systems (NIPS’09). Curran Associates Inc., Red Hook, NY, USA, pp 288–296

  17. Chen Y, Amiri H, Li Z, Chua TS (2013) Emerging topic detection for organizations from microblogs. In: Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, USA, SIGIR ’13, pp 43–52. https://doi.org/10.1145/2484028.2484057

  18. Cheng X, Yan X, Lan Y, Guo J (2014) BTM: Topic modeling over short texts. Trans Knowl Data Eng 26(12):2928–2941. https://doi.org/10.1109/TKDE.2014.2313872

    Article  Google Scholar 

  19. Cheong F, Cheong C (2011) Social media data mining: a social network analysis of tweets during the Australian 2010-2011 floods. In: Proceedings of the 15th Pacific Asia conference on information systems (PACIS). Queensland University of Technology, pp 1–16

  20. Chierichetti F, Kleinberg JM, Kumar R, Mahdian M, Pandey S (2014) Event detection via communication pattern analysis. In: ICWSM

  21. Cichocki A, Zdunek R, Amari Si (2006) New algorithms for non-negative matrix factorization in applications to blind source separation. In: Proceedings of the 2006 IEEE international conference on acoustics speech and signal processing proceedings. IEEE, vol 5, pp V–V

  22. Cong Y, Chen B, Liu H, Zhou M (2017) Deep latent dirichlet allocation with topic-layer-adaptive stochastic gradient Riemannian MCMC. In: Proceedings of the 34th international conference on machine learning-volume 70, JMLR. org, pp 864–873

  23. Cordero-Gutiérrez R, de la Prieta-Pintado F, Corchado-Rodríguez JM (2018) Decision support for digital marketing through virtual organizations-influencers on twitter. In: International conference on knowledge management in organizations. Springer, pp 574–585

  24. Council DP (2015) Researh note: world leader ranking on Twitter. http://www.digitaldaya.com/admin/modulos/galeria/pdfs/73/161_o59ontgs.pdf. Online, Accessed 6 Oct 2016

  25. Cover TM, Thomas JA (2012) Elements of information theory. Wiley, Hoboken

    MATH  Google Scholar 

  26. Deerwester S, Dumais ST, Furnas GW, Landauer TK, Harshman R (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci 41(6):391

    Article  Google Scholar 

  27. Devarajan K (2008) Nonnegative matrix factorization: an analytical and interpretive tool in computational biology. PLoS Comput Biol 4(7):e1000029

    Article  Google Scholar 

  28. Dietz L, Bickel S, Scheffer T (2007) Unsupervised prediction of citation influences. In: Proceedings of the 24th international conference on machine learning. ACM, New York, NY, USA, ICML ’07, pp 233–240. https://doi.org/10.1145/1273496.1273526

  29. Dubey A, Hefny A, Williamson S, Xing EP (2013) A nonparametric mixture model for topic modeling over time. In: Proceedings of the 2013 SIAM international conference on data mining. pp 530–538, https://doi.org/10.1137/1.9781611972832.59

  30. Edwards M, Rashid A, Rayson P (2015) A systematic survey of online data mining technology intended for law enforcement. ACM Comput Surv 48(1):15:1–15:54. https://doi.org/10.1145/2811403

    Article  Google Scholar 

  31. Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378

    Article  Google Scholar 

  32. Forsythe GE, Moler CB, Malcolm MA (1977) Computer methods for mathematical computations. Prentice-Hall, Upper Saddle River

    MATH  Google Scholar 

  33. Gaussier E, Goutte C (2005) Relation between PLSA and NMF and implications. In: Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 601–602

  34. Girolami M, Kabán A (2003) On an equivalence between PLSI and LDA. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in informaion retrieval. ACM, pp 433–434

  35. Gotoh Y, Renals S (1997) Document space models using latent semantic analysis

  36. Griffiths TL, Steyvers M (2004) Finding scientific topics. Proc Natl Acad Sci 101(suppl 1):5228–5235

    Article  Google Scholar 

  37. Guo W, Li H, Ji H, Diab MT (2013) Linking tweets to news: a framework to enrich short text data in social media. In: Proceedings of the 2013 association for computational linguistics conference. Citeseer, Sofia, Bulgaria, pp 239–249

  38. He Z, Xie S, Zdunek R, Zhou G, Cichocki A (2011) Symmetric nonnegative matrix factorization: algorithms and applications to probabilistic clustering. IEEE Trans Neural Netw 22(12):2117–2131

    Article  Google Scholar 

  39. Hermida A, Lewis SC, Zamith R (2014) Sourcing the arab spring: a case study of andy carvin’s sources on Twitter during the Tunisian and Egyptian revolutions. J Comput Med Commun 19(3):479–499

    Article  Google Scholar 

  40. Hoffman MD, Blei DM, Bach F (2010) Online learning for latent Dirichlet allocation. In: Proceedings of the 23rd international conference on neural information processing systems (NIPS’10), vol 1. Curran Associates Inc., Red Hook, NY, USA, pp 856–864

  41. Hofmann T (1999) Probabilistic latent semantic indexing. In: Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval. ACM, pp 50–57

  42. Hong L, Davison BD (2010) Empirical study of topic modeling in twitter. In: Proceedings of the first workshop on social media analytics. ACM, New York, NY, USA, SOMA ’10, pp 80–88. https://doi.org/10.1145/1964858.1964870

  43. Hoyer PO (2004) Non-negative matrix factorization with sparseness constraints. J Mach Learn Res 5(Nov):1457–1469

    MathSciNet  MATH  Google Scholar 

  44. Hu X, Sun N, Zhang C, Chua TS (2009) Exploiting internal and external semantics for the clustering of short texts using world knowledge. In: Proceedings of the 18th ACM conference on information and knowledge management. ACM, New York, NY, USA, CIKM ’09, pp 919–928. https://doi.org/10.1145/1645953.1646071

  45. Jamali M, Ester M (2010) A matrix factorization technique with trust propagation for recommendation in social networks. In: Proceedings of the fourth ACM conference on Recommender systems. ACM, pp 135–142

  46. Jelisavc̀ić V, Furlan B, Protić J, Milutinović V (2012) Topic models and advanced algorithms for profiling of knowledge in scientific papers. In: MIPRO, 2012 Proceedings of the 35th international convention. pp 1030–1035

  47. Jin O, Liu NN, Zhao K, Yu Y, Yang Q (2011) Transferring topical knowledge from auxiliary long texts for short text clustering. In: Proceedings of the 20th ACM international conference on information and knowledge management. ACM, New York, NY, USA, CIKM ’11, pp 775–784. https://doi.org/10.1145/2063576.2063689

  48. Joshi A, Sparks R, McHugh J, Karimi S, Paris C, MacIntyre RC (2019) Harnessing tweets for early detection of an acute disease event. Epidemiology 31:90–97

    Article  Google Scholar 

  49. Karimi S, Wang C, Metke-Jimenez A, Gaire R, Paris C (2015) Text and data mining techniques in adverse drug reaction detection. ACM Comput Surv 47(4):56:1–56:39. https://doi.org/10.1145/2719920

    Article  Google Scholar 

  50. Kim H, Park H (2007) Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis. Bioinformatics 23(12):1495–1502

    Article  Google Scholar 

  51. Kim J, Park H (2008) Sparse nonnegative matrix factorization for clustering

  52. Kireyev K, Palen L, Anderson K (2009) Applications of topics models to analysis of disaster-related Twitter data. In: Proceedings of the NIPS workshop on applications for topic models: text and beyond. Whistler, Canada, vol 1

  53. Kuang D, Park H, Ding C (2012) Symmetric nonnegative matrix factorization for graph clustering. In: Proceedings of the 2012 SIAM international conference on data mining. SIAM, California, USA, vol 12, pp 106–117

  54. Kullback S (1997) Information theory and statistics. Courier Dover Publications, Mineola

    MATH  Google Scholar 

  55. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  Google Scholar 

  56. Larsen ME, Boonstra TW, Batterham PJ, O’Dea B, Paris C, Christensen H (2015) We feel: mapping emotion on twitter. IEEE J Biomed Health Inform 19(4):1246–1252

    Article  Google Scholar 

  57. Lau JH, Collier N, Baldwin T (2012) On-line trend analysis with topic models:\(\backslash \)# Twitter trends detection topic model online. In: Proceedings of the 24th international conference on computational linguistics. Mumbai, India, pp 1519–1534

  58. Lee D, Seung H (2000) Algorithms for non-negative matrix factorization. In: Proceedings of the advances in neural information processing systems 13 (NIPS 2000). Denver, CO, USA, pp 556–562

  59. Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788–791

    Article  Google Scholar 

  60. Li C, Wang H, Zhang Z, Sun A, Ma Z (2016) Topic modeling for short texts with auxiliary word embeddings. In: Proceedings of the 39th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, USA, SIGIR ’16, pp 165–174. https://doi.org/10.1145/2911451.2911499

  61. Li W, Feng Y, Li D, Yu Z (2016) Micro-blog topic detection method based on BTM topic model and K-means clustering algorithm. Autom Control Comput Sci 50(4):271–277

    Article  Google Scholar 

  62. Lin J, Efron M, Wang Y, Sherman G (2014) Overview of the trec-2014 microblog track. Tech. rep., NIST. http://trec.nist.gov/pubs/trec23/trec2014.html

  63. Lin J, Efron M, Wang Y, Vorhees EM (2014) Overview of the trec-2015 microblog track. Tech. rep., NIST. http://trec.nist.gov/pubs/trec24/trec2015.html

  64. Liu Y, Niculescu-Mizil A, Gryc W (2009) Topic-link LDA: Joint models of topic and author community. In: Proceedings of the 26th annual international conference on machine learning. ACM, New York, NY, USA, ICML ’09, pp 665–672. https://doi.org/10.1145/1553374.1553460

  65. López-Sáncez D, Revuelta J, de la Prieta F, Corchado JM (2018) Towards the automatic identification and monitoring of radicalization activities in twitter. In: International conference on knowledge management in organizations. Springer, pp 589–599

  66. Lv C, Qiang R, Fan F, Yang J (2015) Proceedings of the information retrieval technology proceedings : 11th asia information retrieval societies conference, airs 2015, brisbane, qld, australia, december 2–4 (2015). Springer, Cham, pp 43–55

  67. Ma H, Yang H, Lyu MR, King I (2008) Sorec: social recommendation using probabilistic matrix factorization. In: Proceedings of the 17th ACM conference on Information and knowledge management. ACM, pp 931–940

  68. Ma H, Zhou D, Liu C, Lyu MR, King I (2011) Recommender systems with social regularization. In: Proceedings of the fourth ACM international conference on Web search and data mining. ACM, pp 287–296

  69. Ma HF, Sun YX, Jia MHZ, Zhang ZC (2014) Microblog hot topic detection based on topic model using term correlation matrix. In: Proceedings of the 2014 international conference on machine learning and cybernetics. vol 1, pp 126–130. https://doi.org/10.1109/ICMLC.2014.7009104

  70. Ma Z, Dou W, Wang X, Akella S (2013) Tag-Latent Dirichlet Allocation: Understanding hashtags and their relationships. In: 2013 IEEE/WIC/ACM international joint conferences on proceedings of the web intelligence (WI) and intelligent agent technologies (IAT). vol 1, pp 260–267

  71. Maletic JI, Valluri N (1999) Automatic software clustering via latent semantic analysis. In: Proceedings of the 14th IEEE international conference on automated software engineering. pp 251–254. https://doi.org/10.1109/ASE.1999.802296

  72. Manning C, Raghavan P, Schütze H (2008) Introduction to information retrieval, vol 1. Cambridge University Press, Cambridge

    Book  Google Scholar 

  73. Masada T, Kiyasu S, Miyahara S (2008) Comparing LDA with pLSI as a dimensionality reduction method in document clustering. In: Large-scale knowledge resources. Construction and application. Springer, pp 13–26

  74. McCallum A, Corrada-Emmanuel A, Wang X (2005) The author–recipient–topic model for topic and role discovery in social networks: experiments with enron and academic email

  75. McCallum A, Wang X, Corrada-Emmanuel A (2007) Topic and role discovery in social networks with experiments on enron and academic email. J Artif Intell Res 30:249–272

    Article  Google Scholar 

  76. Mehrotra R, Sanner S, Buntine W, Xie L (2013) Improving LDA topic models for microblogs via tweet pooling and automatic labeling. In: Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, USA, SIGIR ’13, pp 889–892. https://doi.org/10.1145/2484028.2484166

  77. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

  78. Mikolov T, Sutskever I, Chen K, Corrado G, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th international conference on neural information processing systems (NIPS’13), vol 2. Curran Associates Inc., Red Hook, NY, USA, pp 3111–3119

  79. Mimno D, Wallach H, Talley E, Leenders M, McCallum A (2011) Optimizing semantic coherence in topic models. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP), association for computational linguistics. pp 262–272

  80. Myers SA, Sharma A, Gupta P, Lin J (2014) Information network or social network?: The structure of the Twitter follow graph. In: Proceedings of the companion publication of the 23rd international conference on World wide web companion. International World Wide Web Conferences Steering Committee, pp 493–498

  81. Newman D, Noh Y, Talley E, Karimi S, Baldwin T (2010) Evaluating topic models for digital libraries. In: Proceedings of the 10th annual joint conference on digital libraries. ACM, New York, NY, USA, JCDL ’10, pp 215–224. https://doi.org/10.1145/1816123.1816156

  82. Nguyen DQ, Billingsley R, Du L, Johnson M (2015) Improving topic models with latent feature word representations. Trans Assoc Comput Linguist 3:299–313

    Article  Google Scholar 

  83. Nigam K, McCallum AK, Thrun S, Mitchell T (2000) Text classification from labeled and unlabeled documents using em. Mach Learn 39(2–3):103–134

    Article  Google Scholar 

  84. Nugroho R, Yang J, Zhong Y, Paris C, Nepal S (2015) Deriving topics in Twitter by exploiting tweet interactions. In: Proceedings of the 2015 IEEE international congress on big data. pp 87–94. https://doi.org/10.1109/BigDataCongress.2015.22

  85. Nugroho R, Zhong Y, Yang J, Paris C, Nepal S (2015) Matrix inter-joint factorization—a new approach for topic derivation in Twitter. In: Proceedings of the 2015 IEEE international congress on big data. pp 79–86. https://doi.org/10.1109/BigDataCongress.2015.21

  86. Nugroho R, Molla-Aliod D, Yang J, Zhong Y, Paris C, Nepal S (2016) Incorporating tweet relationships into topic derivation. In: Hasida K, Purwarianti A (eds) Proceedings of the computational linguistics: 14th international conference of the pacific association for computational linguistics. PACLING 2015, Bali, Indonesia, May 19-21, 2015, Revised Selected Papers, Springer Singapore, Singapore, pp 177–190

  87. Nugroho R, Zhao W, Yang J, Paris C, Nepal S (2016) Using time-sensitive interactions to improve topic derivation in Twitter. World Wide Web 20:1–27

    Google Scholar 

  88. Nurwidyantoro A, Winarko E (2013) Event detection in social media: a survey. In: Proceedings of the 2013 international conference on ICT for smart society (ICISS). IEEE, pp 1–5

  89. Ostrow A (2009) Japan earthquake shakes Twitter users... and beyonce. http://mashable.com/2009/08/12/japan-earthquake/#4IvI9oMp8kqd, [Online, Accessed 6 October 2016]

  90. Ozdikis O, Senkul P, Oguztuzun H (2012) Semantic expansion of tweet contents for enhanced event detection in Twitter. In: Proceedings of the 2012 IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM). pp 20–24. https://doi.org/10.1109/ASONAM.2012.14

  91. Phan XH, Nguyen LM, Horiguchi S (2008) Learning to classify short and sparse text and web with hidden topics from large-scale data collections. In: Proceedings of the 17th international conference on World Wide Web. ACM, New York, NY, USA, WWW ’08, pp 91–100. https://doi.org/10.1145/1367497.1367510

  92. Phan XH, Nguyen CT, Le DT, Nguyen LM, Horiguchi S, Ha QT (2011) A hidden topic-based framework toward building applications with short web documents. IEEE Trans Knowl Data Eng 23(7):961–976. https://doi.org/10.1109/TKDE.2010.27

    Article  Google Scholar 

  93. Prateek M, Vasudeva V (2016) Improved topic models for social media via community detection using user interaction and content similarity. In: Proceedings of the 2016 international fruct conference on intelligence, social media and web (ISMW FRUCT). pp 1–7. https://doi.org/10.1109/FRUCT.2016.7584770

  94. Prier KW, Smith MS, Giraud-Carrier C, Hanson CL (2011) Identifying health-related topics on Twitter. In: Salerno J, Yang SJ, Nau D, Chai SK (eds) Proceedings of the social computing, behavioral-cultural modeling and prediction: 4th international conference, SBP 2011. College Park, MD, USA, March 29–31, 2011., Springer, Berlin, Heidelberg, pp 18–25

  95. Qiu M, Zhu F, Jiang J (2013) It is not just what we say, but how we say them: LDA-based behavior-topic model. In: Proceedings of the 2013 SIAM international conference on data mining. pp 794–802. https://doi.org/10.1137/1.9781611972832.88

  96. Rafea A, Mostafa NA (2013) Topic extraction in social media. In: 2013 International conference on collaboration technologies and systems (CTS). IEEE, pp 94–98

  97. Rafeeque PC, Sendhilkumar S (2011) A survey on short text analysis in web. In: Proceedings of the 2011 third international conference on advanced computing. pp 365–371. https://doi.org/10.1109/ICoAC.2011.6165203

  98. Rajani NFN, McArdle K, Baldridge J (2014) Extracting topics based on authors, recipients and content in microblogs. In: Proceedings of the 37th international acm sigir conference on research and development in information retrieval. ACM, New York, NY, USA, SIGIR ’14, pp 1171–1174. https://doi.org/10.1145/2600428.2609537

  99. Ramage D, Hall D, Nallapati R, Manning CD (2009) Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora. In: Proceedings of the 2009 conference on empirical methods in natural language processing: volume 1–volume 1, association for computational linguistics. Stroudsburg, PA, USA, EMNLP ’09, pp 248–256. http://dl.acm.org/citation.cfm?id=1699510.1699543

  100. Ramage D, Dumais ST, Liebling DJ (2010) Characterizing microblogs with topic models, vol 10. AAAI, Washington, pp 130–137

    Google Scholar 

  101. Rosen-Zvi M, Griffiths T, Steyvers M, Smyth P (2004) The author-topic model for authors and documents. In: Proceedings of the 20th conference on uncertainty in artificial intelligence. AUAI Press, Arlington, Virginia, United States, UAI ’04, pp 487–494. http://dl.acm.org/citation.cfm?id=1036843.1036902

  102. Rosen-Zvi M, Chemudugunta C, Griffiths T, Smyth P, Steyvers M (2010) Learning author–topic models from text corpora. ACM Trans Inf Syst 28(1):4:1–4:38. https://doi.org/10.1145/1658377.1658381

    Article  Google Scholar 

  103. Saha A, Sindhwani V (2012) Learning evolving and emerging topics in social media: a dynamic NMF approach with temporal regularization. In: Proceedings of the fifth ACM international conference on web search and data mining. ACM, New York, NY, USA, WSDM ’12, pp 693–702. https://doi.org/10.1145/2124295.2124376

  104. Sánchez DL, Revuelta J, De la Prieta F, Gil-González AB, Dang C (2016) Twitter user clustering based on their preferences and the Louvain algorithm. In: International conference on practical applications of agents and multi-agent systems. Springer, pp 349–356

  105. Shahnaz F, Berry MW, Pauca VP, Plemmons RJ (2006) Document clustering using nonnegative matrix factorization. Inf Process Manag 42(2):373–386

    Article  Google Scholar 

  106. Silva JA, Faria ER, Barros RC, Hruschka ER, Carvalho ACPLF, Gama JA (2013) Data stream clustering: a survey. ACM Comput Surv 46(1):13:1–13:31. https://doi.org/10.1145/2522968.2522981

    Article  MATH  Google Scholar 

  107. Song G, Ye Y, Du X, Huang X, Bie S (2014) Short text classification: a survey. J Multimedia 9(5):635–643

    Article  Google Scholar 

  108. Sparks RS (2018) Sentiment monitoring of social media from Oceania. Glob J Med Res 18(5-K). Retrieved from https://medicalresearchjournal.org/index.php/GJMR/article/view/1568

  109. Steyvers M, Smyth P, Rosen-Zvi M, Griffiths T (2004) Probabilistic author-topic models for information discovery. In: Proceedings of the tenth ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, USA, KDD ’04, pp 306–315. https://doi.org/10.1145/1014052.1014087

  110. Stilo G, Velardi P (2014) Time makes sense: event discovery in Twitter using temporal similarity. In: Proceedings of the 2014 IEEE/WIC/ACM international joint conferences on web intelligence (WI) and intelligent agent technologies (IAT)—Volume 02. IEEE Computer Society, Washington, DC, USA, WI-IAT ’14, pp 186–193. https://doi.org/10.1109/WI-IAT.2014.97

  111. Strehl A, Ghosh J (2003) Cluster ensembles—a knowledge reuse framework for combining multiple partitions. J Mach Learn Res 3:583–617

    MathSciNet  MATH  Google Scholar 

  112. Taslaman L, Nilsson B (2012) A framework for regularized non-negative matrix factorization, with application to the analysis of gene expression data. PLoS ONE 7(11):e46331

    Article  Google Scholar 

  113. Teh YW, Jordan MI, Beal MJ, Blei DM (2004) Sharing clusters among related groups: hierarchical Dirichlet processes. In: Proceedings of the 17th international conference on neural information processing systems (NIPS’04). MIT Press, Cambridge, MA, USA, pp 1385–1392

    Google Scholar 

  114. Tsur O, Littman A, Rappoport A (2013) Efficient clustering of short messages into general domains. In: Proceedings of the international AAAI conference on web and social media. http://www.aaai.org/ocs/index.php/ICWSM/ICWSM13/paper/view/6103

  115. Twitter (2011) #numbers. https://blog.twitter.com/2011/numbers. Online, Accessed 6 Oct 2016

  116. Twitter (2011) Twitter milestones. https://about.twitter.com/company/press/milestones. Online, Accessed 6 Oct 2016

  117. Vosecky J, Jiang D, Leung KWT, Xing K, Ng W (2014) Integrating social and auxiliary semantics for multifaceted topic modeling in Twitter. ACM Trans Internet Technol (TOIT) 14(4):27

    Article  Google Scholar 

  118. Wallach HM, Murray I, Salakhutdinov R, Mimno D (2009) Evaluation methods for topic models. In: Proceedings of the 26th annual international conference on machine learning. ACM, New York, NY, USA, ICML ’09, pp 1105–1112. https://doi.org/10.1145/1553374.1553515

  119. Wan S, Paris C (2014) Improving government services with social media feedback. In: Proceedings of the 19th international conference on intelligent user interfaces. ACM, New York, NY, USA, IUI ’14, pp 27–36. https://doi.org/10.1145/2557500.2557513

  120. Wang X, McCallum A (2006) Topics over time: A non-markov continuous-time model of topical trends. In: Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, USA, KDD ’06, pp 424–433. https://doi.org/10.1145/1150402.1150450

  121. Wang X, Gerber MS, Brown DE (2012) Automatic crime prediction using events extracted from Twitter posts. In: Yang SJ, Greenberg AM, Endsley M (eds) Proceedings of the social computing, behavioral—cultural modeling and prediction: 5th international conference, SBP 2012. College Park, MD, USA, April 3–5, 2012. Proceedings, Springer, Berlin, Heidelberg, pp 231–238

  122. Wang Y, Agichtein E, Benzi M (2012) TM-LDA: Efficient online modeling of latent topic transitions in social media. In: Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, USA, KDD ’12, pp 123–131. https://doi.org/10.1145/2339530.2339552

  123. Wang Y, Liu J, Qu J, Huang Y, Chen J, Feng X (2014) Hashtag graph based topic model for tweet mining. In: Proceedings of the 2014 IEEE international conference on data mining. pp 1025–1030. https://doi.org/10.1109/ICDM.2014.60

  124. Weng J, Lim EP, Jiang J, He Q (2010) Twitterrank: finding topic-sensitive influential Twitterers. In: Proceedings of the third ACM international conference on Web search and data mining. ACM, pp 261–270

  125. Xia H, Li J, Tang J, Moens MF (2012) Plink-lda: Using link as prior information in topic modeling. In: Lee Sg, Peng Z, Zhou X, Moon YS, Unland R, Yoo J (eds) Proceedings of the database systems for advanced applications: 17th international conference, DASFAA 2012. Busan, South Korea, April 15–19, 2012, Part I, Springer, Berlin, Heidelberg, pp 213–227

  126. Xie W, Zhu F, Jiang J, Lim EP, Wang K (2016) Topicsketch: real-time bursty topic detection from twitter. IEEE Trans Knowl Data Eng 28(8):2216–2229

    Article  Google Scholar 

  127. Xu J, Liu P, Wu G, Sun Z, Xu B, Hao H (2013) A fast matching method based on semantic similarity for short texts. In: Zhou G, Li J, Zhao D, Feng Y (eds) Proceedings of the natural language processing and chinese computing: second CCF conference, NLPCC 2013. Chongqing, China, November 15–19, 2013,, Springer, Berlin, Heidelberg, pp 299–309

  128. Yan X, Guo J, Lan Y, Cheng X (2013) A biterm topic model for short texts. In: Proceedings of the 22nd international conference on World Wide Web. ACM, New York, NY, USA, WWW ’13, pp 1445–1456. https://doi.org/10.1145/2488388.2488514

  129. Yan X, Guo J, Liu S, Cheng X, Wang Y (2013) Learning topics in short texts by non-negative matrix factorization on term correlation matrix. In: Proceedings of the SIAM international conference on data mining (SIAM 2013). SDM, San Diego, California, USA

  130. Yang Z, Yuan Z, Laaksonen J (2007) Projective non-negative matrix factorization with applications to facial image processing. Int J Pattern Recognit Artif Intell 21(08):1353–1362

    Article  Google Scholar 

  131. Yıldırım A, Üsküdarlı S, Özgür A (2016) Identifying topics in microblogs using wikipedia. PLoS ONE 11(3):e0151885

    Article  Google Scholar 

  132. Zhang C, Lu S, Zhang C, Xiao X, Wang Q, Chen G (2019) A novel hot topic detection framework with integration of image and short text information from twitter. IEEE Access 7:9225–9231

    Article  Google Scholar 

  133. Zhao H, Du L, Buntine W, Zhou M (2018) Dirichlet belief networks for topic structure learning. In: Advances in neural information processing systems. pp 7955–7966

  134. Zhao Y, Karypis G (2001) Criterion functions for document clustering: experiments and analysis. Tech. rep, Citeseer

  135. Zhong Y, Yang J, Nugroho R (2015) Incorporating tie strength in robust social recommendation. In: Proceedings of the 4th IEEE international congress on big data. IEEE Services Computing Community, New York, USA, pp 63–70

  136. Zhou T, Shan H, Banerjee A, Sapiro G (2012) Kernelized probabilistic matrix factorization: exploiting graphs and side information. In: Proceedings of the 2012 SIAM international conference on data mining. SIAM, California, USA, vol 12, pp 403–414

  137. Zuo Y, Zhao J, Xu K (2016) Word network topic model: a simple but general solution for short and imbalanced texts. Knowl Inf Syst 48(2):379–398

    Article  Google Scholar 

Download references

Acknowledgements

This work is partially supported by the CSIRO Data61, Macquarie University, Soegijapranata Catholic University, The Australian Research Council LP120200231, and The Australian Research Council DP140101369.

Author information

Authors and Affiliations

  1. Soegijaparanata Catholic University, Semarang, Indonesia

    Robertus Nugroho

  2. CSIRO Data61 Australia, Marsfield, Australia

    Robertus Nugroho, Cecile Paris & Surya Nepal

  3. Macquarie University, Macquarie Park, Australia

    Jian Yang

  4. Donghua University, Changning, China

    Weiliang Zhao

Authors
  1. Robertus Nugroho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cecile Paris
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Surya Nepal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jian Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weiliang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robertus Nugroho.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nugroho, R., Paris, C., Nepal, S. et al. A survey of recent methods on deriving topics from Twitter: algorithm to evaluation. Knowl Inf Syst 62, 2485–2519 (2020). https://doi.org/10.1007/s10115-019-01429-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-019-01429-z

Keywords

Search

Navigation