Concurrent Inference of Topic Models and Distributed Vector Representations

  • Debakar Shamanta
  • Sheikh Motahar Naim
  • Parang Saraf
  • Naren Ramakrishnan
  • M. Shahriar Hossain
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9285)


Topic modeling techniques have been widely used to uncover dominant themes hidden inside an unstructured document collection. Though these techniques first originated in the probabilistic analysis of word distributions, many deep learning approaches have been adopted recently. In this paper, we propose a novel neural network based architecture that produces distributed representation of topics to capture topical themes in a dataset. Unlike many state-of-the-art techniques for generating distributed representation of words and documents that directly use neighboring words for training, we leverage the outcome of a sophisticated deep neural network to estimate the topic labels of each document. The networks, for topic modeling and generation of distributed representations, are trained concurrently in a cascaded style with better runtime without sacrificing the quality of the topics. Empirical studies reported in the paper show that the distributed representations of topics represent intuitive themes using smaller dimensions than conventional topic modeling approaches.


Topic modeling Distributed representation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    AlSumait, L., Barbará, D., Domeniconi, C.: On-line lda: adaptive topic models for mining text streams with applications to topic detection and tracking. In: ICDM 2008, pp. 3–12 (2008)Google Scholar
  2. 2.
    Bengio, Y., Ducharme, R., Vincent, P., Janvin, C.: A neural probabilistic language model. Machine Learning Research 3, 1137–1155 (2003)Google Scholar
  3. 3.
    Blei, D., Lafferty, J.: Correlated topic models. Advances in Neural Information Processing Systems 18, 147 (2006)zbMATHGoogle Scholar
  4. 4.
    Blei, D.M., Lafferty, J.D.: Dynamic topic models. In: ICML 2006, pp. 113–120 (2006)Google Scholar
  5. 5.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. Machine Learning Research 3, 993–1022 (2003)Google Scholar
  6. 6.
    Cao, Z., Li, S., Liu, Y., Li, W., Ji, H.: A novel neural topic model and its supervised extension. In: AAAI 2015 (2015)Google Scholar
  7. 7.
    Chaitin, G.J.: Algorithmic information theory. Wiley Online Library (1982)Google Scholar
  8. 8.
    Chalmers, D.J.: Syntactic transformations on distributed representations. In: Connectionist Natural Language Processing, pp. 46–55. Springer (1992)Google Scholar
  9. 9.
    Deerwester, S.C., Dumais, S.T., Landauer, T.K., Furnas, G.W., Harshman, R.A.: Indexing by latent semantic analysis. American Society for Information Science 41(6), 391–407 (1990)CrossRefzbMATHGoogle Scholar
  10. 10.
    Dunn, J.C.: A fuzzy relative of the isodata process and its use in detecting compact well-separated clusters (1973)Google Scholar
  11. 11.
    Griffiths, T.L., Steyvers, M., Blei, D.M., Tenenbaum, J.B.: Integrating topics and syntax. In: NIPS 2004, pp. 537–544 (2004)Google Scholar
  12. 12.
    G. E. Hinton. Learning distributed representations of concepts. In: CogSci 1986, vol. 1, p. 12 (1986)Google Scholar
  13. 13.
    Hofmann, T.: Probabilistic latent semantic indexing. In: SIGIR 1999, pp. 50–57. ACM (1999)Google Scholar
  14. 14.
    Hummel, J.E., Holyoak, K.J.: Distributed representations of structure: A theory of analogical access and mapping. Psychological Review 104(3), 427 (1997)CrossRefGoogle Scholar
  15. 15.
    Larochelle, H., Lauly, S.: A neural autoregressive topic model. In: NIPS 2012, pp. 2708–2716 (2012)Google Scholar
  16. 16.
    Le, Q.V., Mikolov, T.: Distributed representations of sentences and documents. In: ICML 2014, pp, 1188–1196 (2014)Google Scholar
  17. 17.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space (2013). arXiv preprint arXiv:1301.3781
  18. 18.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NIPS 2013, pp. 3111–3119 (2013)Google Scholar
  19. 19.
    Pollack, J.B.: Recursive distributed representations. Artificial Intelligence 46(1), 77–105 (1990)CrossRefGoogle Scholar
  20. 20.
    Ramakrishnan, N., et al.: ‘Beating the news’ with EMBERS: Forecasting civil unrest using open source indicators. In: SIGKDD 2014, pp. 1799–1808 (2014)Google Scholar
  21. 21.
    Rousseeuw, P.J.: Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Computational and Applied Mathematics 20, 53–65 (1987)CrossRefzbMATHGoogle Scholar
  22. 22.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Cognitive Modeling 5, (1988)Google Scholar
  23. 23.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Enrichment or depletion of a go category within a class of genes: which test? Bioinformatics 23(4), 401–407 (2007)CrossRefGoogle Scholar
  24. 24.
    Steinley, D.: Properties of the hubert-arable adjusted rand index. Psychological Methods 9(3), 386 (2004)CrossRefzbMATHGoogle Scholar
  25. 25.
    Wallach, H.M.: Topic modeling: beyond bag-of-words. In: ICML 2006, pp. 977–984 (2006)Google Scholar
  26. 26.
    Wan, L., Zhu, L., Fergus, R.: A hybrid neural network-latent topic model. In: AISTATS 2012, pp. 1287–1294 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Debakar Shamanta
    • 1
  • Sheikh Motahar Naim
    • 1
  • Parang Saraf
    • 2
  • Naren Ramakrishnan
    • 2
  • M. Shahriar Hossain
    • 1
  1. 1.Department of Computer ScienceUniversity of Texas at El PasoEl PasoUSA
  2. 2.Department of Computer ScienceVirginia TechArlingtonUSA

Personalised recommendations