Abstract
In this paper, we provide a revised inference for correlated topic model (CTM) [3]. CTM is proposed by Blei et al. for modeling correlations among latent topics more expressively than latent Dirichlet allocation (LDA) [2] and has been attracting attention of researchers. However, we have found that the variational inference of the original paper is unstable due to almost-singularity of the covariance matrix when the number of topics is large. This means that we may be reluctant to use CTM for analyzing a large document set, which may cover a rich diversity of topics. Therefore, we revise the inference and improve its quality. First, we modify the formula for updating the covariance matrix in a manner that enables us to recover the original inference by adjusting a parameter. Second, we regularize posterior parameters for reducing a side effect caused by the formula modification. While our method is based on a heuristic intuition, an experiment conducted on large document sets showed that it worked effectively in terms of perplexity.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Asuncion, A., Welling, M., Smyth, P., Teh, Y.W.: On smoothing and inference for topic models. In: Proc. of UAI 2009, pp. 27–34 (2009)
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. JMLR 3, 993–1022 (2003)
Blei, D., Lafferty, J.: Correlated topic models. NIPS 18, 147–154 (2006)
Chen, C., Ding, N., Buntine, W.: Dependent hierarchical normalized random measures for dynamic topic modeling. In: Proc. of ICML 2012 (2012)
Chen, Y., Wiesel, A., Eldar, Y.C., Hero III, A.O.: Shrinkage algorithms for MMSE covariance estimation. IEEE Trans. on Sign. Proc. 58(10), 5016–5029 (2010)
Griffiths, T.L., Steyvers, M.: Finding scientific topics. PNAS 101(suppl. 1), 5228–5235 (2004)
Liu, D.C., Nocedal, J.: On the limited memory method for large scale optimization. Math. Programming B 45(3), 503–528 (1989)
Minka, T.P.: Estimating a Dirichlet distribution (2000), http://research.microsoft.com/en-us/um/people/minka/papers/dirichlet/
Newman, D., Karimi, S., Cavedon, L.: External Evaluation of Topic Models. In: Proc. of ADCS 2009, pp. 11–18 (2009)
Nocedal, J.: Updating quasi-Newton matrices with limited storage. Math. Comp. 35, 773–782 (1980)
Paisley, J., Wang, C., Blei, D.: The discrete infinite logistic normal distribution for mixed-membership modeling. In: Proc. of AISTATS 2011, pp. 74–82 (2011)
Salakhutdinov, R., Hinton, G.: Replicated Softmax: an Undirected Topic Model. NIPS 23, 1607–1614 (2009)
Wan, L., Zhu, L., Fergus, R.: A Hybrid Neural Network-Latent Topic Model. In: Proc. of AISTATS 2012, pp. 1287–1294 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Masada, T., Takasu, A. (2013). A Revised Inference for Correlated Topic Model. In: Guo, C., Hou, ZG., Zeng, Z. (eds) Advances in Neural Networks – ISNN 2013. ISNN 2013. Lecture Notes in Computer Science, vol 7952. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39068-5_54
Download citation
DOI: https://doi.org/10.1007/978-3-642-39068-5_54
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39067-8
Online ISBN: 978-3-642-39068-5
eBook Packages: Computer ScienceComputer Science (R0)