Subspace Tracking for Latent Semantic Analysis

  • Radim Řehůřek
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6611)


Modern applications of Latent Semantic Analysis (LSA) must deal with enormous (often practically infinite) data collections, calling for a single-pass matrix decomposition algorithm that operates in constant memory w.r.t. the collection size. This paper introduces a streamed distributed algorithm for incremental SVD updates. Apart from the theoretical derivation, we present experiments measuring numerical accuracy and runtime performance of the algorithm over several data collections, one of which is the whole of the English Wikipedia.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Berry, M.: Large-scale sparse singular value computations. The International Journal of Supercomputer Applications 6(1), 13–49 (1992)Google Scholar
  2. 2.
    Brand, M.: Fast low-rank modifications of the thin singular value decomposition. Linear Algebra and its Applications 415(1), 20–30 (2006)MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Comon, P., Golub, G.: Tracking a few extreme singular values and vectors in signal processing. Proceedings of the IEEE 78(8), 1327–1343 (1990)CrossRefGoogle Scholar
  4. 4.
    Deerwester, S., Dumais, S., Furnas, G., Landauer, T., Harshman, R.: Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science 41(6), 391–407 (1990)CrossRefGoogle Scholar
  5. 5.
    Golub, G., Van Loan, C.: Matrix computations. Johns Hopkins University Press, Baltimore (1996)MATHGoogle Scholar
  6. 6.
    Gorrell, G., Webb, B.: Generalized hebbian algorithm for incremental Latent Semantic Analysis. In: Ninth European Conference on Speech Communication and Technology (2005)Google Scholar
  7. 7.
    Griffiths, T., Steyvers, M., Tenenbaum, J.: Topics in semantic representation. Psychological Review 114(2), 211–244 (2007)CrossRefGoogle Scholar
  8. 8.
    Halko, N., Martinsson, P., Tropp, J.: Finding structure with randomness: Stochastic algorithms for constructing approximate matrix decompositions. arXiv 909 (2009)Google Scholar
  9. 9.
    Levy, A., Lindenbaum, M.: Sequential Karhunen–Loeve basis extraction and its application to images. IEEE Transactions on Image Processing 9(8), 1371 (2000)CrossRefMATHGoogle Scholar
  10. 10.
    Salton, G.: Automatic text processing: the transformation, analysis, and retrieval of information by computer. Addison-Wesley Longman Publishing Co., Inc., Boston (1989)Google Scholar
  11. 11.
    Vigna, S.: Distributed, large-scale latent semantic analysis by index interpolation. In: Proceedings of the 3rd International Conference on Scalable Information Systems, pp. 1–10. ICST (2008)Google Scholar
  12. 12.
    William, H., Teukolsky, S., Vetterling, W.: Numerical Recipes in C: The art of scientific computing. Cambridge University Press, New York (1988)MATHGoogle Scholar
  13. 13.
    Zha, H., Marques, O., Simon, H.: Large-scale SVD and subspace-based methods for Information Retrieval. Solving Irregularly Structured Problems in Parallel, 29–42 (1998)Google Scholar
  14. 14.
    Zha, H., Simon, H.: On updating problems in Latent Semantic Indexing. SIAM Journal on Scientific Computing 21, 782 (1999)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Zha, H., Zhang, Z.: Matrices with low-rank-plus-shift structure: Partial SVD and Latent Semantic Indexing. SIAM Journal on Matrix Analysis and Applications 21, 522 (2000)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Řehůřek, R., Sojka, P.: Software Framework for Topic Modelling with Large Corpora. In: Proceedings of LREC 2010 Workshop on New Challenges for NLP Frameworks, pp. 45–50 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Radim Řehůřek
    • 1
  1. 1.NLP labMasaryk University in BrnoCzech Republic

Personalised recommendations