On Spectral Learning of Mixtures of Distributions

  • Dimitris Achlioptas
  • Frank McSherry
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3559)


We consider the problem of learning mixtures of distributions via spectral methods and derive a characterization of when such methods are useful. Specifically, given a mixture-sample, let \(\bar\mu_{i}, {\bar C_{i}}, \bar w_{i}\) denote the empirical mean, covariance matrix, and mixing weight of the samples from the i-th component. We prove that a very simple algorithm, namely spectral projection followed by single-linkage clustering, properly classifies every point in the sample provided that each pair of means \(\bar\mu_{i},\bar\mu_{j}\) is well separated, in the sense that \(\|\bar\mu_{i} - \bar\mu_{j}\|^{2}\) is at least \(\|{\bar C_{i}\|_{2}(1/\bar w_{i}+1/\bar w_{j})}\) plus a term that depends on the concentration properties of the distributions in the mixture. This second term is very small for many distributions, including Gaussians, Log-concave, and many others. As a result, we get the best known bounds for learning mixtures of arbitrary Gaussians in terms of the required mean separation. At the same time, we prove that there are many Gaussian mixtures {(μ i ,C i ,w i )} such that each pair of means is separated by ||C i ||2(1/w i  + 1/w j ), yet upon spectral projection the mixture collapses completely, i.e., all means and covariance matrices in the projected mixture are identical.


learning mixtures of distributions spectral methods singular value decomposition gaussians mixtures log-concave and concentrated distributions 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Arora, S., Kannan, R.: Learning mixtures of arbitrary Gaussians. In: Proc. 33rd ACM Symposium on Theory of Computation, pp. 247–257 (2001)Google Scholar
  2. 2.
    Dasgupta, S.: Learning mixtures of Gaussians. In: Proc. 40th IEEE Symposium on Foundations of Computer Science, pp. 634–644 (1999)Google Scholar
  3. 3.
    Dasgupta, S., Schulman, L.: A 2-round variant of EM for Gaussian mixtures. In: Proc. 16th Conference on Uncertainty in Artificial Intelligence, pp. 152–159 (2000)Google Scholar
  4. 4.
    Lindsay, B.: Mixture models: theory, geometry and applications. American Statistical Association, Virginia (2002)Google Scholar
  5. 5.
    Titterington, D.M., Smith, A.F.M., Makov, U.E.: Statistical analysis of finite mixture distributions. Wiley, Chichester (1985)zbMATHGoogle Scholar
  6. 6.
    Vempala, S., Wang, G.: A Spectral Algorithm of Learning Mixtures of Distributions. In: Proc. 43rd IEEE Symposium on Foundations of Computer Science, pp. 113–123 (2002)Google Scholar
  7. 7.
    Soshnikov, A.: A Note on Universality of the Distribution of the Largest Eigenvalues in Certain Sample Covariance Matrices. J. Stat. Phys. 108(5/6), 1033–1056 (2002)zbMATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Salmasian, H., Kannan, R., Vempala, S.: The Spectral Method for Mixture Models. In: Electronic Colloquium on Computational Complexity (ECCC), vol. (067) (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Dimitris Achlioptas
    • 1
  • Frank McSherry
    • 2
  1. 1.Microsoft Research, One Microsoft WayRedmondUSA
  2. 2.Microsoft ResearchMountain ViewUSA

Personalised recommendations