Skip to main content
SpringerLink
Log in

Foundations of a Multi-way Spectral Clustering Framework for Hybrid Linear Modeling

  • Published:
Foundations of Computational Mathematics Aims and scope Submit manuscript

Abstract

The problem of Hybrid Linear Modeling (HLM) is to model and segment data using a mixture of affine subspaces. Different strategies have been proposed to solve this problem, however, rigorous analysis justifying their performance is missing. This paper suggests the Theoretical Spectral Curvature Clustering (TSCC) algorithm for solving the HLM problem and provides careful analysis to justify it. The TSCC algorithm is practically a combination of Govindu’s multi-way spectral clustering framework (CVPR 2005) and Ng et al.’s spectral clustering algorithm (NIPS 2001). The main result of this paper states that if the given data is sampled from a mixture of distributions concentrated around affine subspaces, then with high sampling probability the TSCC algorithm segments well the different underlying clusters. The goodness of clustering depends on the within-cluster errors, the between-clusters interaction, and a tuning parameter applied by TSCC. The proof also provides new insights for the analysis of Ng et al. (NIPS 2001).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. S. Agarwal, K. Branson, S. Belongie, Higher order learning with graphs, in Proceedings of the 23rd International Conference on Machine learning, vol. 148 (2006), pp. 17–24.

  2. S. Agarwal, J. Lim, L. Zelnik-Manor, P. Perona, D. Kriegman, S. Belongie, Beyond pairwise clustering, in Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 2 (2005), pp. 838–845.

  3. E. Arias-Castro, D. Donoho, X. Huo, Near-optimal detection of geometric objects by fast multiscale methods, IEEE Trans. Inf. Theory 51(7) (2005).

  4. B. Bader, T. Kolda, Algorithm 862: MATLAB tensor classes for fast algorithm prototyping. ACM Trans. Math. Softw. 32(4), 635–653 (2006). http://www.citeulike.org/user/bamberg/article/2875626

    Article  MathSciNet  Google Scholar 

  5. P. Bradley, O. Mangasarian, k-plane clustering, J. Glob. Optim. 16(1), 23–32 (2000).

    Article  MATH  MathSciNet  Google Scholar 

  6. M. Brand, K. Huang, A unifying theorem for spectral embedding and clustering, in Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, January 2003.

  7. G. Chen, G. Lerman, Spectral curvature clustering (SCC), Int. J. Comput. Vis. 81(3), 317–330 (2009).

    Article  Google Scholar 

  8. J. Costeira, T. Kanade, A multibody factorization method for independently moving objects, Int. J. Comput. Vis. 29(3), 159–179 (1998).

    Article  Google Scholar 

  9. M. Fischler, R. Bolles, Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, Commun. ACM 24(6), 381–395 (1981).

    Article  MathSciNet  Google Scholar 

  10. G. Golub, C. Van Loan, Matrix Computations (John Hopkins University Press, Baltimore, 1996).

    MATH  Google Scholar 

  11. V. Govindu, A tensor decomposition for geometric grouping and segmentation, in CVPR, vol. 1, June 2005, pp. 1150–1157.

  12. P. Gruber, F. Theis, Grassmann clustering, in Proc. EUSIPCO 2006, Florence, Italy, 2006.

  13. G. Haro, G. Randall, G. Sapiro, Translated Poisson mixture model for stratification learning, Int. J. Comput. Vis. 80(3), 358–374 (2008).

    Article  Google Scholar 

  14. J. Ho, M. Yang, J. Lim, K. Lee, D. Kriegman, Clustering appearances of objects under varying illumination conditions, in Proceedings of International Conference on Computer Vision and Pattern Recognition, vol. 1 (2003), pp. 11–18.

  15. A. Hyvärinen, E. Oja, Independent component analysis: algorithms and applications, Neural Netw. 13(4–5), 411–430 (2000).

    Article  Google Scholar 

  16. A. Kambhatla, T. Leen, Fast non-linear dimension reduction, in Advances in Neural Information Processing Systems 6, (1994), pp. 152–159.

  17. K. Kanatani, Motion segmentation by subspace separation and model selection, in Proc. of 8th ICCV, vol. 3, Vancouver, Canada (2001), pp. 586–591.

  18. K. Kanatani, Evaluation and selection of models for motion segmentation, in 7th ECCV, vol. 3, May 2002, pp. 335–349.

  19. D. Kushnir, M. Galun, A. Brandt, Fast multiscale clustering and manifold identification, Pattern Recognit. 39(10), 1876–1891 (2006).

    Article  MATH  Google Scholar 

  20. L. De Lathauwer, B. De Moor, J. Vandewalle, A multilinear singular value decomposition, SIAM J. Matrix Anal. A 21(4), 1253–1278 (2000).

    Article  MATH  Google Scholar 

  21. G. Lerman, J.T. Whitehouse, On d-dimensional d-semimetrics and simplex-type inequalities for high-dimensional sine functions. J. Approx. Theory 56(1), 52–81 (2009). http://portal.acm.org/citation.scfm?id=1498013.

    Article  MathSciNet  Google Scholar 

  22. G. Lerman, J.T. Whitehouse, High-dimensional Menger-type curvatures—part I: Geometric multipoles and multiscale inequalities (2008, submitted). Available from http://arxiv.org/abs/0805.1425v1.

  23. G. Lerman, J.T. Whitehouse, High-dimensional Menger-type curvatures—part II: d-separation and a menagerie of curvatures. Constr. Approx. (2009, accepted). Available from http://arxiv.org/abs/0809.0137v1.

  24. G. Lerman, J.T. Whitehouse, Least squares for probability measures via multi-way curvatures (2009, in preparation).

  25. Y. Ma, H. Derksen, W. Hong, J. Wright, Segmentation of multivariate mixed data via lossy coding and compression, IEEE Trans. Pattern Anal. Mach. Intell. 29(9), 1546–1562 (2007).

    Article  Google Scholar 

  26. Y. Ma, A.Y. Yang, H. Derksen, R. Fossum, Estimation of subspace arrangements with applications in modeling and segmenting mixed data, SIAM Rev. 50(3), 413–458 (2008).

    Article  MATH  MathSciNet  Google Scholar 

  27. J. MacQueen, Some methods for classification and analysis of multivariate observations, in Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, vol. 1 (University of California Press, Berkeley, 1967), pp. 281–297.

    Google Scholar 

  28. J. Mairal, F. Bach, J. Ponce, G. Sapiro, A. Zisserman, Discriminative learned dictionaries for local image analysis, in Proc. CVPR, Alaska, June 2008.

  29. C. McDiarmid, On the method of bounded differences, in Surveys in Combinatorics (Cambridge University Press, Cambridge, 1989), pp. 148–188.

    Google Scholar 

  30. G. Medioni, M.-S. Lee, C.-K. Tang, A Computational Framework for Segmentation and Grouping (Elsevier, Amsterdam, 2000).

    MATH  Google Scholar 

  31. A. Ng, M. Jordan, Y. Weiss, On spectral clustering: Analysis and an algorithm, in Advances in Neural Information Processing Systems 14, (2001), pp. 849–856.

  32. A. Shashua, R. Zass, T. Hazan, Multi-way clustering using super-symmetric non-negative tensor factorization, in ECCV06, vol. IV (2006), pp. 595–608.

  33. J. Shi, J. Malik, Normalized cuts and image segmentation, IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000).

    Article  Google Scholar 

  34. R. Souvenir, R. Pless, Manifold clustering, in The 10th International Conference on Computer Vision (ICCV 2005), 2005.

  35. A. Szlam, Modifications of k q-flats for supervised learning (2008).

  36. M. Tipping, C. Bishop, Mixtures of probabilistic principal component analysers, Neural Comput. 11(2), 443–482 (1999).

    Article  Google Scholar 

  37. P.H.S. Torr, Geometric motion segmentation and model selection, Philos. Trans. R. Soc. Lond. A 356, 1321–1340 (1998).

    Article  MATH  MathSciNet  Google Scholar 

  38. P. Tseng, Nearest q-flat to m points, J. Optim. Theory Appl. 105(1), 249–252 (2000).

    Article  MATH  MathSciNet  Google Scholar 

  39. R. Vidal, Y. Ma, S. Sastry, Generalized principal component analysis (GPCA), IEEE Trans. Pattern Anal. Mach. Intell. 27(12) (2005).

  40. U. von Luxburg, M. Belkin, O. Bousquet, Consistency of spectral clustering, Ann. Stat. 36(2), 555–586 (2008).

    Article  MATH  Google Scholar 

  41. J. Yan, M. Pollefeys, A general framework for motion segmentation: Independent, articulated, rigid, non-rigid, degenerate and nondegenerate, in ECCV, vol. 4 (2006), pp. 94–106.

  42. A.Y. Yang, S.R. Rao, Y. Ma, Robust statistical estimation and segmentation of multiple subspaces, in Computer Vision and Pattern Recognition Workshop, June 2006.

  43. L. Zwald, G. Blanchard, On the convergence of eigenspaces in kernel principal components analysis, in Advances in Neural Information Processing Systems 18 (2005), pp. 1649–1656.

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, University of Minnesota, 127 Vincent Hall, 206 Church Street SE, Minneapolis, MN, 55455, USA

    Guangliang Chen & Gilad Lerman

Authors
  1. Guangliang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gilad Lerman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gilad Lerman.

Additional information

Communicated by Albert Cohen.

This work was supported by NSF grant #0612608.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chen, G., Lerman, G. Foundations of a Multi-way Spectral Clustering Framework for Hybrid Linear Modeling. Found Comput Math 9, 517–558 (2009). https://doi.org/10.1007/s10208-009-9043-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10208-009-9043-7

Keywords

Mathematics Subject Classification (2000)

Search

Navigation