Advertisement

\(\ell ^{0}\)-Sparse Subspace Clustering

  • Yingzhen YangEmail author
  • Jiashi Feng
  • Nebojsa Jojic
  • Jianchao Yang
  • Thomas S. Huang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9906)

Abstract

Subspace clustering methods with sparsity prior, such as Sparse Subspace Clustering (SSC) [1], are effective in partitioning the data that lie in a union of subspaces. Most of those methods require certain assumptions, e.g. independence or disjointness, on the subspaces. These assumptions are not guaranteed to hold in practice and they limit the application of existing sparse subspace clustering methods. In this paper, we propose \(\ell ^{0}\)-induced sparse subspace clustering (\(\ell ^{0}\)-SSC). In contrast to the required assumptions, such as independence or disjointness, on subspaces for most existing sparse subspace clustering methods, we prove that subspace-sparse representation, a key element in subspace clustering, can be obtained by \(\ell ^{0}\)-SSC for arbitrary distinct underlying subspaces almost surely under the mild i.i.d. assumption on the data generation. We also present the “no free lunch” theorem that obtaining the subspace representation under our general assumptions can not be much computationally cheaper than solving the corresponding \(\ell ^{0}\) problem of \(\ell ^{0}\)-SSC. We develop a novel approximate algorithm named Approximate \(\ell ^{0}\)-SSC (\(\hbox {A}\ell ^{0}\)-SSC) that employs proximal gradient descent to obtain a sub-optimal solution to the optimization problem of \(\ell ^{0}\)-SSC with theoretical guarantee, and the sub-optimal solution is used to build a sparse similarity matrix for clustering. Extensive experimental results on various data sets demonstrate the superiority of \(\hbox {A}\ell ^{0}\)-SSC compared to other competing clustering methods.

Keywords

Sparse subspace clustering Proximal gradient descent 

Supplementary material

419974_1_En_45_MOESM1_ESM.pdf (153 kb)
Supplementary material 1 (pdf 152 KB)

References

  1. 1.
    Elhamifar, E., Vidal, R.: Sparse subspace clustering: algorithm, theory, and applications. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2765–2781 (2013)CrossRefGoogle Scholar
  2. 2.
    Vidal, R.: Subspace clustering. IEEE Sig. Process. Mag. 28(2), 52–68 (2011)CrossRefGoogle Scholar
  3. 3.
    Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: NIPS, pp. 849–856 (2001)Google Scholar
  4. 4.
    Liu, G., Lin, Z., Yu, Y.: Robust subspace segmentation by low-rank representation. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), 21–24 June 2010, Haifa, Israel, pp. 663–670 (2010)Google Scholar
  5. 5.
    Liu, G., Lin, Z., Yan, S., Sun, J., Yu, Y., Ma, Y.: Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 171–184 (2013)CrossRefGoogle Scholar
  6. 6.
    Park, D., Caramanis, C., Sanghavi, S.: Greedy subspace clustering. In: Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 8–13 December 2014, Montreal, Quebec, Canada, pp. 2753–2761 (2014)Google Scholar
  7. 7.
    Wang, Y.X., Xu, H., Leng, C.: Provable subspace clustering: when LRR meets SSC. In: Burges, C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K. (eds.) Advances in Neural Information Processing Systems 26, pp. 64–72. Curran Associates, Inc. (2013)Google Scholar
  8. 8.
    Soltanolkotabi, M., Cands, E.J.: A geometric analysis of subspace clustering with outliers. Ann. Statist. 40(4), 2195–2238 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Wang, Y., Xu, H.: Noisy sparse subspace clustering. In: Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, pp. 89–97, 16–21 June 2013Google Scholar
  10. 10.
    Yan, S., Wang, H.: Semi-supervised learning by sparse representation. In: SDM, pp. 792–801 (2009)Google Scholar
  11. 11.
    Cheng, B., Yang, J., Yan, S., Fu, Y., Huang, T.S.: Learning with l1-graph for image analysis. IEEE Trans. Image Process. 19(4), 858–866 (2010)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Elhamifar, E., Vidal, R.: Sparse manifold clustering and embedding. In: NIPS, pp. 55–63 (2011)Google Scholar
  13. 13.
    Yang, Y., Wang, Z., Yang, J., Han, J., Huang, T.: Regularized l1-graph for data clustering. In: Proceedings of the British Machine Vision Conference. BMVA Press (2014)Google Scholar
  14. 14.
    Yang, Y., Wang, Z., Yang, J., Wang, J., Chang, S., Huang, T.S.: Data clustering by Laplacian regularized l1-graph. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, 27–31 July 2014, Québec City, Québec, Canada, pp. 3148–3149 (2014)Google Scholar
  15. 15.
    Peng, X., Yi, Z., Tang, H.: Robust subspace clustering via thresholding ridge regression. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 3827–3833 (2015)Google Scholar
  16. 16.
    Jenatton, R., Mairal, J., Bach, F.R., Obozinski, G.R.: Proximal methods for sparse hierarchical dictionary learning. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 487–494 (2010)Google Scholar
  17. 17.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res. 11, 19–60 (2010)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Mairal, J., Bach, F.R., Ponce, J., Sapiro, G., Zisserman, A.: Supervised dictionary learning. In: Advances in Neural Information Processing Systems 21, Proceedings of the Twenty-Second Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, 8–11 December 2008, pp. 1033–1040 (2008)Google Scholar
  19. 19.
    Mancera, L., Portilla, J.: L0-norm-based sparse representation through alternate projections. In: 2006 IEEE International Conference on Image Processing, pp. 2089–2092, October 2006Google Scholar
  20. 20.
    Yang, J., Yu, K., Gong, Y., Huang, T.S.: Linear spatial pyramid matching using sparse coding for image classification. In: CVPR, pp. 1794–1801 (2009)Google Scholar
  21. 21.
    Cheng, H., Liu, Z., Yang, L., Chen, X.: Sparse representation and learning in visual recognition: theory and applications. Sig. Process. 93(6), 1408–1425 (2013)CrossRefGoogle Scholar
  22. 22.
    Zhang, T., Ghanem, B., Liu, S., Xu, C., Ahuja, N.: Low-rank sparse coding for image classification. In: IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, 1–8 December 2013, pp. 281–288 (2013)Google Scholar
  23. 23.
    Soltanolkotabi, M., Elhamifar, E., Cands, E.J.: Robust subspace clustering. Ann. Statist. 2(04), 669–699Google Scholar
  24. 24.
    Wang, Y., Wang, Y.X., Singh, A.: Graph connectivity in noisy sparse subspace clustering. CoRR abs/1504.01046 (2016)Google Scholar
  25. 25.
    Dyer, E.L., Sankaranarayanan, A.C., Baraniuk, R.G.: Greedy feature selection for subspace clustering. J. Mach. Learn. Res. 14, 2487–2517 (2013)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theor. 50(10), 2231–2242 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Hyder, M., Mahata, K.: An approximate l0 norm minimization algorithm for compressed sensing. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2009, pp. 3365–3368, April 2009Google Scholar
  28. 28.
    Zhang, C.H., Zhang, T.: A general theory of concave regularization for high-dimensional sparse estimation problems. Statist. Sci. 27(4), 576–593 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Candes, E., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theor. 51(12), 4203–4215 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Cands, E.J.: The restricted isometry property and its implications for compressed sensing. C.R. Math. 346(910), 589–592 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Zheng, X., Cai, D., He, X., Ma, W.Y., Lin, X.: Locality preserving clustering for image database. In: Proceedings of the 12th Annual ACM International Conference on Multimedia, MULTIMEDIA 2004, pp. 885–891. ACM, New York (2004)Google Scholar
  32. 32.
    Asuncion, A., D.N.: UCI machine learning repository (2007)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Yingzhen Yang
    • 1
    Email author
  • Jiashi Feng
    • 2
  • Nebojsa Jojic
    • 3
  • Jianchao Yang
    • 4
  • Thomas S. Huang
    • 1
  1. 1.Beckman Institute, University of Illinois at Urbana-ChampaignUrbanaUSA
  2. 2.Department of ECENational University of SingaporeSingaporeSingapore
  3. 3.Microsoft ResearchRedmondUSA
  4. 4.SnapchatLos AngelesUSA

Personalised recommendations