Advertisement

Computational Optimization and Applications

, Volume 70, Issue 2, pp 395–418 | Cite as

A nonconvex formulation for low rank subspace clustering: algorithms and convergence analysis

  • Hao Jiang
  • Daniel P. Robinson
  • René Vidal
  • Chong You
Article
  • 252 Downloads

Abstract

We consider the problem of subspace clustering with data that is potentially corrupted by both dense noise and sparse gross errors. In particular, we study a recently proposed low rank subspace clustering approach based on a nonconvex modeling formulation. This formulation includes a nonconvex spectral function in the objective function that makes the optimization task challenging, e.g., it is unknown whether the alternating direction method of multipliers (ADMM) framework proposed to solve the nonconvex model formulation is provably convergent. In this paper, we establish that the spectral function is differentiable and give a formula for computing the derivative. Moreover, we show that the derivative of the spectral function is Lipschitz continuous and provide an explicit value for the Lipschitz constant. These facts are then used to provide a lower bound for how the penalty parameter in the ADMM method should be chosen. As long as the penalty parameter is chosen according to this bound, we show that the ADMM algorithm computes iterates that have a limit point satisfying first-order optimality conditions. We also present a second strategy for solving the nonconvex problem that is based on proximal gradient calculations. The convergence and performance of the algorithms is verified through experiments on real data from face and digit clustering and motion segmentation.

Keywords

ADMM Nonconvex Subspace clustering 

Notes

Acknowledgements

The authors thank the financial support of NSF Grants 1447822, 1618637, and 1704458.

References

  1. 1.
    Andersson, F., Carlsson, M., Perfekt, K.M.: Operator-Lipschitz estimates for the singular value functional calculus. In: Proceedings of the American Mathematical Society (2015)Google Scholar
  2. 2.
    Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137(1–2), 91–129 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Basri, R., Jacobs, D.W.: Lambertian reflectance and linear subspaces. IEEE Trans. Pattern Anal. Mac. Intel. 25(2), 218–233 (2003)CrossRefGoogle Scholar
  4. 4.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends\({\textregistered }\). Mach. Learn. 3(1), 1–122 (2011)zbMATHGoogle Scholar
  6. 6.
    Bredies, K., Lorenz, D.A., Reiterer, S.: Minimization of non-smooth, non-convex functionals by iterative thresholding. J. Optim. Theory Appl. 165(1), 78–112 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Bruna, J., Mallat, S.: Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1872–1886 (2013)CrossRefGoogle Scholar
  8. 8.
    Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58(3), 11 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Chartrand, R., Wohlberg, B.: A nonconvex ADMM algorithm for group sparsity with sparse groups. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6009–6013. IEEE (2013)Google Scholar
  10. 10.
    Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of admm for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155(1–2), 57–79 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Conn, A.R., Gould, N.I.M., Toint, P.L.: Trust-Region Methods. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA (2000)Google Scholar
  12. 12.
    Eckstein, J., Yao, W.: Understanding the convergence of the alternating direction method of multipliers: Theoretical and computational perspectives. Technical Report Center for Operations Research, Rutgers University, 640 Bartholomew Road, Piscataway, New Jersey (2015)Google Scholar
  13. 13.
    Elhamifar, E., Vidal, R.: Clustering disjoint subspaces via sparse representation. In: IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 1926–1929. IEEE (2010)Google Scholar
  14. 14.
    Elhamifar, E., Vidal, R.: Sparse subspace clustering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790–2797. IEEE (2009)Google Scholar
  15. 15.
    Elhamifar, E., Vidal, R.: Sparse subspace clustering: algorithm, theory, and applications. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2765–2781 (2013)CrossRefGoogle Scholar
  16. 16.
    Fortin, M., Glowinksi, R.: On decomposition-coordination methods using an augmented Lagrangian. In: Fortin, M., Glowinksi, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary-Value Problems. Chapter 3, pp. 97–144. Elsevier, Amsterdam (1983)CrossRefGoogle Scholar
  17. 17.
    Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinksi, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary-Value Problems, pp. 299–331. Elsevier, Amsterdam (1983). Chapter 9CrossRefGoogle Scholar
  18. 18.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2(1), 17–40 (1976)CrossRefzbMATHGoogle Scholar
  19. 19.
    Glowinski, R., Marroco, A.: Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de dirichlet non linéaires. ESAIM: Mathematical Modelling and Numerical Analysis - Modélisation Mathématique et Analyse Numérique 9(R2), 41–76 (1975). http://eudml.org/doc/193269
  20. 20.
    Ho, J., Yang, M.H., Lim, J., Lee, K.C., Kriegman, D.: Clustering appearances of objects under varying illumination conditions. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 11–18. IEEE (2003)Google Scholar
  21. 21.
    Hong, W., Wright, J., Huang, K., Ma, Y.: Multiscale hybrid linear models for lossy image representation. IEEE Trans. Image Process. 15(12), 3655–3671 (2006)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Hong, M., Luo, Z.Q., Razaviyayn, M.: Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. SIAM J. Optim. 26(1), 337–364 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Jiang, H.: Augmented Lagrangian based algorithms for nonconvex optimization with applications in subspace clustering. Ph.D. thesis, Johns Hopkins University (2016). https://jscholarship.library.jhu.edu/handle/1774.2/838
  24. 24.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  25. 25.
    Lee, K.C., Ho, J., Kriegman, D.J.: Acquiring linear subspaces for face recognition under variable lighting. IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 684–698 (2005)CrossRefGoogle Scholar
  26. 26.
    Lewis, A.S., Sendov, H.S.: Nonsmooth analysis of singular values. part I: Theory. Set-Valued Anal. 13(3), 213–241 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979).  https://doi.org/10.1137/0716071 MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Liu, R., Lin, Z., De la Torre, F., Su, Z.: Fixed-rank representation for unsupervised visual learning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 598–605. IEEE (2012)Google Scholar
  29. 29.
    Liu, G., Lin, Z., Yu, Y.: Robust subspace segmentation by low-rank representation. In: International Conference on Machine Learning, pp. 663–670 (2010)Google Scholar
  30. 30.
    Liu, G., Yan, S.: Latent low-rank representation for subspace segmentation and feature extraction. In: 2011 IEEE International Conference on Computer Vision, pp. 1615–1622. IEEE (2011)Google Scholar
  31. 31.
    Liu, G., Lin, Z., Yan, S., Sun, J., Yu, Y., Ma, Y.: Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 171–184 (2013)CrossRefGoogle Scholar
  32. 32.
    Lu, C., Lin, Z., Yan, S.: Correlation adaptive subspace segmentation by trace lasso. In: IEEE International Conference on Computer Vision, pp. 1345–1352 (2013)Google Scholar
  33. 33.
    Lu, C.Y., Min, H., Zhao, Z.Q., Zhu, L., Huang, D.S., Yan, S.: Robust and efficient subspace segmentation via least squares regression. In: European Conference on Computer Vision, pp. 347–360 (2012)Google Scholar
  34. 34.
    Open source computer vision library (OpenCV). http://sourceforge.net/projects/opencvlibrary
  35. 35.
    Patel, V.M., Nguyen, H.V., Vidal, R.: Latent space sparse subspace clustering. In: International Conference on Computer Vision (2013)Google Scholar
  36. 36.
    Patel, V.M., Nguyen, H., Vidal, R.: Latent space sparse and low-rank subspace clustering. IEEE J. Sel. Top. Signal Process. 9(4), 691–701 (2015)CrossRefGoogle Scholar
  37. 37.
    Soltanolkotabi, M., Candes, E.J.: A geometric analysis of subspace clustering with outliers. The Annals of Statistics, pp. 2195–2238 (2012)Google Scholar
  38. 38.
    Soltanolkotabi, M., Elhamifar, E., Candes, E.: Robust subspace clustering. Ann. Stat. 42(2), 669–699 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Toh, K.C., Yun, S.: An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pac. J. Optim. 6(615–640), 15 (2010)MathSciNetzbMATHGoogle Scholar
  40. 40.
    Tron, R., Vidal, R.: A benchmark for the comparsion of 3-D motion segmentation algorithms. In: IEEE Conference on Computer Vision and Pattern Recognition (2007)Google Scholar
  41. 41.
    Vidal, R., Ma, Y., Sastry, S.: Generalized Principal Component Analysis. Springer Interdisciplinary Applied Mathematics (2016)Google Scholar
  42. 42.
    Vidal, R.: Subspace clustering. IEEE Signal Process. Mag. 28(2), 52–68 (2010)CrossRefGoogle Scholar
  43. 43.
    Vidal, R., Favaro, P.: Low rank subspace clustering (LRSC). Pattern Recognit. Lett. 43, 47–61 (2014)CrossRefGoogle Scholar
  44. 44.
    Vidal, R., Tron, R., Hartley, R.: Multiframe motion segmentation with missing data using PowerFactorization and GPCA. Int. J. Comput. Vis. 79(1), 85–105 (2008)CrossRefGoogle Scholar
  45. 45.
    Von Luxburg, U.: A tutorial on spectral clustering. Stat. Comput. 17(4), 395–416 (2007)MathSciNetCrossRefGoogle Scholar
  46. 46.
    Wang, F., Cao, W., Xu, Z.: Convergence of multi-block Bregman ADMM for nonconvex composite problems (2015). arXiv preprint arXiv:1505.03063
  47. 47.
    Wang, Y.X., Xu, H., Leng, C.: Provable subspace clustering: When LRR meets SSC. In: Neural Information Processing Systems (2013)Google Scholar
  48. 48.
    Wang, Y., Yin, W., Zeng, J.: Global convergence of ADMM in nonconvex nonsmooth optimization (2015). arXiv preprint arXiv:1511.06324
  49. 49.
    Yin, M., Gao, J., Lin, Z.: Laplacian regularized low-rank representation and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 38(3), 504–517 (2016)CrossRefGoogle Scholar
  50. 50.
    You, C., Li, C.G., Robinson, D., Vidal, R.: Oracle based active set algorithm for scalable elastic net subspace clustering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3928–3937 (2016)Google Scholar
  51. 51.
    You, C., Robinson, D., Vidal, R.: Scalable sparse subspace clustering by orthogonal matching pursuit. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3918–3927 (2016)Google Scholar
  52. 52.
    You, C., Vidal, R.: Geometric conditions for subspace-sparse recovery. In: Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1585–1593 (2015)Google Scholar
  53. 53.
    Zhang, H., Lin, Z., Zhang, C., Gao, J.: Relations among some low-rank subspace recovery models. Neural Comput. 27(9), 1915–1950 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Johns Hopkins UniversityBaltimoreUSA

Personalised recommendations