Advertisement

Journal of Mathematical Imaging and Vision

, Volume 49, Issue 1, pp 191–201 | Cite as

Multi-class Transductive Learning Based on 1 Relaxations of Cheeger Cut and Mumford-Shah-Potts Model

  • Xavier BressonEmail author
  • Xue-Cheng Tai
  • Tony F. Chan
  • Arthur Szlam
Article

Abstract

Recent advances in 1 optimization for imaging problems provide promising tools to solve the fundamental high-dimensional data classification in machine learning. In this paper, we extend the main result of Szlam and Bresson (Proceedings of the 27th International Conference on Machine Learning, pp. 1039–1046, 2010), which introduced an exact 1 relaxation of the Cheeger ratio cut problem for unsupervised data classification. The proposed extension deals with the multi-class transductive learning problem, which consists in learning several classes with a set of labels for each class. Learning several classes (i.e. more than two classes) simultaneously is generally a challenging problem, but the proposed method builds on strong results introduced in imaging to overcome the multi-class issue. Besides, the proposed multi-class transductive learning algorithms also benefit from recent fast 1 solvers, specifically designed for the total variation norm, which plays a central role in our approach. Finally, experiments demonstrate that the proposed 1 relaxation algorithms are more accurate and robust than standard 2 relaxation methods s.a. spectral clustering, particularly when considering a very small number of labels for each class to be classified. For instance, the mean error of classification for the benchmark MNIST dataset of 60,000 data in \(\mathbb{R}^{784}\) using the proposed 1 relaxation of the multi-class Cheeger cut is 2.4 % when only one label is considered for each class, while the error of classification for the 2 relaxation method of spectral clustering is 24.7 %.

Keywords

Data analysis Clustering Transductive learning Multi-class Ratio cut Min cut Potts and Mumford-Shah energies Exact relaxation Fast L1 optimization Total variation 

Notes

Acknowledgement

Xavier Bresson is supported by the Hong Kong RGC under Grant GRF110311.

References

  1. 1.
    Bae, E., Yuan, J., Tai, X.-C.: Global minimization for continuous multiphase partitioning problems using a dual approach. Int. J. Comput. Vis. 92(1), 112–129 (2009) CrossRefMathSciNetGoogle Scholar
  2. 2.
    Belkin, M.: Problems of learning on manifolds. PhD thesis, University of Chicago (2003) Google Scholar
  3. 3.
    Bertozzi, A., Flenner, A.: Diffuse interface models on graphs for classification of high dimensional data. UCLA CAM Report 11-27 (2011) Google Scholar
  4. 4.
    Bioucas-Dias, J.M., Figueiredo, M.A.: A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16(12), 2992–3004 (2007) CrossRefMathSciNetGoogle Scholar
  5. 5.
    Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004) CrossRefGoogle Scholar
  6. 6.
    Brown, E.S., Chan, T.F., Bresson, X.: A convex relaxation method for a class of vector-valued minimization problems with applications to Mumford-Shah segmentation. UCLA CAM Report 10-43 (2010) Google Scholar
  7. 7.
    Bühler, T., Hein, M.: Spectral clustering based on the graph p-Laplacian. In: International Conference on Machine Learning, pp. 81–88 (2009) Google Scholar
  8. 8.
    Chambolle, A., Cremers, D., Pock, T.: A convex approach for computing minimal partitions. Technical Report TR-2008-05, Dept. of Computer Science, University of Bonn, Bonn (2008) Google Scholar
  9. 9.
    Cheeger, J.: A lower bound for the smallest eigenvalue of the Laplacian. Problems in Analysis, 195–199 (1970) Google Scholar
  10. 10.
    Chung, F.R.K.: Spectral Graph Theory. CBMS Regional Conference Series in Mathematics, vol. 92 (1997). Published for the Conference Board of the Mathematical Sciences, Washington, DC zbMATHGoogle Scholar
  11. 11.
    Dinkelbach, W.: On nonlinear fractional programming. Manag. Sci. 13, 492–498 (1967) CrossRefzbMATHMathSciNetGoogle Scholar
  12. 12.
    Donoho, D.: De-noising by soft-thresholding. IEEE Trans. Inf. Theory 41(33), 613–627 (1995) CrossRefzbMATHMathSciNetGoogle Scholar
  13. 13.
    Glowinski, R., Le Tallec, P.: Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. SIAM, Philadelphia (1989) CrossRefzbMATHGoogle Scholar
  14. 14.
    Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009) CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Hein, M., Bühler, T.: An inverse power method for nonlinear eigenproblems with applications in 1-spectral clustering and sparse PCA. In: Advances in Neural Information Processing Systems (NIPS), pp. 847–855 (2010) Google Scholar
  16. 16.
    Hein, M., Setzer, S.: Beyond spectral clustering—tight relaxations of balanced graph cuts. In: Advances in Neural Information Processing Systems (NIPS) (2011) Google Scholar
  17. 17.
    Kolev, K., Cremers, D.: Continuous ratio optimization via convex relaxation with applications to multiview 3D reconstruction. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2009) Google Scholar
  18. 18.
    Kolmogorov, V., Boykov, Y., Rother, C.: Applications of parametric maxflow in computer vision. In: International Conference on Computer Vision, pp. 1–8 (2007) Google Scholar
  19. 19.
    Lellmann, J., Kappes, J., Yuan, J., Becker, F., Schnörr, C.: Convex multi-class image labeling by simplex-constrained total variation. In: International Conference on Scale Space and Variational Methods in Computer Vision, pp. 150–162 (2009) CrossRefGoogle Scholar
  20. 20.
    Lellmann, J., Schnörr, C.: Continuous multiclass labeling approaches and algorithms. Univ. of Heidelberg, Tech. Rep. (2010) Google Scholar
  21. 21.
    Michelot, C.: A finite algorithm for finding the projection of a point onto the canonical simplex of rn. J. Optim. Theory Appl. 50(1), 195–200 (1986) CrossRefzbMATHMathSciNetGoogle Scholar
  22. 22.
    Mumford, D., Shah, J.: Optimal approximations of piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. 42, 577–685 (1989) CrossRefzbMATHMathSciNetGoogle Scholar
  23. 23.
    Potts, R.B., Domb, C.: Some generalized order-disorder transformations. Math. Proc. Camb. Philos. Soc. 48, 106–109 (1952) CrossRefzbMATHGoogle Scholar
  24. 24.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000) CrossRefGoogle Scholar
  25. 25.
    Strang, G.: Maximal flow through a domain. Math. Program. 26, 123–143 (1983) CrossRefzbMATHMathSciNetGoogle Scholar
  26. 26.
    Szlam, A., Bresson, X.: Total variation and cheeger cuts. In: Proceedings of the 27th International Conference on Machine Learning, pp. 1039–1046 (2010) Google Scholar
  27. 27.
    Vese, L.A., Chan, T.F.: A multiphase level set framework for image segmentation using the Mumford and Shah model. Int. J. Comput. Vis. 50(3), 271–293 (2002) CrossRefzbMATHGoogle Scholar
  28. 28.
    Zach, C., Gallup, D., Frahm, J.M., Niethammer, M.: Fast global labeling for real-time stereo using multiple plane sweeps. In: Vision, Modeling, and Visualization, pp. 243–252 (2008) Google Scholar
  29. 29.
    Zelnik-Manor, L., Perona, P.: Self-tuning spectral clustering. In: Advances in Neural Information Processing Systems 17 (NIPS 2004) (2004) Google Scholar
  30. 30.
    Zhao, H.K., Chan, T.F., Merriman, B., Osher, S.: A variational level set approach to multiphase motion. J. Comput. Phys. 127, 179–195 (1996) CrossRefzbMATHMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Xavier Bresson
    • 1
    Email author
  • Xue-Cheng Tai
    • 2
  • Tony F. Chan
    • 3
  • Arthur Szlam
    • 4
  1. 1.Department of Computer ScienceCity University of Hong KongKowloon TangHong Kong
  2. 2.Department of MathematicsUniversity of BergenBergenNorway
  3. 3.Department of Mathematics and Computer ScienceHong Kong University of Science and TechnologyKowloonHong Kong
  4. 4.Department of MathematicsThe City College of New YorkNew YorkUSA

Personalised recommendations