Normalized Cut Meets MRF

  • Meng Tang
  • Dmitrii MarinEmail author
  • Ismail Ben Ayed
  • Yuri Boykov
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9906)


We propose a new segmentation or clustering model that combines Markov Random Field (MRF) and Normalized Cut (NC) objectives. Both NC and MRF models are widely used in machine learning and computer vision, but they were not combined before due to significant differences in the corresponding optimization, e.g. spectral relaxation and combinatorial max-flow techniques. On the one hand, we show that many common applications for multi-label MRF segmentation energies can benefit from a high-order NC term, e.g. enforcing balanced clustering of arbitrary high-dimensional image features combining color, texture, location, depth, motion, etc. On the other hand, standard NC applications benefit from an inclusion of common pairwise or higher-order MRF constraints, e.g. edge alignment, bin-consistency, label cost, etc. To address NC+MRF energy, we propose two efficient multi-label combinatorial optimization techniques, spectral cut and kernel cut, using new unary bounds for different NC formulations.


Markov Random Fields Spectral Cluster Normalize Mutual Information Swap Move Label Cost 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000)CrossRefGoogle Scholar
  2. 2.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)CrossRefGoogle Scholar
  3. 3.
    Kohli, P., Ladicky, L., Torr, P.H.S.: Robust higher order potentials for enforcing label consistency. Int. J. Comput. Vis. (IJCV) 82(3), 302–324 (2009)CrossRefGoogle Scholar
  4. 4.
    Delong, A., Osokin, A., Isack, H., Boykov, Y.: Fast approximate energy minization with label costs. Int. J. Comput. Vis. (IJCV) 96(1), 1–27 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images. In: ICCV, vol. I, pp. 105–112 (2001)Google Scholar
  6. 6.
    Tang, M., Gorelick, L., Veksler, O., Boykov, Y.: Grabcut in one cut. In: International Conference on Computer Vision (ICCV), Sydney, Australia, December 2013Google Scholar
  7. 7.
    Malik, J., Belongie, S., Leung, T., Shi, J.: Contour and texture analysis for image segmentation. Int. J. Comput. Vis. 43(1), 7–27 (2001)CrossRefzbMATHGoogle Scholar
  8. 8.
    Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)CrossRefGoogle Scholar
  9. 9.
    Chan, T., Vese, L.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)CrossRefzbMATHGoogle Scholar
  10. 10.
    Zhu, S.C., Yuille, A.: Region competition: unifying snakes, region growing, and Bayes/MDL for multiband image segmentation. IEEE Trans. PAMI 18(9), 884–900 (1996)CrossRefGoogle Scholar
  11. 11.
    Rother, C., Kolmogorov, V., Blake, A.: Grabcut - interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. (SIGGRAPH) 23(3), 309–314 (2004)CrossRefGoogle Scholar
  12. 12.
    Kearns, M., Mansour, Y., Ng, A.: An information-theoretic analysis of hard and soft assignment methods for clustering. In: Thirteenth Conference on Uncertainty in Artificial Intelligence (UAI), August 1997Google Scholar
  13. 13.
    Tang, M., Ayed, I.B., Marin, D., Boykov, Y.: Secrets of grabcut and kernel k-means. In: International Conference on Computer Vision (ICCV), Santiago, Chile, December 2015Google Scholar
  14. 14.
    Yu, S.X., Shi, J.: Segmentation given partial grouping constraints. IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 173–183 (2004)CrossRefGoogle Scholar
  15. 15.
    Eriksson, A., Olsson, C., Kahl, F.: Normalized cuts revisited: a reformulation for segmentation with linear grouping constraints. J. Math. Imaging Vis. 39(1), 45–61 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Chew, S.E., Cahill, N.D.: Semi-supervised normalized cuts for image segmentation. In: IEEE International Conference on Computer Vision (ICCV), December 2015Google Scholar
  17. 17.
    Ayed, I.B., Gorelick, L., Boykov, Y.: Auxiliary cuts for general classes of higher order functionals. In: CVPR, pp. 1304–1311 (2013)Google Scholar
  18. 18.
    Tang, M., Ben Ayed, I., Boykov, Y.: Pseudo-bound optimization for binary energies. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part V. LNCS, vol. 8693, pp. 691–707. Springer, Heidelberg (2014)Google Scholar
  19. 19.
    Von Luxburg, U.: A tutorial on spectral clustering. Stat. Comput. 17(4), 395–416 (2007)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Tang, M., Ayed, I.B., Marin, D., Boykov, Y.: Kernel Cuts: MRF meets kernel & spectral clustering. In: (July 2016, also submitted to PAMI). arXiv:1506.07439
  21. 21.
    Bach, F., Jordan, M.: Learning spectral clustering. Adv. Neural Inf. Process. Syst. (NIPS) 16, 305–312 (2003)Google Scholar
  22. 22.
    Dhillon, I., Guan, Y., Kulis, B.: Kernel k-means, spectral clustering and normalized cuts. In: KDD (2004)Google Scholar
  23. 23.
    Roth, V., Laub, J., Kawanabe, M., Buhmann, J.: Optimal cluster preserving embedding of nonmetric proximity data. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 25(12), 1540–1551 (2003)CrossRefGoogle Scholar
  24. 24.
    Dhillon, I., Guan, Y., Kulis, B.: Weighted graph cuts without eigenvectors: a multilevel approach. IEEE Trans. Pattern Anal. Mach. Learn. (PAMI) 29(11), 1944–1957 (2007)CrossRefGoogle Scholar
  25. 25.
    Belongie, S., Malik, J.: Finding boundaries in natural images: a new method using point descriptors and area completion. In: Burkhardt, H.-J., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1406, pp. 751–766. Springer, Heidelberg (1998)Google Scholar
  26. 26.
    Cox, T.F., Cox, M.A.: Multidimensional Scaling. CRC Press, Boca Raton (2000)zbMATHGoogle Scholar
  27. 27.
    Chitta, R., Jin, R., Havens, T.C., Jain, A.K.: Approximate kernel k-means: solution to large scale kernel clustering. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 895–903. ACM (2011)Google Scholar
  28. 28.
    Fowlkes, C., Belongie, S., Chung, F., Malik, J.: Spectral grouping using the nystrom method. IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 214–225 (2004)CrossRefGoogle Scholar
  29. 29.
    Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. IEEE Trans. Pattern Anal. Mach. Intell. 28(10), 1568–1583 (2006)CrossRefGoogle Scholar
  30. 30.
    Werner, T.: A linear programming approach to max-sum problem: a review. IEEE Trans. Pattern Anal. Mach. Intell. 29(7), 1165–1179 (2007)CrossRefGoogle Scholar
  31. 31.
    Kappes, J.H., Andres, B., Hamprecht, F.A., Schnörr, C., Nowozin, S., Batra, D., Kim, S., Kausler, B.X., Kröger, T., Lellmann, J., et al.: A comparative study of modern inference techniques for structured discrete energy minimization problems. Int. J. Comput. Vis. 115(2), 155–184 (2015)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20(1–2), 89–97 (2004)MathSciNetGoogle Scholar
  33. 33.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Cremers, D., Rousson, M., Deriche, R.: A review of statistical approaches to level set segmentation: integrating color, texture, motion and shape. Int. J. Comput. Vis. 72(2), 195–215 (2007)CrossRefGoogle Scholar
  35. 35.
    Park, K., Gould, S.: On learning higher-order consistency potentials for multi-class pixel labeling. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part II. LNCS, vol. 7573, pp. 202–215. Springer, Heidelberg (2012)Google Scholar
  36. 36.
    Zelnik-Manor, L., Perona, P.: Self-tuning spectral clustering. In: Advances in Neural Information Processing Systems, pp. 1601–1608 (2004)Google Scholar
  37. 37.
    Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)CrossRefGoogle Scholar
  38. 38.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42(3), 145–175 (2001)CrossRefzbMATHGoogle Scholar
  39. 39.
    Collins, M.D., Liu, J., Xu, J., Mukherjee, L., Singh, V.: Spectral clustering with a convex regularizer on millions of images. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part III. LNCS, vol. 8691, pp. 282–298. Springer, Heidelberg (2014)Google Scholar
  40. 40.
    Nieuwenhuis, C., Cremers, D.: Spatially varying color distributions for interactive multilabel segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1234–1247 (2013)CrossRefGoogle Scholar
  41. 41.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of Eighth IEEE International Conference on Computer Vision, ICCV 2001, vol. 2, pp. 416–423. IEEE (2001)Google Scholar
  42. 42.
    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012)Google Scholar
  43. 43.
    Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  44. 44.
    Dou, M., Taylor, J., Fuchs, H., Fitzgibbon, A., Izadi, S.: 3D scanning deformable objects with a single RGBD sensor. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 493–501 (2015)Google Scholar
  45. 45.
    Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: real-time dense surface mapping and tracking. In: 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 127–136. IEEE (2011)Google Scholar
  46. 46.
    Deng, Z., Todorovic, S., Latecki, L.J.: Semantic segmentation of RGBD images with mutex constraints. In: International Conference on Computer Vision (ICCV), Santiago, Chile, December 2015Google Scholar
  47. 47.
    Gulshan, V., Lempitsky, V., Zisserman, A.: Humanising grabcut: learning to segment humans using the kinect. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1127–1133. IEEE (2011)Google Scholar
  48. 48.
    Ren, X., Bo, L., Fox, D.: Rgb-(d) scene labeling: features and algorithms. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2759–2766. IEEE (2012)Google Scholar
  49. 49.
    Gottfried, J.M., Fehr, J., Garbe, C.S.: Computing range flow from multi-modal kinect data. In: Bebis, G., et al. (eds.) ISVC 2011. LNCS, vol. 6938, pp. 758–767. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  50. 50.
    Brox, T., Malik, J.: Object segmentation by long term analysis of point trajectories. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 282–295. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  51. 51.
    Ochs, P., Brox, T.: Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1583–1590. IEEE (2011)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Meng Tang
    • 1
  • Dmitrii Marin
    • 1
    Email author
  • Ismail Ben Ayed
    • 2
  • Yuri Boykov
    • 1
  1. 1.Computer ScienceUniversity of Western OntarioLondonCanada
  2. 2.Ecole de Technologie SupérieureUniversity of QuebecMontrealCanada

Personalised recommendations