Journal of Scientific Computing

, Volume 72, Issue 3, pp 1313–1332 | Cite as

A Three-Stage Approach for Segmenting Degraded Color Images: Smoothing, Lifting and Thresholding (SLaT)

  • Xiaohao Cai
  • Raymond Chan
  • Mila Nikolova
  • Tieyong ZengEmail author


In this paper, we propose a Smoothing, Lifting and Thresholding (SLaT) method with three stages for multiphase segmentation of color images corrupted by different degradations: noise, information loss and blur. At the first stage, a convex variant of the Mumford–Shah model is applied to each channel to obtain a smooth image. We show that the model has unique solution under different degradations. In order to properly handle the color information, the second stage is dimension lifting where we consider a new vector-valued image composed of the restored image and its transform in a secondary color space to provide additional information. This ensures that even if the first color space has highly correlated channels, we can still have enough information to give good segmentation results. In the last stage, we apply multichannel thresholding to the combined vector-valued image to find the segmentation. The number of phases is only required in the last stage, so users can modify it without the need of solving the previous stages again. Experiments demonstrate that our SLaT method gives excellent results in terms of segmentation quality and CPU time in comparison with other state-of-the-art segmentation methods.


Mumford–Shah model Convex variational models Multiphase color image segmentation Color spaces 



The authors thank G. Steidl and M. Bertalmío for constructive discussions. The work of X. Cai is partially supported by Welcome Trust, Issac Newton Trust, and KAUST Award No. KUK-I1-007-43. The work of R. Chan is partially supported by HKRGC GRF Grant No. CUHK300614, CUHK14306316, CRF Grant No. CUHK2/CRF/11G, CRF Grant C1007-15G, and AoE Grant AoE/M-05/12 The work of M. Nikolova is partially supported by HKRGC GRF Grant No. CUHK300614, and by the French Research Agency (ANR) under Grant No. ANR-14-CE27-001 (MIRIAM). The work of T. Zeng is partially supported by NSFC 11271049, 11671002, RGC 211911, 12302714 and RFGs of HKBU.


  1. 1.
    Bar, L., Chan, T.F., Chung, G., Jung, M., Kiryati, N., Mohieddine, R., Sochen, N., Vese, L.A.: Mumford and Shah model and its applications to image segmentation and image restoration. In: Scherzer, O. (ed.) Handbook of Mathematical Methods in Imaging, pp. 1539–1598. Springer, Berlin (2015)CrossRefGoogle Scholar
  2. 2.
    Benninghoff, H., Garcke, H.: Efficient image segmentation and restoration using parametric curve evolution with junctions and topology changes. SIAM J. Imaging Sci. 7(3), 1451–1483 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)zbMATHCrossRefGoogle Scholar
  4. 4.
    Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J.P., Osher, S.: Fast global minimization of the active contour/snake model. J. Math. Imaging Vis. 28(2), 151–167 (2007)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Cai, X.: Variational image segmentation model coupled with image restoration achievements. Pattern Recognit. 48(6), 2029–2042 (2015)CrossRefGoogle Scholar
  6. 6.
    Cai, X., Chan, R., Zeng, T.: A two-stage image segmentation method using a convex variant of the Mumford–Shah model and thresholding. SIAM J. Imaging Sci. 6(1), 368–390 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Cai, X., Steidl, G.: Multiclass segmentation by iterated ROF thresholding. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.C. (eds.) Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 237–250. Springer, Berlin (2013)CrossRefGoogle Scholar
  8. 8.
    Cardelino, J., Caselles, V., Bertalmio, M., Randall, G.: A contrario selection of optimal partitions for image segmentation. SIAM J. Imaging Sci. 6(3), 1274–1317 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Chambolle, A., Cremers, D., Pock, T.: A convex approach to minimal partitions. SIAM J. Imaging Sci. 5(4), 1113–1158 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Chan, R., Yang, H., Zeng, T.: A two-stage image segmentation method for blurry images with poisson or multiplicative gamma noise. SIAM J. Imaging Sci. 7(1), 98–127 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Chan, T.F., Esedoglu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math. 66(5), 1632–1648 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Chan, T.F., Sandberg, B.Y., Vese, L.A.: Active contours without edges for vector-valued images. J. Vis. Commun. Image Represent. 11(2), 130–141 (2000)CrossRefGoogle Scholar
  14. 14.
    Chan, T.F., Vese, L., et al.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)zbMATHCrossRefGoogle Scholar
  15. 15.
    Chen, Y., Lan, G., Ouyang, Y.: Optimal primal-dual methods for a class of saddle point problems. SIAM J. Optim. 24(4), 1779–1814 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Cremers, D., Rousson, M., Deriche, R.: A review of statistical approaches to level set segmentation: integrating color, texture, motion and shape. Int. J. Comput. Vis. 72(2), 195–215 (2007)CrossRefGoogle Scholar
  17. 17.
    Ekeland, I., Temam, R.: Convex analysis and variational problems. SIAM Classics in Applied Mathematics, Philadelphia (1976)zbMATHGoogle Scholar
  18. 18.
    Evans, L.C.: Partial differential equations and Monge-Kantorovich mass transfer. Curr. Dev. Math. 1997(1), 65–126 (1997)zbMATHCrossRefGoogle Scholar
  19. 19.
    Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell. 6, 721–741 (1984)zbMATHCrossRefGoogle Scholar
  20. 20.
    Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Grady, L.: Random walks for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 28(11), 1768–1783 (2006)CrossRefGoogle Scholar
  22. 22.
    Grady, L., Alvino, C.: Reformulating and optimizing the Mumford–Shah functional on a graph—faster, lower energy solution. In: ECCV 2008, pp. 248–261. Springer, Berlin (2008)Google Scholar
  23. 23.
    Jung, Y.M., Kang, S.H., Shen, J.: Multiphase image segmentation via Modica–Mortola phase transition. SIAM J. Appl. Math. 67(5), 1213–1232 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Kanungo, T., Mount, D.M., Netanyahu, N.S., Piatko, C.D., Silverman, R., Wu, A.Y.: An efficient k-means clustering algorithm: analysis and implementation. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 881–892 (2002)zbMATHCrossRefGoogle Scholar
  25. 25.
    Kay, D., Tomasi, A., et al.: Color image segmentation by the vector-valued Allen–Cahn phase-field model: a multigrid solution. IEEE Trans. Image Process. 18(10), 2330–2339 (2009)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Levinshtein, A., Stere, A., Kutulakos, K.N., Fleet, D.J., Dickinson, S.J., Siddiqi, K.: Turbopixels: fast superpixels using geometric flows. IEEE Trans. Pattern Anal. Mach. Intell. 31(12), 2290–2297 (2009)CrossRefGoogle Scholar
  27. 27.
    Li, C., Huang, R., Ding, Z., Gatenby, J.C., Metaxas, D.N., C, G.J.: A level set method for image segmentation in the presence of intensity inhomogeneity with application to MRI. IEEE Trans. Image Process. 20(7), 2007–2016 (2011)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Li, F., Ng, M.K., Zeng, T.Y., Shen, C.: A multiphase image segmentation method based on fuzzy region competition. SIAM J. Imaging Sci. 3(3), 277–299 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    Lukac, R., Plataniotis, K.N.: Color Image Processing: Methods and Applications. CRC Press, Boca Raton (2007)Google Scholar
  30. 30.
    Luong, Q.T.: Color in computer vision. In: Chen, C.H., Pau, L.F., Wang, P.S.P. (eds.) Handbook of Pattern Recognition & Computer Vision, pp. 311–368. World Scientific Publishing Co., Inc., River Edge, NJ, USA (1993)CrossRefGoogle Scholar
  31. 31.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. ICCV 2, 416–423 (2001)Google Scholar
  32. 32.
    Mumford, D., Shah, J.: Boundary detection by minimizing functionals. In: Ullman, S., Richards, W. (eds.) Image Understanding 1989. Ablex Publishing Corporation, New Jersey (1990)Google Scholar
  33. 33.
    Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. 42(5), 577–685 (1989)MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    Paschos, G.: Perceptually uniform color spaces for color texture analysis: an empirical evaluation. IEEE Trans. Image Process. 10(6), 932–937 (2001)zbMATHCrossRefGoogle Scholar
  35. 35.
    Plaza, A., Benediktsson, J.A., Boardman, J.W., Brazile, J., Bruzzone, L., Camps-Valls, G., Chanussot, J., Fauvel, M., Gamba, P., Gualtieri, A., et al.: Recent advances in techniques for hyperspectral image processing. Remote Sens. Environ. 113, S110–S122 (2009)CrossRefGoogle Scholar
  36. 36.
    Pock, T., Chambolle, A., Cremers, D., Bischof, H.: A convex relaxation approach for computing minimal partitions. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 810–817 (2009)Google Scholar
  37. 37.
    Potts, R.B.: Some generalized order-disorder transformations. In: Mathematical Proceedings of the Cambridge Philosophical Society, vol. 48, pp. 106–109. Cambridge University Press, Cambridge (1952)Google Scholar
  38. 38.
    Rotaru, C., Graf, T., Zhang, J.: Color image segmentation in HSI space for automotive applications. J. Real-Time Image Process. 3(4), 311–322 (2008)CrossRefGoogle Scholar
  39. 39.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60(1), 259–268 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000)CrossRefGoogle Scholar
  41. 41.
    Storath, M., Weinmann, A.: Fast partitioning of vector-valued images. SIAM J. Imaging Sci. 7(3), 1826–1852 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    Tai, C., Zhang, X., Shen, Z.: Wavelet frame based multiphase image segmentation. SIAM J. Imaging Sci. 6(4), 2521–2546 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  43. 43.
    Tai, Y.W., Jia, J., Tang, C.K.: Soft color segmentation and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 29(9), 1520–1537 (2007)CrossRefGoogle Scholar
  44. 44.
    Townsend, D.: Multimodality imaging of structure and function. Phys. Med. Biol. 53(4), R1 (2008)MathSciNetCrossRefGoogle Scholar
  45. 45.
    Vandenbroucke, N., Macaire, L., Postaire, J.: Color image segmentation by pixel classification in an adapted hybrid color space. Application to soccer image analysis. Comput. Vis. Image Underst. 90(2), 190–216 (2003)CrossRefGoogle Scholar
  46. 46.
    Vese, L.A., Chan, T.F.: A multiphase level set framework for image segmentation using the Mumford and Shah model. Int. J. Comput. Vis. 50(3), 271–293 (2002)zbMATHCrossRefGoogle Scholar
  47. 47.
    Wang, X., Tang, Y., Masnou, S., Chen, L.: A global/local affinity graph for image segmentation. IEEE Trans. Image Process. 24(4), 1399–1411 (2015)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Department of Applied Mathematics and Theoretical Physics (DAMTP)University of CambridgeCambridgeUK
  2. 2.Mullard Space Science Laboratory (MSSL)University College LondonDorkingUK
  3. 3.Department of MathematicsThe Chinese University of Hong KongShatinHong Kong
  4. 4.CMLA, ENS Cachan, CNRSUniversité Paris-SaclayCachanFrance
  5. 5.Department of MathematicsHong Kong Baptist UniversityKowloon TongHong Kong

Personalised recommendations