A Three-Stage Approach for Segmenting Degraded Color Images: Smoothing, Lifting and Thresholding (SLaT)

Abstract

In this paper, we propose a Smoothing, Lifting and Thresholding (SLaT) method with three stages for multiphase segmentation of color images corrupted by different degradations: noise, information loss and blur. At the first stage, a convex variant of the Mumford–Shah model is applied to each channel to obtain a smooth image. We show that the model has unique solution under different degradations. In order to properly handle the color information, the second stage is dimension lifting where we consider a new vector-valued image composed of the restored image and its transform in a secondary color space to provide additional information. This ensures that even if the first color space has highly correlated channels, we can still have enough information to give good segmentation results. In the last stage, we apply multichannel thresholding to the combined vector-valued image to find the segmentation. The number of phases is only required in the last stage, so users can modify it without the need of solving the previous stages again. Experiments demonstrate that our SLaT method gives excellent results in terms of segmentation quality and CPU time in comparison with other state-of-the-art segmentation methods.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Notes

  1. 1.

    https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/

References

  1. 1.

    Bar, L., Chan, T.F., Chung, G., Jung, M., Kiryati, N., Mohieddine, R., Sochen, N., Vese, L.A.: Mumford and Shah model and its applications to image segmentation and image restoration. In: Scherzer, O. (ed.) Handbook of Mathematical Methods in Imaging, pp. 1539–1598. Springer, Berlin (2015)

    Google Scholar 

  2. 2.

    Benninghoff, H., Garcke, H.: Efficient image segmentation and restoration using parametric curve evolution with junctions and topology changes. SIAM J. Imaging Sci. 7(3), 1451–1483 (2014)

    MathSciNet  MATH  Article  Google Scholar 

  3. 3.

    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)

    MATH  Article  Google Scholar 

  4. 4.

    Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J.P., Osher, S.: Fast global minimization of the active contour/snake model. J. Math. Imaging Vis. 28(2), 151–167 (2007)

    MathSciNet  Article  Google Scholar 

  5. 5.

    Cai, X.: Variational image segmentation model coupled with image restoration achievements. Pattern Recognit. 48(6), 2029–2042 (2015)

    Article  Google Scholar 

  6. 6.

    Cai, X., Chan, R., Zeng, T.: A two-stage image segmentation method using a convex variant of the Mumford–Shah model and thresholding. SIAM J. Imaging Sci. 6(1), 368–390 (2013)

    MathSciNet  MATH  Article  Google Scholar 

  7. 7.

    Cai, X., Steidl, G.: Multiclass segmentation by iterated ROF thresholding. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.C. (eds.) Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 237–250. Springer, Berlin (2013)

    Google Scholar 

  8. 8.

    Cardelino, J., Caselles, V., Bertalmio, M., Randall, G.: A contrario selection of optimal partitions for image segmentation. SIAM J. Imaging Sci. 6(3), 1274–1317 (2013)

    MathSciNet  MATH  Article  Google Scholar 

  9. 9.

    Chambolle, A., Cremers, D., Pock, T.: A convex approach to minimal partitions. SIAM J. Imaging Sci. 5(4), 1113–1158 (2012)

    MathSciNet  MATH  Article  Google Scholar 

  10. 10.

    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)

    MathSciNet  MATH  Article  Google Scholar 

  11. 11.

    Chan, R., Yang, H., Zeng, T.: A two-stage image segmentation method for blurry images with poisson or multiplicative gamma noise. SIAM J. Imaging Sci. 7(1), 98–127 (2014)

    MathSciNet  MATH  Article  Google Scholar 

  12. 12.

    Chan, T.F., Esedoglu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math. 66(5), 1632–1648 (2006)

    MathSciNet  MATH  Article  Google Scholar 

  13. 13.

    Chan, T.F., Sandberg, B.Y., Vese, L.A.: Active contours without edges for vector-valued images. J. Vis. Commun. Image Represent. 11(2), 130–141 (2000)

    Article  Google Scholar 

  14. 14.

    Chan, T.F., Vese, L., et al.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)

    MATH  Article  Google Scholar 

  15. 15.

    Chen, Y., Lan, G., Ouyang, Y.: Optimal primal-dual methods for a class of saddle point problems. SIAM J. Optim. 24(4), 1779–1814 (2014)

    MathSciNet  MATH  Article  Google Scholar 

  16. 16.

    Cremers, D., Rousson, M., Deriche, R.: A review of statistical approaches to level set segmentation: integrating color, texture, motion and shape. Int. J. Comput. Vis. 72(2), 195–215 (2007)

    Article  Google Scholar 

  17. 17.

    Ekeland, I., Temam, R.: Convex analysis and variational problems. SIAM Classics in Applied Mathematics, Philadelphia (1976)

    MATH  Google Scholar 

  18. 18.

    Evans, L.C.: Partial differential equations and Monge-Kantorovich mass transfer. Curr. Dev. Math. 1997(1), 65–126 (1997)

    MATH  Article  Google Scholar 

  19. 19.

    Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell. 6, 721–741 (1984)

    MATH  Article  Google Scholar 

  20. 20.

    Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)

    MathSciNet  MATH  Article  Google Scholar 

  21. 21.

    Grady, L.: Random walks for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 28(11), 1768–1783 (2006)

    Article  Google Scholar 

  22. 22.

    Grady, L., Alvino, C.: Reformulating and optimizing the Mumford–Shah functional on a graph—faster, lower energy solution. In: ECCV 2008, pp. 248–261. Springer, Berlin (2008)

  23. 23.

    Jung, Y.M., Kang, S.H., Shen, J.: Multiphase image segmentation via Modica–Mortola phase transition. SIAM J. Appl. Math. 67(5), 1213–1232 (2007)

    MathSciNet  MATH  Article  Google Scholar 

  24. 24.

    Kanungo, T., Mount, D.M., Netanyahu, N.S., Piatko, C.D., Silverman, R., Wu, A.Y.: An efficient k-means clustering algorithm: analysis and implementation. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 881–892 (2002)

    MATH  Article  Google Scholar 

  25. 25.

    Kay, D., Tomasi, A., et al.: Color image segmentation by the vector-valued Allen–Cahn phase-field model: a multigrid solution. IEEE Trans. Image Process. 18(10), 2330–2339 (2009)

    MathSciNet  Article  Google Scholar 

  26. 26.

    Levinshtein, A., Stere, A., Kutulakos, K.N., Fleet, D.J., Dickinson, S.J., Siddiqi, K.: Turbopixels: fast superpixels using geometric flows. IEEE Trans. Pattern Anal. Mach. Intell. 31(12), 2290–2297 (2009)

    Article  Google Scholar 

  27. 27.

    Li, C., Huang, R., Ding, Z., Gatenby, J.C., Metaxas, D.N., C, G.J.: A level set method for image segmentation in the presence of intensity inhomogeneity with application to MRI. IEEE Trans. Image Process. 20(7), 2007–2016 (2011)

    MathSciNet  Article  Google Scholar 

  28. 28.

    Li, F., Ng, M.K., Zeng, T.Y., Shen, C.: A multiphase image segmentation method based on fuzzy region competition. SIAM J. Imaging Sci. 3(3), 277–299 (2010)

    MathSciNet  MATH  Article  Google Scholar 

  29. 29.

    Lukac, R., Plataniotis, K.N.: Color Image Processing: Methods and Applications. CRC Press, Boca Raton (2007)

    Google Scholar 

  30. 30.

    Luong, Q.T.: Color in computer vision. In: Chen, C.H., Pau, L.F., Wang, P.S.P. (eds.) Handbook of Pattern Recognition & Computer Vision, pp. 311–368. World Scientific Publishing Co., Inc., River Edge, NJ, USA (1993)

    Google Scholar 

  31. 31.

    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. ICCV 2, 416–423 (2001)

    Google Scholar 

  32. 32.

    Mumford, D., Shah, J.: Boundary detection by minimizing functionals. In: Ullman, S., Richards, W. (eds.) Image Understanding 1989. Ablex Publishing Corporation, New Jersey (1990)

    Google Scholar 

  33. 33.

    Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. 42(5), 577–685 (1989)

    MathSciNet  MATH  Article  Google Scholar 

  34. 34.

    Paschos, G.: Perceptually uniform color spaces for color texture analysis: an empirical evaluation. IEEE Trans. Image Process. 10(6), 932–937 (2001)

    MATH  Article  Google Scholar 

  35. 35.

    Plaza, A., Benediktsson, J.A., Boardman, J.W., Brazile, J., Bruzzone, L., Camps-Valls, G., Chanussot, J., Fauvel, M., Gamba, P., Gualtieri, A., et al.: Recent advances in techniques for hyperspectral image processing. Remote Sens. Environ. 113, S110–S122 (2009)

    Article  Google Scholar 

  36. 36.

    Pock, T., Chambolle, A., Cremers, D., Bischof, H.: A convex relaxation approach for computing minimal partitions. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 810–817 (2009)

  37. 37.

    Potts, R.B.: Some generalized order-disorder transformations. In: Mathematical Proceedings of the Cambridge Philosophical Society, vol. 48, pp. 106–109. Cambridge University Press, Cambridge (1952)

  38. 38.

    Rotaru, C., Graf, T., Zhang, J.: Color image segmentation in HSI space for automotive applications. J. Real-Time Image Process. 3(4), 311–322 (2008)

    Article  Google Scholar 

  39. 39.

    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60(1), 259–268 (1992)

    MathSciNet  MATH  Article  Google Scholar 

  40. 40.

    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000)

    Article  Google Scholar 

  41. 41.

    Storath, M., Weinmann, A.: Fast partitioning of vector-valued images. SIAM J. Imaging Sci. 7(3), 1826–1852 (2014)

    MathSciNet  MATH  Article  Google Scholar 

  42. 42.

    Tai, C., Zhang, X., Shen, Z.: Wavelet frame based multiphase image segmentation. SIAM J. Imaging Sci. 6(4), 2521–2546 (2013)

    MathSciNet  MATH  Article  Google Scholar 

  43. 43.

    Tai, Y.W., Jia, J., Tang, C.K.: Soft color segmentation and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 29(9), 1520–1537 (2007)

    Article  Google Scholar 

  44. 44.

    Townsend, D.: Multimodality imaging of structure and function. Phys. Med. Biol. 53(4), R1 (2008)

    MathSciNet  Article  Google Scholar 

  45. 45.

    Vandenbroucke, N., Macaire, L., Postaire, J.: Color image segmentation by pixel classification in an adapted hybrid color space. Application to soccer image analysis. Comput. Vis. Image Underst. 90(2), 190–216 (2003)

    Article  Google Scholar 

  46. 46.

    Vese, L.A., Chan, T.F.: A multiphase level set framework for image segmentation using the Mumford and Shah model. Int. J. Comput. Vis. 50(3), 271–293 (2002)

    MATH  Article  Google Scholar 

  47. 47.

    Wang, X., Tang, Y., Masnou, S., Chen, L.: A global/local affinity graph for image segmentation. IEEE Trans. Image Process. 24(4), 1399–1411 (2015)

    MathSciNet  Article  Google Scholar 

Download references

Acknowledgements

The authors thank G. Steidl and M. Bertalmío for constructive discussions. The work of X. Cai is partially supported by Welcome Trust, Issac Newton Trust, and KAUST Award No. KUK-I1-007-43. The work of R. Chan is partially supported by HKRGC GRF Grant No. CUHK300614, CUHK14306316, CRF Grant No. CUHK2/CRF/11G, CRF Grant C1007-15G, and AoE Grant AoE/M-05/12 The work of M. Nikolova is partially supported by HKRGC GRF Grant No. CUHK300614, and by the French Research Agency (ANR) under Grant No. ANR-14-CE27-001 (MIRIAM). The work of T. Zeng is partially supported by NSFC 11271049, 11671002, RGC 211911, 12302714 and RFGs of HKBU.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Tieyong Zeng.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cai, X., Chan, R., Nikolova, M. et al. A Three-Stage Approach for Segmenting Degraded Color Images: Smoothing, Lifting and Thresholding (SLaT). J Sci Comput 72, 1313–1332 (2017). https://doi.org/10.1007/s10915-017-0402-2

Download citation

Keywords

  • Mumford–Shah model
  • Convex variational models
  • Multiphase color image segmentation
  • Color spaces