The Visual Computer

, Volume 33, Issue 11, pp 1467–1482 | Cite as

A nonlocal \(L_{0}\) model with regression predictor for saliency detection and extension

  • Yiyang Wang
  • Risheng LiuEmail author
  • Xiaoliang Song
  • Zhixun Su
Original Article


Estimating salient object regions automatically has enhanced many computer vision applications in recent years. By observing the intrinsic sparsity of saliency map, we propose a graph-based nonlocal \(L_{0}\) (NL\(L_{0}\)) minimization framework to extract its sparse geometric structure. Our experimental results demonstrate that our method with artificially designed control map yields a significant improvement compared with the state-of-the-art saliency detection methods on four publicly available data sets. These saliency maps are further used for content-aware image resizing and unsupervised matting to test their uniformity. Moreover, we propose to learn the control map adaptively from training data. This strategy totally differs from the previously designed one, which is verified to be effective on image-classified data set. NL\(L_{0}\) with data-driven strategy is extended to interactive segmentation task and is affirmed to be better-performed than other advanced interactive approaches.


Nonlocal \(L_{0}\) Regression predictor Saliency detection Interactive segmentation 

Supplementary material

371_2016_1292_MOESM1_ESM.pdf (54.9 mb)
Supplementary material 1 (pdf 56241 KB)


  1. 1.
    Achanta, R., Estrada, F., Wils, P., Süsstrunk, S.: Salient region detection and segmentation. In: ICVS (2008)Google Scholar
  2. 2.
    Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: CVPR (2009)Google Scholar
  3. 3.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. PAMI 34(11), 2274–2282 (2012)CrossRefGoogle Scholar
  4. 4.
    Achanta, R., Susstrunk, S.: Saliency detection using maximum symmetric surround. In: ICIP (2010)Google Scholar
  5. 5.
    Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM TOG 26(3), 10 (2007)CrossRefGoogle Scholar
  6. 6.
    Bao, C., Ji, H., Quan, Y., Shen, Z.: \(l_{0}\) norm based dictionary learning by proximal methods with global convergence. In: CVPR (2014)Google Scholar
  7. 7.
    Borji, A., Cheng, M.M., Jiang, H., Li, J.: Salient object detection: a survey (2014, arXiv preprint)Google Scholar
  8. 8.
    Bougleux, S., Elmoataz, A., Melkemi, M.: Discrete regularization on weighted graphs for image and mesh filtering. In: Scale Space and Variational Methods in Computer Vision, pp. 128–139. Springer, Berlin (2007)Google Scholar
  9. 9.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefzbMATHGoogle Scholar
  10. 10.
    Bresson, X.: A short note for nonlocal tv minimization (2009)Google Scholar
  11. 11.
    Cao, X., Tao, Z., Bao, Z., Fu, H., Wei, F.: Self-adaptively weighted co-saliency detection via rank constraint. IEEE TIP 23(9), 4175–4186 (2014)MathSciNetGoogle Scholar
  12. 12.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2, 27:1–27:27 (2011)CrossRefGoogle Scholar
  13. 13.
    Chang, K.Y., Liu, T.L., Chen, H.T., Lai, S.H.: Fusing generic objectness and visual saliency for salient object detection. In: ICCV (2011)Google Scholar
  14. 14.
    Chen, Q., Li, D., Tang, C.K.: Knn matting. In: CVPR (2012)Google Scholar
  15. 15.
    Chen, T., Cheng, M.M., Tan, P., Shamir, A., Hu, S.M.: Sketch2photo: internet image montage. In: ACM TOG, vol. 28, p. 124 (2009)Google Scholar
  16. 16.
    Cheng, M.M., Zhang, G.X., Mitra, N.J., Huang, X., Hu, S.M.: Global contrast based salient region detection. In: CVPR (2011)Google Scholar
  17. 17.
    Chia, A.Y.S., Zhuo, S., Gupta, R.K., Tai, Y.W., Cho, S.Y., Tan, P., Lin, S.: Semantic colorization with internet images. In: ACM TOG, vol. 30, p. 156 (2011)Google Scholar
  18. 18.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)zbMATHGoogle Scholar
  19. 19.
    Einhäuser, W., König, P.: Does luminance-contrast contribute to a saliency map for overt visual attention? Eur. J. Neurosci. 17(5), 1089–1097 (2003)CrossRefGoogle Scholar
  20. 20.
    Fu, H., Cao, X., Tu, Z.: Cluster-based co-saliency detection. IEEE TIP 22(10), 3766–3778 (2013)MathSciNetGoogle Scholar
  21. 21.
    Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(l_{p}\) minimization. Math. Program. 129(2), 285–299 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Multiscale Model. Simul. 7(3), 1005–1028 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Goferman, S., Zelnik-Manor, L., Tal, A.: Context-aware saliency detection. IEEE Trans. PAMI 34(10), 1915–1926 (2012)CrossRefGoogle Scholar
  24. 24.
    Grady, L.: Random walks for image segmentation. IEEE Trans. PAMI 28(11), 1768–1783 (2006)CrossRefGoogle Scholar
  25. 25.
    Harel, J., Koch, C., Perona, P.: Graph-based visual saliency. In: NIPS (2006)Google Scholar
  26. 26.
    He, K., Sun, J., Tang, X.: Guided image filtering. In: ECCV (2010)Google Scholar
  27. 27.
    Hou, X., Zhang, L.: Saliency detection: A spectral residual approach. In: CVPR (2007)Google Scholar
  28. 28.
    Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. PAMI 20(11), 1254–1259 (1998)CrossRefGoogle Scholar
  29. 29.
    Jia, Y., Han, M.: Category-independent object-level saliency detection. In: ICCV (2013)Google Scholar
  30. 30.
    Jiang, B., Zhang, L., Lu, H., Yang, C., Yang, M.H.: Saliency detection via absorbing markov chain. In: ICCV (2013)Google Scholar
  31. 31.
    Jiang, H., Wang, J., Yuan, Z., Liu, T., Zheng, N., Li, S.: Automatic salient object segmentation based on context and shape prior. In: BMVC (2011)Google Scholar
  32. 32.
    Jiang, P., Ling, H., Yu, J., Peng, J.: Salient region detection by ufo: Uniqueness, focusness and objectness. In: ICCV (2013)Google Scholar
  33. 33.
    Jiang, Z., Davis, L.S.: Submodular salient region detection. In: CVPR (2013)Google Scholar
  34. 34.
    Kanan, C., Cottrell, G.: Robust classification of objects, faces, and flowers using natural image statistics. In: CVPR (2010)Google Scholar
  35. 35.
    Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: ACM TOG, vol. 23, pp. 689–694 (2004)Google Scholar
  36. 36.
    Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. PAMI 30(2), 228–242 (2008)CrossRefGoogle Scholar
  37. 37.
    Lin, Z., Liu, R., Su, Z.: Linearized alternating direction method with adaptive penalty for low-rank representation. In: NIPS (2011)Google Scholar
  38. 38.
    Liu, R., Lin, Z., Shan, S.: Adaptive partial differential equation learning for visual saliency detection. In: CVPR (2014)Google Scholar
  39. 39.
    Liu, R., Lin, Z., Su, Z.: Linearized alternating direction method with parallel splitting and adaptive penalty for separable convex programs in machine learning. In: ACML (2013)Google Scholar
  40. 40.
    Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.Y.: Learning to detect a salient object. IEEE Trans. PAMI 33(2), 353–367 (2011)CrossRefGoogle Scholar
  41. 41.
    Ma, Y.F., Zhang, H.J.: Contrast-based image attention analysis by using fuzzy growing. In: ACM Multimedia (2003)Google Scholar
  42. 42.
    Mai, L., Niu, Y., Liu, F.: Saliency aggregation: A data-driven approach. In: CVPR (2013)Google Scholar
  43. 43.
    Margolin, R., Zelnik-Manor, L., Tal, A.: How to evaluate foreground maps? In: CVPR (2014)Google Scholar
  44. 44.
    Nesterov, Y., Nemirovskii, A., Ye, Y.: Interior-point Polynomial Algorithms in Convex Programming, vol. 13. SIAM, Philadelphia (1994)CrossRefzbMATHGoogle Scholar
  45. 45.
    Nocedal, J., Wright, S.: Numerical Optimization. Springer, Berlin (2006)zbMATHGoogle Scholar
  46. 46.
    Ochs, P., Malik, J., Brox, T.: Segmentation of moving objects by long term video analysis. IEEE Trans. PAMI 36(6), 1187–1200 (2014)CrossRefGoogle Scholar
  47. 47.
    Pan, J., Hu, Z., Su, Z., Yang, M.H.: Deblurring text images via \(l_{0}\)-regularized intensity and gradient prior. In: CVPR (2014)Google Scholar
  48. 48.
    Perazzi, F., Krahenbuhl, P., Pritch, Y., Hornung, A.: Saliency filters: Contrast based filtering for salient region detection. In: CVPR (2012)Google Scholar
  49. 49.
    Scharfenberger, C., Wong, A., Fergani, K., Zelek, J.S., Clausi, D.A.: Statistical textural distinctiveness for salient region detection in natural images. In: CVPR (2013)Google Scholar
  50. 50.
    Shen, X., Wu, Y.: A unified approach to salient object detection via low rank matrix recovery. In: CVPR (2012)Google Scholar
  51. 51.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. PAMI 22(8), 888–905 (2000)CrossRefGoogle Scholar
  52. 52.
    Stalder, S., Grabner, H., Van Gool, L.: Dynamic objectness for adaptive tracking. In: ACCV (2013)Google Scholar
  53. 53.
    Tsai, D., Flagg, M., Nakazawa, A., Rehg, J.M.: Motion coherent tracking using multi-label mrf optimization. IJCV 100(2), 190–202 (2012)MathSciNetCrossRefGoogle Scholar
  54. 54.
    Wang, Y., Liu, R., Song, X., Su, Z.: Saliency detection via nonlocal \(l_{0}\) minimization. In: ACCV (2014)Google Scholar
  55. 55.
    Wang, Y., Liu, R., Song, X., Su, Z.: Linearized alternating direction method with penalization for nonconvex and nonsmooth optimization. In: AAAI (2016)Google Scholar
  56. 56.
    Wei, Y., Wen, F., Zhu, W., Sun, J.: Geodesic saliency using background priors. In: ECCV (2012)Google Scholar
  57. 57.
    Xie, Y., Lu, H.: Visual saliency detection based on bayesian model. In: ICIP (2011)Google Scholar
  58. 58.
    Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via \(l_{0}\) gradient minimization. ACM TOG 30(6), 174 (2011)Google Scholar
  59. 59.
    Xu, L., Zheng, S., Jia, J.: Unnatural \(l_{0}\) sparse representation for natural image deblurring. In: CVPR (2013)Google Scholar
  60. 60.
    Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: CVPR (2013)Google Scholar
  61. 61.
    Yang, C., Zhang, L., Lu, H.: Graph-regularized saliency detection with convex-hull-based center prior. IEEE Signal Process. Lett. 20(7), 637–640 (2013)CrossRefGoogle Scholar
  62. 62.
    Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.H.: Saliency detection via graph-based manifold ranking. In: CVPR (2013)Google Scholar
  63. 63.
    Yuan, G., Ghanem, B.: l0tv: A new method for image restoration in the presence of impulse noise. In: CVPR (2015)Google Scholar
  64. 64.
    Zhai, Y., Shah, M.: Visual attention detection in video sequences using spatiotemporal cues. In: ACM Multimedia (2006)Google Scholar
  65. 65.
    Zheng, Y., Kambhamettu, C.: Learning based digital matting. In: ICCV (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Yiyang Wang
    • 1
  • Risheng Liu
    • 2
    • 3
    Email author
  • Xiaoliang Song
    • 1
  • Zhixun Su
    • 1
  1. 1.School of Mathematical SciencesDalian University of TechnologyDalianChina
  2. 2.School of Software TechnologyDalian University of TechnologyDalianChina
  3. 3.Key Laboratory for Ubiquitous Network and Service Software of Liaoning ProvinceDalian University of TechnologyDalianChina

Personalised recommendations