Skip to main content
SpringerLink
Log in

Weighted and truncated \(L_1\) image smoothing based on unsupervised learning

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Edge-preserving image smoothing plays a vital role in the field of computational photography. In this paper, we propose a weighted and truncated \(L_1\)-regularized optimization model for image smoothing. We show that the weighted and truncated scheme significantly promotes the edge-preserving property. Furthermore, we propose a deep unsupervised learning-based filter based on the loss function defined by the proposed optimization model. The proposed filter leverages a U-Net structure, which fully exploits the spatially varying smoothing scales of the edge-preserving filtering. We have conducted extensive experiments to evaluate the proposed filter. The results suggest that our filter outperforms the state-of-the-art filters in image quality on various tasks, such as image smoothing, detail enhancing, HDR tone mapping, and edge detection. Meanwhile, our filter is extremely efficient. It is able to process 720P images in real-time (more than 16 frames per second) on a modern desktop with an Intel i7-8700K CPU, an NVIDIA GTX 1080 GPU and 16GB memory.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

Our source code, trained model, and data are available at https://github.com/dtz-dd/weighted-and-truncated-L1-smooth.

References

  1. Xie, Z., Lau, R.W.H., Gui, Y., Chen, M., Ma, L.: A gradient-domain-based edge-preserving sharpen filter. Vis. Comput. 28(12), 1195–1207 (2012)

    Article  Google Scholar 

  2. Tan, A., Liao, H., Zhang, B., Gao, M., Li, S., Bai, Y., et al.: Infrared image enhancement algorithm based on detail enhancement guided image filtering. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02741-6

    Article  Google Scholar 

  3. Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21(3), 257–266 (2002)

    Article  Google Scholar 

  4. Song, X., Huang, J., Cao, J., Song, D.: Feature spatial pyramid network for low-light image enhancement. Vis. Comput. 39, 489–499 (2022)

    Article  Google Scholar 

  5. Wang, C., Zhang, H., Liu, L.: Total generalized variation-based Retinex image decomposition. Vis. Comput. 37(1), 77–93 (2021)

    Article  Google Scholar 

  6. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)

    Article  Google Scholar 

  7. Xiao, C., Gan, J.: Fast image dehazing using guided joint bilateral filter. Vis. Comput. 28(6–8), 713–721 (2012)

    Article  Google Scholar 

  8. Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM Trans. Graph. 31(6), 139:1-139:10 (2012)

    Article  Google Scholar 

  9. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: IEEE International Conference on Computer Vision, pp. 839–846 (1998)

  10. Rao, S., Wang, H.: Optical flow estimation via weighted guided filtering with non-local steering kernel. Vis. Comput. 39(3), 835–845 (2023)

    Article  Google Scholar 

  11. Gastal, E.S.L., Oliveira, M.M.: Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30(4), 691–6911 (2011)

    Article  Google Scholar 

  12. Yin, H., Gong, Y., Qiu, G.: Side window filtering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8758–8766 (2019)

  13. Zhou, P., Xue, Y., Xue, M.: Adaptive side window joint bilateral filter. Vis. Comput. 39(4), 1533–1555 (2023)

    Google Scholar 

  14. Wang, H., Cao, J., Liu, X., Wang, J., Fan, T., Hu, J.: Least-squares images for edge-preserving smoothing. Comput. Vis. Media 1(1), 27–35 (2015)

    Article  Google Scholar 

  15. Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. 27(3), 67:1-67:10 (2008)

    Article  Google Scholar 

  16. Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via L\({}_{0}\) gradient minimization. ACM Trans. Graph. 30(6), 174:1-174:11 (2011)

    Article  Google Scholar 

  17. Bi, S., Han, X., Yu, Y.: An L\({}_{{1}}\) image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Trans. Graph. 34(4), 78:1-78:12 (2015)

    Article  MATH  Google Scholar 

  18. Ma, X., Li, X., Zhou, Y., Zhang, C.: Image smoothing based on global sparsity decomposition and a variable parameter. Comput. Vis. Media 7(4), 483–497 (2021)

    Article  Google Scholar 

  19. Feng, Y., Deng, S., Yan, X., Yang, X., Wei, M., Liu, L.: Easy2Hard: learning to solve the intractables from a synthetic dataset for structure-preserving image smoothing. IEEE Trans. Neural Netw. Learn. Syst. 33(12), 7223–7236 (2022)

    Article  Google Scholar 

  20. Wu, H., Zheng, S., Zhang, J., Huang, K.: Fast end-to-end trainable guided filter. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1838–1847 (2018)

  21. Zhu, F., Liang, Z., Jia, X., Zhang, L., Yu, Y.: A benchmark for edge-preserving image smoothing. IEEE Trans. Image Process. 28(7), 3556–3570 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  22. Chen, Q., Xu, J., Koltun, V.: Fast image processing with fully-convolutional networks. In: IEEE International Conference on Computer Vision, pp. 2516–2525 (2017)

  23. Xu, L., Ren, J.S.J., Yan, Q., Liao, R., Jia, J.: Deep edge-aware filters. In: International Conference on Machine Learning, pp. 1669–1678 (2015)

  24. Zhang, Q., Xu, L., Jia, J.: 100+ times faster weighted median filter (WMF). In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2830–2837 (2014)

  25. Fattal, R.: Edge-avoiding wavelets and their applications. ACM Trans. Graph. 28(3), 22 (2009)

    Article  Google Scholar 

  26. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12(7), 629–639 (1990)

    Article  Google Scholar 

  27. Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. In: European Conference on Computer Vision, pp. 815–830 (2014)

  28. Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. Int. J. Comput. Vis. 81(1), 24–52 (2009)

    Article  Google Scholar 

  29. Dai, L., Tang, L., Tang, J.: Speed up bilateral filtering via sparse approximation on a learned cosine dictionary. IEEE Trans. Circuits Syst. Video Technol. 30(3), 603–617 (2020)

    Article  Google Scholar 

  30. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60, 259–268 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  31. Yang, Y., Zheng, H., Zeng, L., Shen, X., Zhan, Y.: L1-regularized reconstruction model for edge-preserving filtering. IEEE Trans. Multimed. 25, 4148–4162 (2022)

    Article  Google Scholar 

  32. Yang, Y., Hui, H., Zeng, L., Zhao, Y., Zhan, Y., Yan, T.: Edge-preserving image filtering based on soft clustering. IEEE Trans. Circuits Syst. Video Technol. 32(7), 4150–4162 (2022)

    Article  Google Scholar 

  33. Liu, W., Zhang, P., Chen, X., Shen, C., Huang, X., Yang, J.: Embedding bilateral filter in least squares for efficient edge-preserving image smoothing. IEEE Trans. Circuits Syst. Video Technol. 30(1), 23–35 (2020)

    Article  Google Scholar 

  34. Liu, W., Zhang, P., Lei, Y., Huang, X., Yang, J., Ng, M.: A generalized framework for edge-preserving and structure-preserving image smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6631–6648 (2022)

    Article  Google Scholar 

  35. Liu, W., Zhang, P., Huang, X., Yang, J., Shen, C., Reid, I.: Real-time image smoothing via iterative least squares. ACM Trans. Graph. 39(3), 28:1-28:24 (2020)

    Article  Google Scholar 

  36. Fan, Q., Yang, J., Wipf, D.P., Chen, B., Tong, X.: Image smoothing via unsupervised learning. ACM Trans. Graph. 37(6), 259:1-259:14 (2018)

    Article  Google Scholar 

  37. Gharbi, M., Chen, J., Barron, J.T., Hasinoff, S.W., Durand, F.: Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. 36(4), 118:1-118:12 (2017)

    Article  Google Scholar 

  38. Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017)

  39. Liu, S., Pan, J., Yang, M.: Learning recursive filters for low-level vision via a hybrid neural network. In: European Conference on Computer Vision, pp. 560–576 (2016)

  40. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. Med. Image Comput. Comput. Assist. Interv. 9351, 234–241 (2015)

    Google Scholar 

  41. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)

  42. Xu, J., Liu, Z., Hou, Y., Zhen, X., Shao, L., Cheng, M.: Pixel-level non-local image smoothing with objective evaluation. IEEE Trans. Multimed. 23, 4065–4078 (2021)

    Article  Google Scholar 

  43. Liu, W., Chen, X., Shen, C., Liu, Z., Yang, J.: Semi-global weighted least squares in image filtering. In: IEEE International Conference on Computer Vision, pp. 5862–5870 (2017)

  44. Sun, Z., Han, B., Li, J., Zhang, J., Gao, X.: Weighted guided image filtering with steering kernel. IEEE Trans. Image Process. 29, 500–508 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  45. Arbelaez, P., Maire, M., Fowlkes, C.C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)

    Article  Google Scholar 

  46. Rother, C., Kolmogorov, V., Blake, A.: GrabCut: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)

    Article  Google Scholar 

  47. Ferrari, V., Fevrier, L., Jurie, F., Schmid, C.: Groups of adjacent contour segments for object detection. IEEE Trans. Pattern Anal. Mach. Intell. 30(1), 36–51 (2008)

    Article  Google Scholar 

  48. Girshick, R.B., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)

  49. Canny, J.F.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986)

    Article  Google Scholar 

  50. DeCarlo, D., Santella, A.: Stylization and abstraction of photographs. ACM Trans. Graph. 21(3), 769–776 (2002)

    Article  Google Scholar 

  51. Winnemöller, H., Olsen, S.C., Gooch, B.: Real-time video abstraction. ACM Trans. Graph. 25(3), 1221–1226 (2006)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Jiangsu University, Xuefu, Zhenjiang, 212013, Jiangsu, China

    Yang Yang, Dan Wu, Ling Tang & Lanling Zeng

  2. School of Artificial Intelligence, Nanjing University of Information Science and Technology, Ningliu, Nanjing, 210044, Jiangsu, China

    Zhigeng Pan

Authors
  1. Yang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ling Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lanling Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhigeng Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Yang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Y., Wu, D., Tang, L. et al. Weighted and truncated \(L_1\) image smoothing based on unsupervised learning. Vis Comput (2023). https://doi.org/10.1007/s00371-023-03141-0

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-023-03141-0

Keywords

Search

Navigation