Abstract
Edge-ware image smoothing, which aims at removing fine details while respecting salient edges, is a prevalent topic in the field of computational imaging and photography. In this paper, we propose a novel weighted sparse gradient reconstruction model for edge-aware image smoothing. The proposed method first suppresses the low-amplitude gradients with an edge-aware mapping function. The processed gradients are then fed into a weighted \(L_1\) gradient reconstruction model to derive the output. The \(L_1\) regularization enforces the sparsity on the output gradients, thus facilitating the edge-aware property. The weighted scheme further promotes the edge awareness of the filter. In order to solve the proposed model, we propose an efficient solution based on the combination of the augmented Lagrange multipliers and the Fourier domain optimization. The GPU implementation of our filter takes 39ms to process a 720P color image on an NVIDIA RTX 3070. We have applied the proposed filter in various tasks, including edge-aware smoothing, edge extraction, image abstraction, and low-light image enhancement. Both the qualitative and quantitative results validate the superiority of the proposed filter.
Similar content being viewed by others
Data availability
All the datasets explored in this paper are publicly available.
References
Esfandarani, H.T., Milanfar, P.: Fast multilayer Laplacian enhancement. IEEE Trans. Comput. Imaging 2(4), 496–509 (2016)
Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21(3), 257–266 (2002)
Guo, X., Li, Y., Ling, H.: LIME: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2017)
Hussein, A.A., Yang, X.: Colorization using edge-preserving smoothing filter. Signal Image Video Process. 8(8), 1681–1689 (2014)
Ye, X., Sang, X., Chen, D., Wang, P., Wang, K., Yan, B., Liu, B., Wang, H., Qi, S.: Superpixel guided network for three-dimensional stereo matching. IEEE Trans. Comput. Imaging 8, 54–68 (2022)
Zhu, H., Sun, X., Zhang, Q., Wang, Q., Robles-Kelly, A., Li, H., You, S.: Full view optical flow estimation leveraged from light field superpixel. IEEE Trans. Comput. Imaging 6, 12–23 (2020)
He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)
Chen, B., Wu, S.: Weighted aggregation for guided image filtering. Signal Image Video Process. 14(3), 491–498 (2020)
Yin, H., Gong, Y., Qiu, G.: Side window filtering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 66, pp. 8758–8766 (2019)
Liu, W., Zhang, P., Lei, Y., Huang, X., Yang, J., Ng, M.: A generalized framework for edge-preserving and structure-preserving image smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6631–6648 (2022)
Yang, Y., Hui, H., Zeng, L., Zhao, Y., Zhan, Y., Yan, T.: Edge-preserving image filtering based on soft clustering. IEEE Trans. Circuits Syst. Video Technol. 32(7), 4150–4162 (2022)
Liu, W., Zhang, P., Huang, X., Yang, J., Shen, C., Reid, I.: Real-time image smoothing via iterative least squares. ACM Trans. Graph. 39(3), 28:1-28:24 (2020)
Chen, Y., Gao, Y.: Adaptive fourth-order diffusion smoothing via bilateral kernel. Signal Image Video Process. 15(6), 1125–1133 (2021)
Feng, Y., Deng, S., Yan, X., Yang, X., Wei, M., Liu, L.: Easy2hard: learning to solve the intractables from a synthetic dataset for structure-preserving image smoothing. IEEE Trans. Neural Netw. Learn. Syst. 33(12), 7223–7236 (2022)
Zhu, F., Liang, Z., Jia, X., Zhang, L., Yu, Y.: A benchmark for edge-preserving image smoothing. IEEE Trans. Image Process. 28(7), 3556–3570 (2019)
Fan, Q., Yang, J., Wipf, D.P., Chen, B., Tong, X.: Image smoothing via unsupervised learning. ACM Trans. Graph. 37(6), 259:1-259:14 (2018)
Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of the International Conference on Computer Vision, vol. 66, pp. 839–846 (1998)
Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. Int. J. Comput. Vis. 81(1), 24–52 (2009)
Adams, A., Baek, J., Davis, M.A.: Fast high-dimensional filtering using the permutohedral lattice. Comput. Graph. Forum 29(2), 753–762 (2010)
Yang, Q., Tan, K., Ahuja, N.: Real-time O(1) bilateral filtering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 557–564 (2009)
Sugimoto, K., Fukushima, N., Kamata, S.: 200 FPS constant-time bilateral filter using SVD and tiling strategy. In: IEEE International Conference on Image Processing, pp. 190–194 (2019)
Chaudhury, K.N., Dabhade, S.D.: Fast and provably accurate bilateral filtering. IEEE Trans. Image Process. 25(6), 2519–2528 (2016)
Nair, P., Popli, A., Chaudhury, K.N.: A fast approximation of the bilateral filter using the discrete Fourier transform. Image Process. Line 7, 115–130 (2017)
Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008)
Li, Z., Zheng, J., Zhu, Z., Yao, W., Wu, S.: Weighted guided image filtering. IEEE Trans. Image Process. 24(1), 120–129 (2015)
Sun, Z., Han, B., Li, J., Zhang, J., Gao, X.: Weighted guided image filtering with steering kernel. IEEE Trans. Image Process. 6, 66 (2020)
Gastal, E.S.L., Oliveira, M.M.: Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30(4), 69:1-69:11 (2011)
Thévenaz, P., Sage, D., Unser, M.: Bi-exponential edge-preserving smoother. IEEE Trans. Image Process. 21(9), 3924–3936 (2012)
Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. 27(3), 67:1-67:10 (2008)
Min, D., Choi, S., Lu, J., Ham, B., Sohn, K., Do, M.N.: Fast global image smoothing based on weighted least squares. IEEE Trans. Image Process. 23(12), 5638–5653 (2014)
Liu, W., Chen, X., Shen, C., Liu, Z., Yang, J.: Semi-global weighted least squares in image filtering. In: Proceedings of the International Conference on Computing Vision, pp. 5862–5870 (2017)
Liu, W., Zhang, P., Chen, X., Shen, C., Huang, X., Yang, J.: Embedding bilateral filter in least squares for efficient edge-preserving image smoothing. IEEE Trans. Circuits Syst. Video Technol. 30(1), 23–35 (2020)
Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via L\({}_{0}\) gradient minimization. ACM Trans. Graph. 30(6), 174:1-174:11 (2011)
Bi, S., Han, X., Yu, Y.: An L\({}_{{1}}\) image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Trans. Graph. 34(4), 78:1-78:12 (2015)
Badri, H., Yahia, H., Aboutajdine, D.: Fast edge-aware processing via first order proximal approximation. IEEE Trans. Vis. Comput. Graph. 21(6), 743–755 (2015)
Liu, W., Zhang, P., Lei, Y., Huang, X., Yang, J., Ng, M.: A generalized framework for edge-preserving and structure-preserving image smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6631–6648 (2022)
Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)
Yang, Q., Wang, S., Ahuja, N.: SVM for edge-preserving filtering. In: Proceedings of the IEEE Conference on Computing Vision Pattern Recognition, pp. 1775–1782 (2010)
Xu, L., Ren, J.S.J., Yan, Q., Liao, R., Jia, J.: Deep edge-aware filters. In: Proceedings of the International Conference Machine Learning, pp. 1669–1678 (2015)
Chen, Q., Xu, J., Koltun, V.: Fast image processing with fully-convolutional networks. In: Proceedings of the International Conference on Computing Vision, pp. 2516–2525 (2017)
Kingma, D.P., Ba, J., Adam: A method for stochastic optimization. In: International Conference on Learning Representations (2015)
Liang, Z., Xu, J., Zhang, D., Cao, Z., Zhang, L.: A hybrid l1–l0 layer decomposition model for tone mapping. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4758–4766 (2018)
Shibata, T., Tanaka, M., Okutomi, M.: Gradient-domain image reconstruction framework with intensity-range and base-structure constraints. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2745–2753 (2016)
Tanaka, M., Shibata, T., Okutomi, M.: Gradient-based low-light image enhancement. In: IEEE International Conference on Consumer Electronics, pp. 1–2 (2019)
Yang, Y., Zheng, H., Zeng, L., Shen, X., Zhan, Y.: L1-regularized reconstruction model for edge-preserving filtering. IEEE Trans Multimed. 66, 1 (2022)
Bhat, P., Curless, B., Cohen, M.F., Zitnick, C.L.: Fourier analysis of the 2d screened poisson equation for gradient domain problems. In: Proceedings of the European Conference on Computer Vision, pp. 114–128 (2008)
Donoho, D.L.: De-noising by soft-thresholding. IEEE Trans. Inf. Theory 41(3), 613–627 (1995)
Zhang, Y., Guo, X., Ma, J., Liu, W., Zhang, J.: Beyond brightening low-light images. Int. J. Comput. Vis. 129(4), 1013–1037 (2021)
Liu, R., Ma, L., Zhang, J., Fan, X., Luo, Z.: Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 10561–10570 (2021)
Zhang, Y., Di, X., Zhang, B., Wang, C.: Self-supervised image enhancement network: training with low light images only. CoRR arXiv:2002.11300 (2020)
Wang, W., Wei, C., Yang, W., Liu, J.: Gladnet: Low-light enhancement network with global awareness. In: IEEE International Conference on Automatic face & Gesture Recognition, pp. 751–755 (2018)
Lv, F., Lu, F., Wu, J., Lim, C.: MBLLEN: low-light image/video enhancement using cnns. In: British Machine Vision Conference, p. 220 (2018)
Wei, C., Wang, W., Yang, W., Liu, J.: Deep retinex decomposition for low-light enhancement. British Machine Vision Conference p. 155 (2018)
Guo, C.G., Li, C., Guo, J., Loy, C.C., Hou, J., Kwong, S., Cong, R.: Zero-reference deep curve estimation for low-light image enhancement. In: IEEE conference on Computer Vision and Pattern Recognition, pp. 1780–1789 (2020)
Ma, L., Ma, T., Liu, R., Fan, X., Luo, Z.: Toward fast, flexible, and robust low-light image enhancement. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5627–5636 (2022)
Hao, S., Han, X., Guo, Y., Xu, X., Wang, M.: Low-light image enhancement with semi-decoupled decomposition. IEEE Trans. Multimed. 22(12), 3025–3038 (2020)
Fan, M., Wang, W., Yang, W., Liu, J.: Integrating semantic segmentation and retinex model for low-light image enhancement. In: International Conference on Multimedia, pp. 2317–2325 (2020)
Wu, H., Zheng, S., Zhang, J., Huang, K.: Fast end-to-end trainable guided filter. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1838–1847 (2018)
Ham, B., Cho, M., Ponce, J.: Robust guided image filtering using nonconvex potentials. IEEE Trans. Pattern Anal. Mach. Intell. 40(1), 192–207 (2018)
Sun, Z., Han, B., Li, J., Zhang, J., Gao, X.: Weighted guided image filtering with steering kernel. IEEE Trans. Image Process. 29, 500–508 (2020)
Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. In: European Conference on Computer Vision, vol. 8691, pp. 815–830 (2014)
Xu, J., Liu, Z., Hou, Y., Zhen, X., Shao, L., Cheng, M.: Pixel-level non-local image smoothing with objective evaluation. IEEE Trans. Multimed. 23, 4065–4078 (2021)
Deng, G., Galetto, F., Al-nasrawi, M., Waheed, W.: A guided edge-aware smoothing-sharpening filter based on patch interpolation model and generalized gamma distribution. IEEE Open J. Signal Process. 2, 119–135 (2021)
Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM Trans. Graph. 31(6), 139:1-139:10 (2012)
Yeganeh, H., Wang, Z.: Objective quality assessment of tone-mapped images. IEEE Trans. Image Process. 22(2), 657–667 (2013)
Canny, J.F.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986)
Winnemöller, H., Olsen, S.C., Gooch, B.: Real-time video abstraction. ACM Trans. Graph. 25(3), 1221–1226 (2006)
Hai, J., Xuan, Z., Yang, R., Hao, Y., Zou, F., Lin, F., Han, S.: R2rnet: low-light image enhancement via real-low to real-normal network. CoRR arXiv:2106.14501 (2021)
Yin, H., Gong, Y., Qiu, G.: Side window filtering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8758–8766 (2019)
Funding
This work is supported by National Natural Science Foundation of China, Grant Nos. 61402205 and 62072150.
Author information
Authors and Affiliations
Contributions
LZ supervised the research; YC conducted the experiments; LZ and YC wrote the main manuscript text; YY and ZP revised the manuscript; all authors reviewed the manuscript.
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zeng, L., Chen, Y., Yang, Y. et al. Edge-aware image smoothing via weighted sparse gradient reconstruction. SIViP 17, 4285–4293 (2023). https://doi.org/10.1007/s11760-023-02661-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-023-02661-5