Erasing Appearance Preservation in Optimization-Based Smoothing

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12351)


Optimization-based Image smoothing is routinely formulated as the game between a smoothing energy and an appearance preservation energy. Achieving adequate smoothing is a fundamental goal of these Image smoothing algorithms. We show that partially “erasing” the appearance preservation facilitate adequate Image smoothing. In this paper, we call this manipulation as Erasing Appearance Preservation (EAP). We conduct an user study, allowing users to indicate the “erasing” positions by drawing scribbles interactively, to verify the correctness and effectiveness of EAP. We observe the characteristics of human-indicated “erasing” positions, and then formulate a simple and effective 0-1 knapsack to automatically synthesize the “erasing” positions. We test our synthesized erasing positions in a majority of Image smoothing methods. Experimental results and large-scale perceptual human judgments show that the EAP solution tends to encourage the pattern separation or elimination capabilities of Image smoothing algorithms. We further study the performance of the EAP solution in many image decomposition problems to decompose textures, shadows, and the challenging specular reflections. We also present examinations of diversiform image manipulation applications like texture removal, retexturing, intrinsic decomposition, layer extraction, recoloring, material manipulation, etc. Due to the widespread applicability of Image smoothing, the EAP is also likely to be used in more image editing applications.


Image smoothing 

Supplementary material (70.7 mb)
Supplementary material 1 (zip 72404 KB)


  1. 1.
    Bach, F., et al.: Structured sparsity through convex optimization. Stat. Sci. 27(4), 450–468 (2012)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Bao, L., Song, Y., Yang, Q., Yuan, H., Wang, G.: Tree filtering efficient structure preserving smoothing with a minimum spanning tree. IEEE Trans. Image Process. (2014)Google Scholar
  3. 3.
    Barrow, H.G., Tenenbaum, J.M.: Recovering intrinsic scene characteristics from images. In: Hanson, A., Riseman, E. (eds.) Computer Vision Systems, pp. 3–26. Academic Press (1978)Google Scholar
  4. 4.
    Bell, S., Bala, K., Snavely, N.: Intrinsic images in the wild. ACM Trans. Graph. 33(4), 1–12 (2014)CrossRefGoogle Scholar
  5. 5.
    Bi, S., Han, X., Yu, Y.: An L1 image transform for edge preserving smoothing and scene level intrinsic decomposition. ACM Trans. Graph. 34(4), 1–12 (2015)CrossRefGoogle Scholar
  6. 6.
    Bousseau, A., Paris, S., Durand, F.: User-assisted intrinsic images. ACM Trans. Graph. (2009)Google Scholar
  7. 7.
    Buzug, M, T.: Computed Tomography. Springer, Heidelberg (2008).
  8. 8.
    Carroll, R., Ramamoorthi, R., Agrawala, M.: Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. Graph. (2011)Google Scholar
  9. 9.
    Champandard, A.J.: Semantic style transfer and turning two-bit doodles into fine artworks. CoRR abs/1603.01768 (2016)Google Scholar
  10. 10.
    Chen, Q., Li, D., Tang, C.K.: KNN matting. IEEE Trans. Pattern Anal. Mach. Intell. 35(9), 2175–2188 (2013)CrossRefGoogle Scholar
  11. 11.
    Cho, H., Lee, H., Kang, H., Lee, S.: Bilateral texture filtering. ACM Trans. Graph. 33(4), 1–8 (2014)CrossRefGoogle Scholar
  12. 12.
    Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  13. 13.
    Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.: Revisiting deep intrinsic image decompositions. In: CVPR (2018)Google Scholar
  14. 14.
    Fan, Q., Yang, J., Wipf, D., Chen, B., Tong, X.: Image smoothing via unsupervised learning. ACM Trans. Graph. 37(6), 1–14 (2018)CrossRefGoogle Scholar
  15. 15.
    Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. J. Math. Imaging Vis. 31(2–3), 255–269 (2008)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Garces, E., Munoz, A., Lopez-Moreno, J., Gutierrez, D.: Intrinsic images by clustering. In: Computer Graphics Forum (2012)Google Scholar
  17. 17.
    Grosse, R., Johnson, M.K., Adelson, E.H., Freeman, W.T.: Ground truth dataset and baseline evaluations for intrinsic image algorithms. In: ICCV (2019)Google Scholar
  18. 18.
    He, K., Sun, J., Tang, X.: Guied image filtering. TPAMI 35(6), 1397–1409 (2013)CrossRefGoogle Scholar
  19. 19.
    Hoeltgen, L., Setzer, S., Weickert, J.: An optimal control approach to find sparse data for laplace interpolation. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.-C. (eds.) EMMCVPR 2013. LNCS, vol. 8081, pp. 151–164. Springer, Heidelberg (2013). Scholar
  20. 20.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (Proc. of SIGGRAPH 2017) 36(4), 107:1–107:14 (2017)Google Scholar
  21. 21.
    Kovacs, B., Bell, S., Snavely, N., Bala, K.: Shading annotations in the wild. In: CVPR (2017)Google Scholar
  22. 22.
    Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: ACM SIGGRAPH 2004 Papers, SIGGRAPH 2004, pp. 689–694. Association for Computing Machinery, New York (2004)Google Scholar
  23. 23.
    Li, Z., Snavely, N.: CGIntrinsics: better intrinsic image decomposition through physically-based rendering. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 381–399. Springer, Cham (2018). Scholar
  24. 24.
    Li, Z., Snavely, N.: Learning intrinsic image decomposition from watching the world. In: CVPR (2018)Google Scholar
  25. 25.
    Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: The European Conference on Computer Vision (ECCV) (2018)Google Scholar
  26. 26.
    Min, D., Choi, S., Lu, J., Ham, B., Sohn, K., Do, M.N.: Fast global image smoothing based on weighted least squares. IEEE Trans. Image Process. 23(12), 5638–5653 (2014)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Prasath, V.S., Vorotnikov, D., Pelapur, R., Jose, S., Seetharaman, G., Palaniappan, K.: Multiscale Tikhonovtotal variation image restoration using spatially varying edge coherence exponent. IEEE Trans. Image Process. 24(12), 5220–5235 (2015)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D: Nonlinear Phenomena 60(1–4), 259–268 (1992)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Serra, M., Penacchio, O., Benavente, R., Vanrell, M.: Names and shades of color for intrinsic image estimation. In: CVPR (2012)Google Scholar
  30. 30.
    Shen, J., Yang, X., Jia, Y., Li, X.: Intrinsic images using optimization. In: CVPR (2011)Google Scholar
  31. 31.
    Tomasi, C.: Bilateral filtering for gray and color images. In: ICCV (1998)Google Scholar
  32. 32.
    Holland, P.W., Welsch, R.E.: Robust regression using iteratively reweighted leastsquares. Commun. Stat. Theory Methods 6(9), 813–827 (1977)CrossRefGoogle Scholar
  33. 33.
    Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via L0 gradient minimization. ACM Trans. Graph. (2011)Google Scholar
  34. 34.
    Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM Trans. Graph. 31(6), 1–10 (2012)Google Scholar
  35. 35.
    Yang, J., Zhang, Y., Yin, W.: An efficient TVL1 algorithm for deblurring multichannel images corrupted by impulsive noise. SIAM J. Sci. Comput. 31(4), 2842–2865 (2009)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Yin, H., Gong, Y., Qiu, G.: Side window filtering. In: CVPR (2019)Google Scholar
  37. 37.
    Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., Lin, S.: A closed-form solution to retinex with nonlocal texture constraints. TPAMI 34(7), 1437–1444 (2012)CrossRefGoogle Scholar
  38. 38.
    Zhou, H., Yu, X., Jacobs, D.W.: Glosh: global-local spherical harmonics for intrinsic image decomposition. In: ICCV (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Style2Paints ResearchSuzhouChina
  2. 2.Soochow UniversitySuzhouChina
  3. 3.The Chinese University of Hong KongHong KongChina

Personalised recommendations