Advertisement

Decouple Learning for Parameterized Image Operators

  • Qingnan Fan
  • Dongdong Chen
  • Lu Yuan
  • Gang Hua
  • Nenghai Yu
  • Baoquan ChenEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11217)

Abstract

Many different deep networks have been used to approximate, accelerate or improve traditional image operators, such as image smoothing, super-resolution and denoising. Among these traditional operators, many contain parameters which need to be tweaked to obtain the satisfactory results, which we refer to as “parameterized image operators”. However, most existing deep networks trained for these operators are only designed for one specific parameter configuration, which does not meet the needs of real scenarios that usually require flexible parameters settings. To overcome this limitation, we propose a new decouple learning algorithm to learn from the operator parameters to dynamically adjust the weights of a deep network for image operators, denoted as the base network. The learned algorithm is formed as another network, namely the weight learning network, which can be end-to-end jointly trained with the base network. Experiments demonstrate that the proposed framework can be successfully applied to many traditional parameterized image operators. We provide more analysis to better understand the proposed framework, which may inspire more promising research in this direction. Our codes and models have been released in https://github.com/fqnchina/DecoupleLearning.

Notes

Acknowledgement

This work was supported in part by: National 973 Program (2015CB352501), NSFC-ISF (61561146397), the Natural Science Foundation of China under Grant U1636201 and 61629301.

Supplementary material

474201_1_En_27_MOESM1_ESM.pdf (121 kb)
Supplementary material 1 (pdf 121 KB)

References

  1. 1.
    Andrychowicz, M., et al.: Learning to learn by gradient descent by gradient descent. In: Advances in Neural Information Processing Systems, pp. 3981–3989 (2016)Google Scholar
  2. 2.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 2, pp. 60–65. IEEE (2005)Google Scholar
  3. 3.
    Chen, D., Liao, J., Yuan, L., Yu, N., Hua, G.: Coherent online video style transfer. In: Proceedings of the International Conference on Computer Vision (ICCV) (2017)Google Scholar
  4. 4.
    Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: Stylebank: an explicit representation for neural image style transfer. In: Proceedings of the CVPR, vol. 1, p. 4 (2017)Google Scholar
  5. 5.
    Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: Stereoscopic neural style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 10 (2018)Google Scholar
  6. 6.
    Chen, Q., Xu, J., Koltun, V.: Fast image processing with fully-convolutional networks. In: IEEE International Conference on Computer Vision, vol. 9 (2017)Google Scholar
  7. 7.
    Chen, Y., Hoffman, M.W., Colmenarejo, S.G., Denil, M., Lillicrap, T.P., de Freitas, N.: Learning to learn for global optimization of black box functions. In: International Conference on Machine Learning (2017)Google Scholar
  8. 8.
    Cheng, B., et al.: Robust emotion recognition from low quality and low bit rate video: a deep learning approach. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 65–70. IEEE (2017)Google Scholar
  9. 9.
    Cheng, B., Wei, Y., Shi, H., Feris, R., Xiong, J., Huang, T.: Revisiting RCNN: on awakening the classification power of faster RCNN. In: ECCV (2018)CrossRefGoogle Scholar
  10. 10.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Dai, X., Ng, J.Y.H., Davis, L.S.: FASON: first and second order information fusion network for texture recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7352–7360 (2017)Google Scholar
  12. 12.
    Dai, X., Singh, B., Zhang, G., Davis, L.S., Qiu Chen, Y.: Temporal context network for activity localization in videos. In: The IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  13. 13.
    Dong, C., Deng, Y., Change Loy, C., Tang, X.: Compression artifacts reduction by a deep convolutional network. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 576–584 (2015)Google Scholar
  14. 14.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10593-2_13CrossRefGoogle Scholar
  15. 15.
    Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.: A generic deep architecture for single image reflection removal and image smoothing. In: Proceedings of the 16th International Conference on Computer Vision (ICCV), pp. 3238–3247 (2017)Google Scholar
  17. 17.
    Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.: Revisiting deep intrinsic image decompositions (2018)Google Scholar
  18. 18.
    Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. In: ACM Transactions on Graphics (TOG), vol. 27, p. 67. ACM (2008)CrossRefGoogle Scholar
  19. 19.
    Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1715–1723 (2017)Google Scholar
  21. 21.
    Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2862–2869 (2014)Google Scholar
  22. 22.
    Ha, D., Dai, A., Le, Q.V.: Hypernetworks. In: ICLR (2018)Google Scholar
  23. 23.
    He, M., Chen, D., Liao, J., Sander, P.V., Yuan, L.: Deep exemplar-based colorization. ACM Trans. Graph. 37, 47 (2018). Proceedings of SIGGRAPH 2018Google Scholar
  24. 24.
    Karacan, L., Erdem, E., Erdem, A.: Structure-preserving image smoothing via region covariances. ACM Trans. Graph. (TOG) 32(6), 176 (2013)CrossRefGoogle Scholar
  25. 25.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)Google Scholar
  26. 26.
    Kligvasser, I., Shaham, T.R., Michaeli, T.: xUnit: learning a spatial activation function for efficient image restoration. In: CVPR (2018)Google Scholar
  27. 27.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol. 2, p. 4 (2017)Google Scholar
  28. 28.
    Li, D., He, X., Huang, Q., Sun, M.T., Zhang, L.: Generating diverse and accurate visual captions by comparative adversarial learning. arXiv preprint arXiv:1804.00861 (2018)
  29. 29.
    Li, Y., Dixit, M., Vasconcelos, N.: Deep scene image classification with the MFAFVNet. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5746–5754 (2017)Google Scholar
  30. 30.
    Lin, K., Li, D., He, X., Zhang, Z., Sun, M.T.: Adversarial ranking for language generation. In: Advances in Neural Information Processing Systems, pp. 3155–3165 (2017)Google Scholar
  31. 31.
    Liu, S., Pan, J., Yang, M.-H.: Learning recursive filters for low-level vision via a hybrid neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 560–576. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_34CrossRefGoogle Scholar
  32. 32.
    Ma, S., Fu, J., Chen, C.W., Mei, T.: DA-GAN: instance-level image translation by deep attention generative adversarial networksGoogle Scholar
  33. 33.
    Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems, pp. 2802–2810 (2016)Google Scholar
  34. 34.
    Osher, S., Rudin, L.I.: Feature-oriented image enhancement using shock filters. SIAM J. Numer. Anal. 27(4), 919–940 (1990)CrossRefGoogle Scholar
  35. 35.
    Qi, G.J., Zhang, L., Hu, H., Edraki, M., Wang, J., Hua, X.S.: Global versus localized generative adversarial nets. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  36. 36.
    Schmidhuber, J.: Learning to control fast-weight memories: an alternative to dynamic recurrent networks. Neural Comput. 4(1), 131–139 (1992)CrossRefGoogle Scholar
  37. 37.
    Sun, J., Xu, Z., Shum, H.Y.: Image super-resolution using gradient profile prior. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008, pp. 1–8. IEEE (2008)Google Scholar
  38. 38.
    Tai, Y., Yang, J., Liu, X., Xu, C.: MemNet: A persistent memory network for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4539–4547 (2017)Google Scholar
  39. 39.
    Tipping, M.E., Bishop, C.M.: Bayesian image super-resolution. In: Advances in Neural Information Processing Systems, pp. 1303–1310 (2003)Google Scholar
  40. 40.
    Ulyanov, D., Vedaldi, A., Lempitsky, V.: Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. In: Proceedings of the CVPR (2017)Google Scholar
  41. 41.
    Wichrowska, O., et al.: Learned optimizers that scale and generalize. In: International Conference on Machine Learning (2017)Google Scholar
  42. 42.
    Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via L 0 gradient minimization. In: ACM Transactions on Graphics (TOG), vol. 30, p. 174. ACM (2011)Google Scholar
  43. 43.
    Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via l0 gradient minimization. ACM Trans. Graph. 30, 174 (2011). SIGGRAPH AsiaGoogle Scholar
  44. 44.
    Xu, L., Ren, J., Yan, Q., Liao, R., Jia, J.: Deep edge-aware filters. In: International Conference on Machine Learning, pp. 1669–1678 (2015)Google Scholar
  45. 45.
    Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM Trans. Graph. (TOG) 31(6), 139 (2012)Google Scholar
  46. 46.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: ICLR (2016)Google Scholar
  48. 48.
    Zhang, D., Dai, X., Wang, X., Wang, Y.F.: S3D: Single shot multi-span detector via fully 3D convolutional network. In: British Machine Vision Conference (BMVC) (2018)Google Scholar
  49. 49.
    Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: CVPR (2018)Google Scholar
  50. 50.
    Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 815–830. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10578-9_53CrossRefGoogle Scholar
  51. 51.
    Zhang, Q., Xu, L., Jia, J.: 100+ times faster weighted median filter (WMF). In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2830–2837 (2014)Google Scholar
  52. 52.
    Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 479–486. IEEE (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Qingnan Fan
    • 1
    • 3
  • Dongdong Chen
    • 2
  • Lu Yuan
    • 4
  • Gang Hua
    • 4
  • Nenghai Yu
    • 2
  • Baoquan Chen
    • 1
    • 5
    Email author
  1. 1.Shandong UniversityJinanChina
  2. 2.University of Science and Technology of ChinaHefeiChina
  3. 3.Beijing Film AcademyBeijingChina
  4. 4.Microsoft ResearchBeijingChina
  5. 5.Peking UniversityBeijingChina

Personalised recommendations